No description or website provided.
Ruby Shell HTML
Latest commit 33636ad Feb 16, 2017 @okkez okkez Bump up version to 1.0.0
Permalink
Failed to load latest commit information.
bin
db Add schema.rb Jan 22, 2016
lib Bump up version to 1.0.0 Feb 15, 2017
sample
templates Update PostgreSQL version in Dockerfile.db Feb 14, 2017
test Organize storage API to read data Feb 9, 2017
.gitignore Add new sample Dec 6, 2016
.travis.yml travis: Add Ruby 2.4.0 Jan 19, 2017
.yardopts Add .yardopts Feb 7, 2017
Gemfile Use double quote for string literal Aug 23, 2016
LICENSE.txt We are daimon developers Oct 25, 2016
README.md Production ready! Jan 19, 2017
Rakefile Rename test files Jan 31, 2017
daimon_skycrawlers.gemspec

README.md

DaimonSkycrawlers

Gem Version Build Status

DaimonSkycrawlers is a crawler framework.

Requirements

  • Ruby
  • RabbitMQ
  • RDB
    • PostgreSQL (default)
    • MySQL
    • SQLite3

Installation

Add this line to your application's Gemfile:

gem 'daimon_skycrawlers'

And then execute:

$ bundle

Or install it yourself as:

$ gem install daimon_skycrawlers

Usage

  1. Create project

    $ bundle exec daimon_skycrawlers new mycrawlers
    $ cd mycrawlers
    

    or

    $ daimon_skycrawlers new mycrawlers
    $ cd mycrawlers
    
  2. Install dependencies

    $ bundle install
    
  3. Create database

    $ bundle exec rake db:create
    $ bundle exec rake db:migrate
    
  4. Open new terminal and run crawler/processor

    $ daimon_skycrawlers exec crawler   # on new terminal
    $ daimon_skycrawlers exec processor # on new terminal
    

    NOTE: Execute step 5 as soon as possible. Because crawler and processor will stop after 10 seconds by default if their queues are empty.

    NOTE: You can change shutdown_interval using following code in config/init.rb:

    DaimonSkycrawlers.configure do |config|
      config.shutdown_interval = 30
    end
  5. Enqueue task

    $ daimon_skycrawlers enqueue url http://example.com/
    
  6. You'll see It works with 'http://example.com' on your terminal which runs your processor!

  7. You can re-enqueue task for processor

    $ daimon_skycrawlers enqueue response http://example.com/
    

Display It works with 'http://example.com' again on your terminal which runs your processor.

docker-compose

  1. Create project

    $ bundle exec daimon_skycrawlers new mycrawlers
    $ cd mycrawlers
    

    or

    $ daimon_skycrawlers new mycrawlers
    $ cd mycrawlers
    
  2. Build docker images

    $ docker-compose build
    
  3. Run docker containers

    $ docker-compose up -d
    
  4. Run a command on docker containers

    $ docker-compose exec <service name> <command>
    

    For example,

    $ docker-compose exec mycrawlers-db bash
    $ docker-compose exec mycrawlers-crawler sh
    $ docker-compose exec mycrawlers-crawler bundle exec daimon_skycrawlers enqueue url http://example.com/
    $ docker-compose exec mycrawlers-crawler bundle exec daimon_skycrawlers enqueue response http://example.com/
    
  5. Shutdown docker containers

    $ docker-compose down
    $ docker-compose down --rmi all # Remove all related images
    

Development

After checking out the repo, run bundle install to install dependencies. Then, run bundle exec rake test to run the tests. You can also run bundle console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/bm-sms/daimon_skycrawlers.

License

The gem is available as open source under the terms of the MIT License.