Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.
Ruby
branch: master

Merge pull request #2 from joker1007/fix_spec_helper

Fix spec_helper.rb. base on "rspec --init"
latest commit a9a9aa3763
Toshiaki Koshiba authored
Failed to load latest commit information.
lib first release!
spec
.gitignore
.rspec Fix spec_helper.rb. base on "rspec --init"
.travis.yml
Gemfile first release!
README.md fix Travis CI Build Status image
Rakefile first release!
cosmicrawler.gemspec

README.md

Cosmicrawler

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by http, file, etc using EventMachine.

Build Status

Installation

Add this line to your application's Gemfile:

gem 'cosmicrawler'

And then execute:

$ bundle

Or install it yourself as:

$ gem install cosmicrawler

Usage

http

require 'cosmicrawler'

Cosmicrawler.http_crawl(%w(http://example.com/1 http://example.com/2)) {|request|
  get = request.get
  puts get.response if get.response_header.status == 200
}
require 'cosmicrawler'
require 'em-http-request'

Cosmicrawler.each(%w(http://example.com/1 http://example.com/2)) {|item|
  request = EM::HttpRequest.new(item)
  get = request.get
  puts get.response if get.response_header.status == 200
}      

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

License

Ruby's License

Something went wrong with that request. Please try again.