Skip to content
Scrapinghub Ruby client
Ruby
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib
spec
.gitignore
.rspec
.travis.yml
Gemfile
LICENSE
README.md
Rakefile
scrapinghub.gemspec

README.md

Scrapinghub client

Build Status Dependency Status Code Climate Test Coverage

Ruby client for the Scrapinghub API. So far it only supports the Jobs API (pull requests welcome).

This library tries to take an FP-ish approach. It uses the contracts gem for validating function input and output types (see the docs for the full list of functions and their types) and the kleisli gem for returning composition-friendly output types. Outputs will be a Left if the Scrapinghub API returns failure or if an exception was raised (e.g. a network timeout), or a Right if the operation was successful.

The Kleisli gem introductory blog post gives some great examples on how to work with the output types.

Install

Add to Gemfile:

gem "scrapinghub-client"

Note: although the gem is named scrapinghub-client, the gem's namespace is Scrapinghub.

Example

require "scrapinghub-client"

j = Scrapinghub::Jobs.new(api_key: 'abc123')
j.schedule(project: 123, spider: "foo", add_tag: "bar", extra: { DOWNLOAD_DELAY: "0.5" })
  .fmap{|r| puts "Job scheduled! Jobid: #{r['jobid']}"}
  .or{|f| puts "Failed to schedule job! Reason: #{f.value}"}
You can’t perform that action at this time.