Skip to content
The ruby Diffbot API client http://www.diffbot.com/docs
Find file
Pull request Compare This branch is 32 commits behind greshny:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
lib
test
.gitignore
.travis.yml
Gemfile
LICENSE
README.md
Rakefile
diffbot.gemspec

README.md

Diffbot

This is a ruby client for the Diffbot API.

Install

Get the latest version from RubyGems:

$ gem install diffbot

Global Options

You can pass some settings to Diffbot like this:

Diffbot.configure do |config|
  config.token = ENV["DIFFBOT_TOKEN"]
  config.instrumentor = ActiveSupport::Notifications
end

The list of supported settings is:

  • token: Your Diffbot API token. This will be used for all requests in which you don't specify it manually (see below).
  • instrumentor: An object that matches the ActiveSupport::Notifications API, which will be used to trace network events. None is used by default.
  • article_defaults: Pass a block to this method to configure the global request settings used for Diffbot::Article requests. See below the options supported.

Articles

In order to fetch an article, do this:

require "diffbot"

article = Diffbot::Article.fetch(article_url, diffbot_token)

# Now you can inspect the result:
article.title
article.author
article.date
article.text
# etc. See below for the full list of available response attributes.

This is a list of all the fields returned by the Diffbot::Article.fetch call:

  • url: The URL of the article.
  • title: The title of the article.
  • author: The author of the article.
  • date: The date in which this article was published.
  • media: A list of media items attached to this article.
  • text: The body of the article. This will be plain text unless you specify the HTML option in the request.
  • tags: A list of tags/keywords extracted from the article.
  • xpath: The XPath at which this article was found in the page.

Options

You can customize your request like this:

article = Diffbot::Article.fetch(article_url, diffbot_token) do |request|
  request.html = true # Return HTML instead of plain text.
  request.dont_strip_ads = true # Leave any inline ads within the article.
  request.tags = true # Generate ads for the article.
  request.comments = true # Extract the comments from the article as well.
  request.summary = true # Return a summary text instead of the full text.
  request.stats = true # Return performance, probabilistic scoring stats.
end

Frontpages

In order to fetch and analyze a front page, do this:

require "diffbot"

frontpage = Diffbot::Frontpage.fetch(url, diffbot_token)

# Results are available in the returned object:
frontpage.title
frontpage.icon
frontpage.items #=> An array of Diffbot::Item instances

The fields you can extract from a Frontpage are:

  • title: The title of the page.
  • icon: The favicon of the page.
  • source_type: What kind of page this is.
  • source_url: The URL of the page.
  • items: The list of Diffbot::Item representing each item on the page.

The instances of Diffbot::Item have the following fields:

  • id: Unique identifier for this item.
  • title: Title of the item.
  • link: Extracted permalink of the item (if applicable).
  • description: innerHTML content of the item.
  • summary: A plain-text summary of the item.
  • pub_date: Date when item was detected on page.
  • type: The type of item, according to Diffbot. One of: IMAGE, LINK, STORY, CHUNK.
  • img: The main image extracted from this item.
  • xroot: XPath of where the item was found on the page.
  • cluster: XPath of the cluster of items where this item was found.
  • stats: An object with the following attributes:
    • spam_score: A Float between 0.0 and 1.0 indicating the probability this item is spam/an advertisement.
    • static_rank: A Float between 1.0 and 5.0 indicating the quality score of the item.
    • fresh: The percentage of the item that has changed compared to the previous crawl.

TODO

  • Implement the Follow API.
  • Add tests for Article and Frontpage requests.
  • Add a Frontpage.crawl method that given the URL of a frontpage, it will fetch the article for each item in the page.

License

This is published under an MIT License, see LICENSE for further details.

Something went wrong with that request. Please try again.