Skip to content

firefly21/diffbot

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

42 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Diffbot

This is a ruby client for the Diffbot API.

Install

Get the latest version from RubyGems:

$ gem install diffbot

Global Options

You can pass some settings to Diffbot like this:

Diffbot.configure do |config|
  config.token = ENV["DIFFBOT_TOKEN"]
  config.instrumentor = ActiveSupport::Notifications
end

The list of supported settings is:

  • token: Your Diffbot API token. This will be used for all requests in which you don't specify it manually (see below).
  • instrumentor: An object that matches the ActiveSupport::Notifications API, which will be used to trace network events. None is used by default.
  • article_defaults: Pass a block to this method to configure the global request settings used for Diffbot::Article requests. See below the options supported.

Articles

In order to fetch an article, do this:

require "diffbot"

article = Diffbot::Article.fetch(article_url, diffbot_token)

# Now you can inspect the result:
article.title
article.author
article.date
article.text
# etc. See below for the full list of available response attributes.

This is a list of all the fields returned by the Diffbot::Article.fetch call:

  • url: The URL of the article.
  • title: The title of the article.
  • author: The author of the article.
  • date: The date in which this article was published.
  • media: A list of media items attached to this article.
  • text: The body of the article. This will be plain text unless you specify the HTML option in the request.
  • tags: A list of tags/keywords extracted from the article.
  • xpath: The XPath at which this article was found in the page.

Options

You can customize your request like this:

article = Diffbot::Article.fetch(article_url, diffbot_token) do |request|
  request.html = true # Return HTML instead of plain text.
  request.dont_strip_ads = true # Leave any inline ads within the article.
  request.tags = true # Generate ads for the article.
  request.comments = true # Extract the comments from the article as well.
  request.summary = true # Return a summary text instead of the full text.
  request.stats = true # Return performance, probabilistic scoring stats.
end

Frontpages

In order to fetch and analyze a front page, do this:

require "diffbot"

frontpage = Diffbot::Frontpage.fetch(url, diffbot_token)

# Results are available in the returned object:
frontpage.title
frontpage.icon
frontpage.items #=> An array of Diffbot::Item instances

The fields you can extract from a Frontpage are:

  • title: The title of the page.
  • icon: The favicon of the page.
  • source_type: What kind of page this is.
  • source_url: The URL of the page.
  • items: The list of Diffbot::Item representing each item on the page.

The instances of Diffbot::Item have the following fields:

  • id: Unique identifier for this item.
  • title: Title of the item.
  • link: Extracted permalink of the item (if applicable).
  • description: innerHTML content of the item.
  • summary: A plain-text summary of the item.
  • pub_date: Date when item was detected on page.
  • type: The type of item, according to Diffbot. One of: IMAGE, LINK, STORY, CHUNK.
  • img: The main image extracted from this item.
  • xroot: XPath of where the item was found on the page.
  • cluster: XPath of the cluster of items where this item was found.
  • stats: An object with the following attributes:
    • spam_score: A Float between 0.0 and 1.0 indicating the probability this item is spam/an advertisement.
    • static_rank: A Float between 1.0 and 5.0 indicating the quality score of the item.
    • fresh: The percentage of the item that has changed compared to the previous crawl.

TODO

  • Implement the Follow API.
  • Add tests for Article and Frontpage requests.
  • Add a Frontpage.crawl method that given the URL of a frontpage, it will fetch the article for each item in the page.

License

This is published under an MIT License, see LICENSE for further details.

About

The ruby Diffbot API client

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Ruby 100.0%