Turn any web page into a clean view
HTML JavaScript Shell
Latest commit 8ae769a Jul 25, 2016 @haroldtreen haroldtreen committed on GitHub Merge pull request #77 from haroldtreen/master
Fix for psychology-today + ux milk.
Failed to load latest commit information.
bin chore: add release script Jan 31, 2016
examples Add examples Sep 4, 2014
src Merge branch 'master' of https://github.com/luin/readability Jul 25, 2016
test Add tests for ux-milk Jul 25, 2016
.gitignore Add gitignore Mar 17, 2013
.npmignore Remove test folder from npm package. May 7, 2016
.travis.yml Update travis Sep 15, 2015
CHANGELOG.md 2.2.0 Mar 11, 2016
README.md Fix typo Apr 7, 2016
package.json 2.2.0 Mar 11, 2016

README.md

Readability

Turn any web page into a clean view. This module is based on arc90's readability project.

Build Status

Features

  1. Optimized for more websites.
  2. Supporting HTML5 tags(article, section) and Microdata API.
  3. Focusing on both accuracy and performance. 4x times faster than arc90's version.
  4. Supporting encodings such as GBK and GB2312.
  5. Converting relative urls to absolute for images and links automatically(Thank Guillermo Baigorria & Tom Sutton).

Example

Before -> After

Install

$ npm install node-readability

Note that as of our 2.0.0 release, this module only works with Node.js >= 2.0. In the meantime you are still welcome to install a release in the 1.x series(by npm install node-readability@1) if you use an older Node.js version.

Usage

read(html [, options], callback)

Where

  • html url or html code.
  • options is an optional options object
  • callback is the callback to run - callback(error, article, meta)

Example

var read = require('node-readability');

read('http://howtonode.org/really-simple-file-uploads', function(err, article, meta) {
  // Main Article
  console.log(article.content);
  // Title
  console.log(article.title);

  // HTML Source Code
  console.log(article.html);
  // DOM
  console.log(article.document);

  // Response Object from Request Lib
  console.log(meta);

  // Close article to clean up jsdom and prevent leaks
  article.close();
});

NB If the page has been marked with charset other than utf-8, it will be converted automatically. Charsets such as GBK, GB2312 is also supported.

Options

node-readability will pass the options to request directly. See request lib to view all available options.

node-readability has two additional options:

  • cleanRulers which allow set your own validation rule for tags.

If true rule is valid, otherwise no. options.cleanRulers = [callback(obj, tagName)]

read(url, {
  cleanRulers: [
    function(obj, tag) {
      if(tag === 'object') {
        if(obj.getAttribute('class') === 'BrightcoveExperience') {
          return true;
        }
      }
    }
  ]}, function(err, article, response) {
    //...
  });
  • preprocess which should be a function to check or modify downloaded source before passing it to readability.

options.preprocess = callback(source, response, contentType, callback);

read(url, {
  preprocess: function(source, response, contentType, callback) {
    if (source.length > maxBodySize) {
      return callback(new Error('too big'));
    }
    callback(null, source);
  }, function(err, article, response) {
    //...
  });

article object

content

The article content of the web page. Return false if failed.

title

The article title of the web page. It's may not same to the text in the <title> tag.

textBody

A string containing all the text found on the page

html

The original html of the web page.

document

The document of the web page generated by jsdom. You can use it to access the DOM directly(for example, article.document.getElementById('main')).

meta object

response object from request lib. If you need to get current url after all redirect or get some headers it can be useful.

Why not Cheerio

This lib is using jsdom to parse HTML instead of cheerio because some data such as image size and element visibility isn't able to acquire when using cheerio, which will significantly affect the result.

Contributors

https://github.com/luin/node-readability/graphs/contributors

License

This code is under the Apache License 2.0. http://www.apache.org/licenses/LICENSE-2.0

Bitdeli Badge