Scrape data from websites using Open Graph metadata, regular HTML metadata, and a series of fallbacks.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.github Update ISSUE_TEMPLATE.md Aug 25, 2018
bench Add bench Dec 10, 2017
packages v4.5.3 Oct 11, 2018
src Add copy clipboard button Mar 27, 2018
static Use microlinkhq/normalize-url (#83) May 10, 2018
.bumpedrc WIP Jul 1, 2017
.editorconfig WIP Jul 1, 2017
.gitattributes WIP Jul 1, 2017
.gitignore Remove postinstall script Aug 11, 2018
.nojekyll Add documentation site Dec 15, 2017
.npmrc Add npmrc Dec 19, 2017
.travis.yml Setup travis Dec 14, 2017
CHANGELOG.md Update Changelog Oct 11, 2018
CNAME Add CNAME Dec 15, 2017
LICENSE.md Rename branding Jul 9, 2017
README.md Throw qualified errors (#118) Sep 18, 2018
gulpfile.js Add documentation site Dec 15, 2017
index.html Update index.html Jul 15, 2018
lerna.json v4.5.3 Oct 11, 2018
package.json Add tunnel support (#124) Oct 5, 2018

README.md


metascraper

Last version Build Status Dependency Status NPM Status

A library to easily scrape metadata from an article on the web using Open Graph metadata, regular HTML metadata, and series of fallbacks.

Table of Contents

Getting Started

metascraper is library to easily scrape metadata from an article on the web using Open Graph metadata, regular HTML metadata, and series of fallbacks.

It follows a few principles:

  • Have a high accuracy for online articles by default.
  • Make it simple to add new rules or override existing ones.
  • Don't restrict rules to CSS selectors or text accessors.

Installation

$ npm install metascraper --save

Usage

Let's extract accurate information from the following article:

Then call metascraper with the rules bundle you want to apply for extracting content:

const metascraper = require('metascraper')([
  require('metascraper-author')(),
  require('metascraper-date')(),
  require('metascraper-description')(),
  require('metascraper-image')(),
  require('metascraper-logo')(),
  require('metascraper-clearbit-logo')(),
  require('metascraper-publisher')(),
  require('metascraper-title')(),
  require('metascraper-url')()
])

const got = require('got')

const targetUrl = 'http://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance'

;(async () => {
  const { body: html, url } = await got(targetUrl)
  const metadata = await metascraper({ html, url })
  console.log(metadata)
})()

The output will be something like:

{
  "author": "Ellen Huet",
  "date": "2016-05-24T18:00:03.894Z",
  "description": "The HR startups go to war.",
  "image": "https://assets.bwbx.io/images/users/iqjWHBFdfxIU/ioh_yWEn8gHo/v1/-1x-1.jpg",
  "publisher": "Bloomberg.com",
  "title": "As Zenefits Stumbles, Gusto Goes Head-On by Selling Insurance",
  "url": "http://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance"
}

Metadata

?> Other metadata can be defined using a custom rule bundle.

Here is an example of the metadata that metascraper can collect:

How it works

metascraper is built out of rules bundles.

It was designed to be easy to adapt. You can compose your own transformation pipeline using existing rules or write your own.

Rules bundles are a collection of HTML selectors around a determinate property. When you load the library, implicitly it is loading core rules.

Each set of rules load a set of selectors in order to get a determinate value.

These rules are sorted with priority: The first rule that resolve the value successfully, stop the rest of rules for get the property. Rules are sorted intentionally from specific to more generic.

Rules work as fallback between them:

  • If the first rule fails, then it fallback in the second rule.
  • If the second rule fails, time to third rule.
  • etc

metascraper do that until finish all the rule or find the first rule that resolves the value.

Importing Rules

metascraper exports a constructor that need to be initialized providing a collection of rules to load:

const metascraper = require('metascraper')([
  require('metascraper-author')(),
  require('metascraper-date')(),
  require('metascraper-description')(),
  require('metascraper-image')(),
  require('metascraper-logo')(),
  require('metascraper-clearbit-logo')(),
  require('metascraper-publisher')(),
  require('metascraper-title')(),
  require('metascraper-url')()
])

Again, the order of rules are loaded are important: Just the first rule that resolve the value will be applied.

Use the first parameter to pass custom options specific per each rules bundle:

const metascraper = require('metascraper')([
  require('metascraper-clearbit-logo')({
    size: 256,
    format: 'jpg'
  })
])

Rules bundles

?> Can't find the rules bundle that you want? Let's open an issue to create it.

Package Version Dependencies
metascraper-amazon npm Dependency Status
metascraper-audio npm Dependency Status
metascraper-author npm Dependency Status
metascraper-clearbit-logo npm Dependency Status
metascraper-date npm Dependency Status
metascraper-description npm Dependency Status
@metascraper/helpers npm Dependency Status
metascraper-image npm Dependency Status
metascraper-lang npm Dependency Status
metascraper-lang-detector npm Dependency Status
metascraper-logo npm Dependency Status
metascraper-logo-favicon npm Dependency Status
metascraper-media-provider npm Dependency Status
metascraper-publisher npm Dependency Status
metascraper-soundcloud npm Dependency Status
metascraper-title npm Dependency Status
metascraper-url npm Dependency Status
metascraper-video npm Dependency Status
metascraper-youtube npm Dependency Status

Write your own rules

A rule bundle is the simplest way for extending metascraper functionality.

A rule bundle can add one or more properties support.

The following schema represents the API compromise that a rule bundle need to follow:

'use strict'

// `opts` can be loaded using `.metascraperrc` config file
module.exports = opts => {
  // You can define as props as you want.
  // props are organized based on an object key.
  return ({
    logo: [
      // You can setup more than one rules per prop (priority is important!).
      // They receive as parameter:
      // - `htmlDom`: the cheerio HTML instance.
      // - `url`: The input URL used for extact the content.
      // - `meta`: The current state of the information detected.
      ({ htmlDom: $, meta, url: baseUrl }) => wrap($ => $('meta[property="og:logo"]').attr('content')),
      ({ htmlDom: $, meta, url: baseUrl }) => wrap($ => $('meta[itemprop="logo"]').attr('content'))
    ]
  })
}

We recommend check one of the rule bundles already implemented to have a better perspective.

API

constructor(rules)

Create a new metascraper instance declaring the rules bundle to be used explicitly.

rules

Type: Array

The collection of rules bundle to be loaded.

metascraper(options)

Call the instance for extracting content based on rules bundle provided at the constructor.

options

url

Required
Type: String

The URL associated with the HTML markup.

It is used for resolve relative links that can be present in the HTML markup.

it can be used as fallback field for different rules as well.

html

Type: String

The HTML markup for extracting the content.

rules

Type: Array

You can pass additional rules on execution time. These rules will be merged with your loaded rules.

Benchmark

To give you an idea of how accurate metascraper is, here is a comparison of similar libraries:

Library metascraper html-metadata node-metainspector open-graph-scraper unfluff
Correct 95.54% 74.56% 61.16% 66.52% 70.90%
Incorrect 1.79% 1.79% 0.89% 6.70% 10.27%
Missed 2.68% 23.67% 37.95% 26.34% 8.95%

A big part of the reason for metascraper's higher accuracy is that it relies on a series of fallbacks for each piece of metadata, instead of just looking for the most commonly-used, spec-compliant pieces of metadata, like Open Graph.

metascraper's default settings are targetted specifically at parsing online articles, which is why it's able to be more highly-tuned than the other libraries for that purpose.

If you're interested in the breakdown by individual pieces of metadata, check out the full comparison summary, or dive into the raw result data for each library.

License

metascraper © Ian Storm Taylor, Released under the MIT License.
Maintained by Kiko Beats with help from contributors.