A powerful miner that will scrape html pages for you. ` HTML Scraper ´
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs
examples
lib
test
.codeclimate.yml
.eslintignore
.eslintrc.yml
.gitignore
.travis.yml
LICENSE
README.md
index.js
package-lock.json
package.json

README.md

HTML Miner

Npm Build Status Coverage Status Codacy Badge Code Climate Issue Count

A powerful miner that will scrape html pages for you.

Install

NPM

# using npm
npm i --save html-miner

# using yarn
yarn add html-miner

Usage

Arguments

html-miner accepts two arguments: html and selector.

const htmlMiner = require('html-miner');

// htmlMiner(html, selector);

HTML

html is a string and contains html code.

let html = '<div class="title">Hello <span>Marco</span>!</div>';

SELECTOR

selector could be:

STRING

htmlMiner(html, '.title');
//=> Hello Marco!

If the selector extracts more elements, the result is an array:

let htmlWithDivs = '<div>Element 1</div><div>Element 2</div>';
htmlMiner(htmlWithDivs, 'div');
//=> ['Element 1', 'Element 2']

FUNCTION

Read function in detail paragraph.

htmlMiner(html, () => { return 'Hello everyone!' });
//=> Hello everyone!

ARRAY

htmlMiner(html, ['.title', 'span']);
//=> ['Hello Marco!', 'Marco']

OBJECT

htmlMiner(html, {
    title: '.title',
    who: 'span'
});
//=> {
//     title: 'Hello Marco!',
//     who: 'Marco'
//   }

You can combine array and object with each other or with string and functions.

htmlMiner(html, {
    title: '.title',
    who: '.title span',
    upper: (arg) => { return arg.scopeData.who.toUpperCase(); }
});
//=> {
//     title: 'Hello Marco!',
//     who: 'Marco',
//     upper: 'MARCO'
//   }

Function in detail

A function accepts only one argument that is an object containing:

  • $: is a jQuery-like function pointing to the document ( html argument ). You can use it to query and fetch elements from the html.

    htmlMiner(html, (arg) => { return arg.$('.title').text(); });
    //=> Hello Marco!
  • $scope: useful when combined with _each_ (read special keys paragraph).

    htmlMiner(html, {
        title: '.title',
        spanList: {
            _each_: 'span',
            value: (arg) => {
                // "arg.$scope.find('.title')" doesn't exist.
                return arg.$scope.text();
            }
        }
    });
    //=> {
    //     title: 'Hello Marco!',
    //     spanList: [{
    //         value: 'Marco'
    //     }]
    //   }
  • globalData: is an object that contains all previously fetched datas.

    htmlMiner(html, {
        title: '.title',
        spanList: {
            _each_: '.title span',
            pageTitle: function(arg) {
                // "arg.globalData.who" is undefined because defined later.
                return arg.globalData.title;
            }
        },
        who: '.title span'
    });
    //=> {
    //     title: 'Hello Marco!',
    //     spanList: [{
    //         pageTitle: 'Hello Marco!'
    //     }],
    //     who: 'Marco'
    //   }
  • scopeData: similar to globalData, but only contains scope data. Useful when combined with _each_ (read special keys paragraph).

    htmlMiner(html, {
        title: '.title',
        upper: (arg) => { return arg.scopeData.title.toUpperCase(); },
        sublist: {
            who: '.title span',
            upper: (arg) => {
                // "arg.scopeData.title" is undefined because "title" is out of scope.
                return arg.scopeData.who.toUpperCase();
            },
        }
    });
    //=> {
    //     title: 'Hello Marco!',
    //     upper: 'HELLO MARCO!',
    //     sublist: {
    //         who: 'Marco',
    //         upper: 'MARCO'
    //     }
    //   }

Special keys

When selector is an object, you can use special keys:

  • _each_: creates a list of items. HTML Miner will iterate for the value and will parse siblings keys.

    {
        articles: {
            _each_: '.articles .article',
            title: 'h2',
            content: 'p',
        }
    }
  • _eachId_: useful when combined with _each_. Instead of creating an Array, it creates an Object where keys are the result of _eachId_ function.

    {
        articles: {
            _each_: '.articles .article',
            _eachId_: function(arg) {
                return arg.$scope.data('id');
            }
            title: 'h2',
            content: 'p',
        }
    }
  • _container_: uses the parsed value as container. HTML Miner will parse siblings keys, searching them inside the container.

    {
        footer: {
            _container_: 'footer',
            copyright: (arg) => { return arg.$scope.text().trim(); },
            company: 'span' // find only 'span' inside 'footer'.
        }
    }

For more details see the following example.

Let's try this out

Consider the following html snippet: we will try and fetch some information.

<h1>Hello, <span>world</span>!</h1>
<div class="articles">
    <div class="article" data-id="a001">
        <h2>Heading 1</h2>
        <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>
    </div>
    <div class="article" data-id="a002">
        <h2>Heading 2</h2>
        <p>Donec maximus ipsum quis est tempor, sit amet laoreet libero bibendum.</p>
    </div>
    <div class="article" data-id="a003">
        <h2>Heading 3</h2>
        <p>Suspendisse viverra convallis risus, vitae molestie est tincidunt eget.</p>
    </div>
</div>
<footer>
    <p>&copy; <span>Company</span> 2017</p>
</footer>
const htmlMiner = require('html-miner');

let json = htmlMiner(html, {
    title: 'h1',
    who: 'h1 span',
    h2: 'h2',
    articlesArray: {
        _each_: '.articles .article',
        title: 'h2',
        content: 'p',
    },
    articlesObject: {
        _each_: '.articles .article',
        _eachId_: function(arg) {
            return arg.$scope.data('id');
        },
        title: 'h2',
        content: 'p',
    },
    footer: {
        _container_: 'footer',
        copyright: (arg) => { return arg.$scope.text().trim(); },
        company: 'span',
        year: (arg) => { return arg.scopeData.copyright.match(/[0-9]+/)[0]; },
    },
    greet: () => { return 'Hi!'; }
});

console.log( json );

//=> {
//     title: 'Hello, world!',
//     who: 'world',
//     h2: ['Heading 1', 'Heading 2', 'Heading 3'],
//     articlesArray: [
//         {
//             title: 'Heading 1',
//             content: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit.',
//         },
//         {
//             title: 'Heading 2',
//             content: 'Donec maximus ipsum quis est tempor, sit amet laoreet libero bibendum.',
//         },
//         {
//             title: 'Heading 3',
//             content: 'Suspendisse viverra convallis risus, vitae molestie est tincidunt eget.',
//         }
//     ],
//     articlesObject: {
//         'a001': {
//             title: 'Heading 1',
//             content: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit.',
//         },
//         'a002': {
//             title: 'Heading 2',
//             content: 'Donec maximus ipsum quis est tempor, sit amet laoreet libero bibendum.',
//         },
//         'a003': {
//             title: 'Heading 3',
//             content: 'Suspendisse viverra convallis risus, vitae molestie est tincidunt eget.',
//         }
//     },
//     footer: {
//         copyright: '© Company 2017',
//         company: 'Company',
//         year: '2017'
//     },
//     greet: 'Hi!'
//   }

You can find other examples under the folder /examples.

# you can test examples with nodejs
node examples/demo.js
node examples/site.js

Development

npm install
npm test

# build and start
npm run gh-pages
npm start