Skip to content
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
.babelrc
.eslintrc.json
.gitignore
.prettierrc
.travis.yml
LICENSE
README.md
jest.config.js
launch.json
package.json
tsconfig.json

README.md

Periodic Crawler

Build Status npm

I do NOT recommend using it in real projects.

npm i periodic-crawler

Exsampls

const periodicCrawler = require('periodic-crawler');

async () => {
  const res = await periodicCrawler(['www.example.com/'], {
    proxy: {
      useProxy: false
    },
    settings: {
      pauseMin: 1,
      pauseMax: 2
    }
  });
  console.log(res);
};

// in console:
// [
//   {
//     "url": "www.example.com/",
//     "dowDate": "2019-04-01T02:39:55.268Z",
//     "body": "<!doctype html>\n<html>\n<head></head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is established to be used for illustrative examples in documents. You may use this\n    domain in examples without prior coordination or asking for permission.</p>\n    <p><a href=\"http://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
//   }
// ]

TODO

You can’t perform that action at this time.