Node scraping using Express. Application is for learn purpose and for future usage. The site which is scraped is thecodinglove. If you would like to write own scraper, fill free to fork it.
Thecodinglove.com scraping using express.
1.1. Database configuration
$ cp config/database.example.json config/database.json1.2 Secrets config
$ cp config/secrets.example.json config/secrets.jsonNote: Set the cron schedule in every: config/secrets.json key: cron Note2: Remember that first cron job will start immediate but job will be executed after this time.
$ npm install or
$ yarn$ npm startRemember, it is not cool to DDoS sites. Thecodinglove.com is super cool. Do not run cron job if u do not need it.
| Plugin | Description |
|---|---|
| Express | Fast, unopinionated, minimalist web framework for node |
| Mongoose | Mongoose is a MongoDB object modeling tool designed to work in an asynchronous environment |
| Babel | The compiler for writing next generation JavaScript |
| Request | Simplified HTTP client |
| ESLint | A fully pluggable tool for identifying and reporting on patterns in JavaScript |
| Joi | Object schema validation |
| Body-Parser | Node.js body parsing middleware |
| Cheerio | Fast, flexible, and lean implementation of core jQuery designed specifically for the server |
| Nodemon | Monitor for any changes in your node.js application and automatically restart the server - perfect for development |
| Node-cron | Cron jobs for your node |
MIT © Krystian Błaszczyk