v1.0

Release: Node.js Webpage Scraper

This release includes a Node.js script to scrape webpages for <a> tags with links matching specified patterns and download the linked files to a specified directory. It also includes a scheduler script to run the scraper periodically based on a configuration file. Docker support is provided to simplify the environment setup and ensure consistency.

Features

• Webpage Scraper: Scrapes a webpage for links matching a specified pattern and downloads the linked files.
• Scheduler: Periodically runs the scraper based on a configuration file, supporting both JSON and JavaScript formats.
• Docker Support: Dockerfile and Docker Compose configuration for containerized execution.

Usage

Running the Scraper

node scrap.js <webpage URL> <file RegExp pattern> <local download location:optional, defaults to current folder>

Parameters

• <webpage URL>: The URL of the webpage to scrape.
• <file RegExp pattern>: The regular expression pattern to match file links.
• <local download location>: Optional. The local directory to download the files. Defaults to the current directory if not specified.

Examples

Download all torrent files from a specific webpage to the current directory:

node scrap.js https://torrent.com/ '.*\.torrent$'

Specify a local download location:

node scrap.js https://torrent.com/ '.*\.torrent$' ./downloads

Running the Scheduler

Create a configuration file (config.json or config.js) with the following properties:

• webpageUrls: An array of objects with url and pattern properties.
• downloadLocation: The directory to download the files.
• interval: The interval in seconds between each run.
• runCount: The number of times to run the script (0 for infinite).
• dbPath: The path to the scrap.db file to keep track of downloaded URLs.

Example config.json

{
  "webpageUrls": [
    {
      "url": "https://example.com",
      "pattern": ".*\\.torrent$"
    },
    {
      "url": "https://another.com",
      "pattern": ".*\\.zip$"
    }
  ],
  "downloadLocation": "./downloads",
  "interval": 3600,
  "runCount": 0,
  "dbPath": "./scrap.db"
}

Running the Scheduler with the Configuration File

node scheduler.js <config file>

Example:

node scheduler.js ./config.json

Docker Usage

Dockerfile

A Dockerfile is provided to build the Docker image. See the Dockerfile for details.

Docker Compose (Optional)

A docker-compose.yml file can be used for easier management. See the docker-compose.yml for details.

Building the Docker Image

docker build -t webfile-scheduler .

Running the Docker Container

docker run -v $(pwd)/config:/config -v $(pwd)/downloads:/downloads webfile-scheduler

Using Docker Compose

docker-compose up

License

This project is licensed under the MIT License. See the LICENSE file for details.

---