fetchmaker

Sequential downloader using Tera to provide support for templated URLs.

Synopsys

Use the fetch.example.yaml config file to provide sources to download:

entries:
  - type: range
    name: Range Example
    range: [1, 5]
    destination_path: tmp/
    source_url: https://example.com/

  - type: list
    name: List Example
    destination_path: tmp/
    source_urls:
      - https://example.com/
      - https://example.com/
      - https://example.com/

Result of the config:

tmp
├── List Example - 1.html
├── List Example - 2.html
├── List Example - 3.html
├── Range Example - 1.html
├── Range Example - 2.html
├── Range Example - 3.html
├── Range Example - 4.html
└── Range Example - 5.html

Options

Type: range

• type: range, required
• name: string, required
• range: Tuple [uint, uint] e.g. [1, 10], required, is provided to source_url as i
• source_url: string, required, supports jinja2 variable i
• destination_path: string, required
• referer: string, optional

Example:

entries:
  - type: range
    name: Range Example
    range: [1, 5]
    # {{ i | pad_start(length=3, char="0") }}
    # 1    -->  001
    # 10   -->  010
    # 100  -->  100
    # 1000 --> 1000
    source_url: https://example.com/some/pages/to/download{{ i | pad_start(length=3, char="0") }}
    destination_path: tmp/
    referer: https://example.com/some-referer-page

Type: list

• type: list, required
• name: string, required
• source_urls: list of strings, required
• destination_path: string, required
• referer: string, optional

Example: Example:

entries:
  - type: list
    name: List Example
    source_urls:
      - https://example.com/1.html
      - https://example.com/2.html
      - https://example.com/3.html
    destination_path: tmp/
    referer: https://example.com/some-referer-page

Missing features / Issues

• Customize config file path
• Customize tracing level

License

See LICENSE.md.