novel-downloader

Rust command-line crawler for Vietnamese novel sites. It downloads chapter ranges as clean local HTML files and can package those saved chapters into an EPUB with metadata, cover image, and embedded font support.

Supported hosts:

• metruyenhotvn.com
• metruyenhotne.com
• khodocsach.com
• xtruyen.vn

You never pick a source. Pass a novel URL and the host decides which adapter handles it: the metruyenhot hosts are scraped from HTML, khodocsach.com is read through its JSON API, and xtruyen.vn is scraped from HTML whose chapter text arrives encoded and is decoded locally. URLs from any other host are rejected with a clear error before any network fetch. The hidden --allow-any-host flag bypasses the check for local mock fixtures and integration tests; it is not intended for normal use.

A khodocsach chapter costs two requests, a short-lived ticket followed by the chapter content, so its adapter does more work per chapter than the metruyenhot scraper does. Interrupting a run costs nothing on any host: re-run the same command and already-saved chapters are skipped.

xtruyen.vn enforces a request limit per client address, so its adapter sets the pace: at most 2 concurrent requests with half a second between them, and a refusal is retried after however long the site says to wait. That is applied whatever --workers and --delay say, and the run tells you when it overrides either. The interactive wizard does not ask for those two values for this host at all, since the answers would be discarded, and the confirmation screen shows the pacing the run will actually use. Expect roughly two minutes for a 200-chapter novel. Chapter numbering follows the site's own reading order rather than the number printed on each chapter, because some chapters are published as extensions of an earlier one (chuong-12-1 following chuong-12); on a novel like that, --start and --end count positions, so they can drift from the labels the site shows.

• Crawl one chapter or a chapter range from a supported novel URL.
• Automatically discover the latest available chapter when --end is omitted.
• Save chapters under a per-novel output directory as chapter_NNNN.html.
• Build an EPUB from newly crawled chapters or from an existing chapter folder.
• Run sequentially or with multiple download workers.
• Choose how to handle existing chapter files: ask, skip, or overwrite.
• Use an interactive TUI wizard with live download and EPUB build screens.
• Watch elapsed time and a running estimate on the download screen.
• Pick from recently used custom EPUB fonts, remembered between wizard runs.
• Open each EPUB chapter with a drop-capped first character.
• Build release binaries for Linux, Windows, macOS Intel, and macOS ARM through GitHub Actions.

• Rust stable toolchain
• Network access to fetch novel pages and cover images

cargo build --release

Run the binary from Cargo:

cargo run -- "https://metruyenhotvn.com/your-novel" --start 1 --end 10

Or run the compiled binary:

./target/release/novel-downloader "https://metruyenhotvn.com/your-novel" --start 1 --end 10

Output is written to output/<novel-slug>/chapter_NNNN.html by default.

Start the interactive wizard:

cargo run -- --interactive

Crawl chapters and build an EPUB:

cargo run -- "https://metruyenhotvn.com/your-novel" --start 1 --end 50 --epub

Let the crawler discover the latest chapter:

cargo run -- "https://metruyenhotvn.com/your-novel" --start 1 --epub

Use four parallel workers and skip files that already exist:

cargo run -- "https://metruyenhotvn.com/your-novel" --start 1 --end 100 --workers 4 --if-exists skip

Build an EPUB from an existing chapter directory:

cargo run -- "https://metruyenhotvn.com/your-novel" --epub-only --chapter-dir output/your_novel

Embed a custom font in the EPUB:

cargo run -- "https://metruyenhotvn.com/your-novel" --epub --font-path /path/to/font.ttf

novel-downloader [OPTIONS] [BASE_URL]

Options:
  --start <N>              Start chapter number, inclusive
  --end <N>                End chapter number, inclusive
  --output-root <DIR>      Root output directory [default: output]
  --delay <SECONDS>        Extra pause after each chapter is written [default: 0.5]
  --workers <N>            Number of concurrent download workers [default: 1]
  --epub                   Build an EPUB after crawling
  --epub-only              Build an EPUB from existing saved chapter files
  --chapter-dir <DIR>      Existing chapter directory for --epub-only
  --font-path <FILE>       Font file to embed instead of the bundled font
  --if-exists <POLICY>     ask, skip, or overwrite [default: ask]
  --fast-skip              Skip remote checks when the destination file already exists
  -i, --interactive        Launch the interactive TUI
  -h, --help               Show help
  -V, --version            Show version

--workers > 1 requires --if-exists skip or --if-exists overwrite, because interactive per-file prompts are only safe in the sequential path.

--delay is your own pause after each chapter is written, and it applies to every worker. Each source additionally enforces a minimum spacing between requests that --delay cannot lower, and may cap --workers; the run tells you when that happens.

flowchart TD
    A["User runs novel-downloader"] --> B["cli parses arguments"]
    B --> C{"interactive or missing URL?"}
    C -->|yes| D["ui wizard collects plan"]
    C -->|no| E["build non-interactive plan"]
    D --> F["registry resolves the host to a SiteAdapter"]
    E --> F
    F --> G["adapter fetches metadata and the chapter index"]
    G --> H["selected chapter refs"]
    H --> I{"mode"}
    I -->|crawl| J["runner walks the chapter refs"]
    I -->|crawl + epub| J
    I -->|epub only| N["use existing chapter directory"]
    J --> K{"workers"}
    K -->|1| L["sequential runner"]
    K -->|many| M["parallel runner"]
    L --> O["adapter fetches one chapter"]
    M --> O
    O --> Q["write chapter_NNNN.html"]
    Q --> R{"build EPUB?"}
    N --> R
    R -->|no| S["finish"]
    R -->|yes| T["epub reads saved chapters"]
    T --> U["download the cover, embed the font"]
    U --> V["package EPUB zip"]
    V --> S

• src/cli.rs: argument parsing and option validation.
• src/bin/novel-downloader.rs: process entry point and top-level orchestration.
• src/source/: the site seam. The SiteAdapter trait, the shared Novel, ChapterRef, RatePolicy and SourceError types, the host registry, and one module per site (metruyenhot scrapes HTML, khodocsach reads a JSON API, xtruyen scrapes HTML and decodes an encoded chapter payload).
• src/crawler/: the on-disk chapter document format, plus the fetch-write-skip flow and its existing-file policy.
• src/runner.rs: sequential and parallel chapter runners with progress events, concurrency clamping, and rate-policy pacing.
• src/epub/: saved-chapter reading, cover handling, EPUB XML/XHTML generation, and archive writing.
• src/ui/: interactive TUI widgets, screens, wizard state, and plan summary.
• src/utils.rs: shared HTTP, filesystem, text-cleaning, and slug helpers.
• src/font.rs: embedded font metadata extraction.
• src/recent_fonts.rs: the recently used custom EPUB fonts remembered for the wizard.

Run the full test suite:

cargo test

Run a focused test target:

cargo test --test epub

GitHub Actions are configured for:

• PR and main branch CI: run tests and build all supported platforms.
• Tag releases: when a v* tag is pushed, build and upload release artifacts for:
  • Linux x86_64
  • Windows x86_64
  • macOS Intel
  • macOS ARM

Create a release by pushing a version tag:

git tag v0.1.0
git push origin v0.1.0