Skip to content

Expose post-3.49-2 engine options in webhttrack (SOCKS5, cookies-file, pause, keep-*, strip-query, UA) #587

Description

@xroche

Context

The engine gained several user-facing crawl options between 3.49-3 and 3.49-12 that
the GUIs were frozen before. The Windows GUI now exposes them in
xroche/httrack-windows#31 (green, reviewed). webhttrack (the web GUI, in this repo under
src/webhttrack + html/webhttrack) should expose the same set so the front-ends stay in
parity. This issue carries everything needed to do that independently.

Options to expose

Each is emitted to the engine as the CLI flag shown. Short forms verified in
src/htsalias.c; long forms are the documented public names.

Feature CLI (long / short) Kind Semantics
SOCKS5 proxy -P socks5://[user:pass@]host[:port] proxy string Scheme prefixed onto the single -P. socks5:// and socks5h:// are equivalent (remote DNS). Default port 1080 (vs 8080 for HTTP). HTTP-proxy CONNECT for HTTPS and SOCKS tunneling are automatic — no extra flag. Reuse the existing single proxy field; only a protocol selector is new.
Preload cookies --cookies-file <path> / -%K value Load a Netscape cookies.txt before crawling.
Random pause --pause MIN[:MAX] / -%G value (seconds) Random delay between file downloads. Distinct from the old -G (pause after N bytes).
Strip query keys --strip-query <[host/pattern=]k1,k2,…> / -%g value Drop selected query keys from the dedup naming. Simple form sid,utm_source; scoped form www.example.com/*=sid,ref.
Keep www. prefix --keep-www-prefix / -%j toggle Opt out of one part of the -%u URL hack (don't merge www.host with host).
Keep double slashes --keep-double-slashes / -%o toggle Opt out of -%u (keep // in paths).
Keep query order --keep-query-order / -%y toggle Opt out of -%u (don't reorder query params).

The three keep-* toggles only matter when -%u URL-hacks is on. Value options take a
separate argument; toggles are bare flags.

Stale default to check

The engine (3.49-11) now defaults the User-Agent to the honest
Mozilla/5.0 (compatible; HTTrack/<ver>; +https://www.httrack.com/). Make sure webhttrack
doesn't hardcode the old Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98) and override
it. Also sanity-check the default max transfer rate (engine default 100000 B/s, cap
10000000 B/s since 3.49-4).

No UI needed (automatic engine features)

deflate Accept-Encoding, robots Allow/path wildcards (RFC 9309), <source>/<track>,
full srcset, WHATWG entities, modern MIME types, HTTPS-over-HTTP-proxy CONNECT, SOCKS
tunneling. These already work once the engine is current.

Deferred

--why / -%Y (prints which filter rule accepts/rejects a URL, then exits) is a
diagnose-and-exit mode, not a mirror setting — leave it out or make it a "test a URL"
action, matching the Windows decision.

Reference implementation (WinHTTrack, xroche/httrack-windows#31)

Commit-by-commit, mirrors what webhttrack needs:

  1. Modernize the default User-Agent.
  2. SOCKS5 — a proxy-type (HTTP/SOCKS5) selector that prepends socks5:// to -P, reusing the single proxy field.
  3. --cookies-file field.
  4. --pause field (seconds, MIN:MAX).
  5. Three --keep-* checkboxes.
  6. --strip-query field.
  7. Polish: file-picker for the cookies path; clearer labels.

i18n

webhttrack uses the engine's own string system; new UI labels/tooltips need their strings
added the webhttrack way (not the Windows cpp_lang.h). Keep the wording aligned with the
Windows labels for consistency.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions