Context
The engine gained several user-facing crawl options between 3.49-3 and 3.49-12 that
the GUIs were frozen before. The Windows GUI now exposes them in
xroche/httrack-windows#31 (green, reviewed). webhttrack (the web GUI, in this repo under
src/webhttrack + html/webhttrack) should expose the same set so the front-ends stay in
parity. This issue carries everything needed to do that independently.
Options to expose
Each is emitted to the engine as the CLI flag shown. Short forms verified in
src/htsalias.c; long forms are the documented public names.
| Feature |
CLI (long / short) |
Kind |
Semantics |
| SOCKS5 proxy |
-P socks5://[user:pass@]host[:port] |
proxy string |
Scheme prefixed onto the single -P. socks5:// and socks5h:// are equivalent (remote DNS). Default port 1080 (vs 8080 for HTTP). HTTP-proxy CONNECT for HTTPS and SOCKS tunneling are automatic — no extra flag. Reuse the existing single proxy field; only a protocol selector is new. |
| Preload cookies |
--cookies-file <path> / -%K |
value |
Load a Netscape cookies.txt before crawling. |
| Random pause |
--pause MIN[:MAX] / -%G |
value (seconds) |
Random delay between file downloads. Distinct from the old -G (pause after N bytes). |
| Strip query keys |
--strip-query <[host/pattern=]k1,k2,…> / -%g |
value |
Drop selected query keys from the dedup naming. Simple form sid,utm_source; scoped form www.example.com/*=sid,ref. |
| Keep www. prefix |
--keep-www-prefix / -%j |
toggle |
Opt out of one part of the -%u URL hack (don't merge www.host with host). |
| Keep double slashes |
--keep-double-slashes / -%o |
toggle |
Opt out of -%u (keep // in paths). |
| Keep query order |
--keep-query-order / -%y |
toggle |
Opt out of -%u (don't reorder query params). |
The three keep-* toggles only matter when -%u URL-hacks is on. Value options take a
separate argument; toggles are bare flags.
Stale default to check
The engine (3.49-11) now defaults the User-Agent to the honest
Mozilla/5.0 (compatible; HTTrack/<ver>; +https://www.httrack.com/). Make sure webhttrack
doesn't hardcode the old Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98) and override
it. Also sanity-check the default max transfer rate (engine default 100000 B/s, cap
10000000 B/s since 3.49-4).
No UI needed (automatic engine features)
deflate Accept-Encoding, robots Allow/path wildcards (RFC 9309), <source>/<track>,
full srcset, WHATWG entities, modern MIME types, HTTPS-over-HTTP-proxy CONNECT, SOCKS
tunneling. These already work once the engine is current.
Deferred
--why / -%Y (prints which filter rule accepts/rejects a URL, then exits) is a
diagnose-and-exit mode, not a mirror setting — leave it out or make it a "test a URL"
action, matching the Windows decision.
Commit-by-commit, mirrors what webhttrack needs:
- Modernize the default User-Agent.
- SOCKS5 — a proxy-type (HTTP/SOCKS5) selector that prepends
socks5:// to -P, reusing the single proxy field.
--cookies-file field.
--pause field (seconds, MIN:MAX).
- Three
--keep-* checkboxes.
--strip-query field.
- Polish: file-picker for the cookies path; clearer labels.
i18n
webhttrack uses the engine's own string system; new UI labels/tooltips need their strings
added the webhttrack way (not the Windows cpp_lang.h). Keep the wording aligned with the
Windows labels for consistency.
Context
The engine gained several user-facing crawl options between 3.49-3 and 3.49-12 that
the GUIs were frozen before. The Windows GUI now exposes them in
xroche/httrack-windows#31 (green, reviewed). webhttrack (the web GUI, in this repo under
src/webhttrack+html/webhttrack) should expose the same set so the front-ends stay inparity. This issue carries everything needed to do that independently.
Options to expose
Each is emitted to the engine as the CLI flag shown. Short forms verified in
src/htsalias.c; long forms are the documented public names.-P socks5://[user:pass@]host[:port]-P.socks5://andsocks5h://are equivalent (remote DNS). Default port 1080 (vs 8080 for HTTP). HTTP-proxy CONNECT for HTTPS and SOCKS tunneling are automatic — no extra flag. Reuse the existing single proxy field; only a protocol selector is new.--cookies-file <path>/-%K--pause MIN[:MAX]/-%G-G(pause after N bytes).--strip-query <[host/pattern=]k1,k2,…>/-%gsid,utm_source; scoped formwww.example.com/*=sid,ref.--keep-www-prefix/-%j-%uURL hack (don't merge www.host with host).--keep-double-slashes/-%o-%u(keep//in paths).--keep-query-order/-%y-%u(don't reorder query params).The three
keep-*toggles only matter when-%uURL-hacks is on. Value options take aseparate argument; toggles are bare flags.
Stale default to check
The engine (3.49-11) now defaults the User-Agent to the honest
Mozilla/5.0 (compatible; HTTrack/<ver>; +https://www.httrack.com/). Make sure webhttrackdoesn't hardcode the old
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)and overrideit. Also sanity-check the default max transfer rate (engine default 100000 B/s, cap
10000000 B/s since 3.49-4).
No UI needed (automatic engine features)
deflate
Accept-Encoding, robotsAllow/path wildcards (RFC 9309),<source>/<track>,full
srcset, WHATWG entities, modern MIME types, HTTPS-over-HTTP-proxy CONNECT, SOCKStunneling. These already work once the engine is current.
Deferred
--why/-%Y(prints which filter rule accepts/rejects a URL, then exits) is adiagnose-and-exit mode, not a mirror setting — leave it out or make it a "test a URL"
action, matching the Windows decision.
Reference implementation (WinHTTrack, xroche/httrack-windows#31)
Commit-by-commit, mirrors what webhttrack needs:
socks5://to-P, reusing the single proxy field.--cookies-filefield.--pausefield (seconds,MIN:MAX).--keep-*checkboxes.--strip-queryfield.i18n
webhttrack uses the engine's own string system; new UI labels/tooltips need their strings
added the webhttrack way (not the Windows
cpp_lang.h). Keep the wording aligned with theWindows labels for consistency.