Release 3.18.0 · internetarchive/heritrix3

Web UI: Added protection against cross-site request forgery (CSRF). POST, PUT and DELETE requests that a
browser sends on behalf of another site are now rejected, based on the Sec-Fetch-Site header. To allow requests
from another site, list its origin in the heritrix.trustedOrigins system property, e.g.
-Dheritrix.trustedOrigins=https://dashboard.example.org.
Web UI: Template output is now HTML-escaped by default, fixing cases where values were included in pages without
escaping. #780

Paged log viewer: #780
- Added a search box for filtering log lines, using a small query language, e.g.
  status:4xx host:example.com -type:image size>1MB. Any log can be searched for text; crawl.log also supports
  fields such as status, size, duration, depth, hops, url, host, via, type and annot. Matches
  are highlighted, and a progress bar is shown while searching large logs.
- Added an "include checkpoint logs" link to view a log together with the generations of it rotated at each
  checkpoint, as if they were one file.
BotBlockDetector: Added detection of AWS WAF, Kasada, Nginx Lua Anti-DDoS and PerimeterX. BotBlockDetector is
now enabled in the default job profiles. #778

FetchHTTP: The payload digest of chunked responses is now calculated after transfer decoding, so it no longer
includes the chunk framing. #770
ExtractorJson: URIs nested inside JSON arrays are now extracted. #771
ExtractorJS: JavaScript Unicode code-point escapes (\u{...}) in extracted URLs are now decoded. #772
AMQPUrlReceiver: Improved parsing of URL messages that contain null values. #767
Web UI: The copy job and delete job dialogs now work in browsers without support for invoker commands, such as
Firefox ESR. #773
Paged log viewer: Rotated crawl.log files now get status code highlighting, and unusual status codes no longer
cause an error. #780

Startup: heritrix_dmesg.log is now only written when Heritrix is started in the background by bin/heritrix,
making it possible to run with a read-only $HERITRIX_HOME. The path can be set with the heritrix.dmesg system
property. #779

Navigation Menu