Repository navigation
Download distribution zip (or tar.gz)
Full Changelog | Javadoc | Maven Central
Security
- BrowserProcessor: The recording proxy now listens on the loopback interface only and rejects requests that
don't belong to a page the crawler has open. Previously anyone who could reach the crawler's host could use it as
an open forward proxy. #816
New features
- BrowserProcessor:
- Added support for running several browsers, listed in the new
browsersproperty. Pages from the same site are
visited in the same browser where possible. Browsers are launched in parallel, the crawl starts once
minBrowsersof them are up, failed launches are retried with backoff, and a browser that crashes or hangs is
restarted and its page retried once. The processor report now shows the status of each browser. Note that
concurrencynow limits open pages per browser rather than in total. #817 - Browsers can be launched on other machines over SSH, e.g.
ssh://user@browser2.example.org. The SSH connection
carries both the WebDriver BiDi connection and the browser's traffic to the recording proxy. UseLocalBrowser
orSshBrowserbeans to setconcurrency,executableand options per browser. #819 - Browser-based crawling is now documented, and the Groovy default profile includes a commented-out
BrowserProcessor bean. #820
- Added support for running several browsers, listed in the new
- ExtractorSVG: New extractor for links in SVG images, including
<image>,<use>,<script>,<a>,
<?xml-stylesheet?>,url()in presentation attributes and<style>, and HTML inside<foreignObject>. Added to
the default profiles. #812 - Soft404Detector: New processor that probes each server for a random path and annotates later responses
matching the probe's content assoft404. WithskipExtractionenabled the rest of the fetch chain is skipped for
them, preventing endless loops of fake URLs on sites that return 200 for nonexistent pages. Included but commented
out in the default profiles. #814 - ExtractorHTML: Now extracts
<video poster>and<link imagesrcset>as embeds,formactionon buttons and
inputs (subject toignoreFormActionUrlsandextractOnlyFormGets), and links inside<iframe srcdoc>.
Hrefs on inline SVG elements like<image>and<use>are now embeds. #804 #812 - ExtractorJS: ES module imports (
import ... from "x",export ... from "x",import("x")) are now resolved
relative to the script's own URL rather than the page that included it, and extracted as embeds. #808 - BotBlockDetector: Added detection of Azure WAF and Vercel. #784
- AMQPPublishProcessor: Added a
persistentMessagesoption to publish messages as persistent. #781
Bug fixes
- ToeThread: A thread that died while finishing a URI could keep holding the disposition lock, stalling the whole
crawl at the next checkpoint. #783 - StatisticsTracker: A URI with a host like
0.0.0.0no longer kills a ToeThread, which left the job stuck in
EMPTY. #821 - FetchHTTP: Responses rejected by
shouldFetchBodyRulenow get the new fetch statusS_MIDFETCH_ABORTED
(-5005) instead of being counted as successful fetches in crawl statistics and reports. #792 - FetchHTTP2: An
Alt-Svcheader without anmaparameter now defaults to a 24 hour max age, so HTTP/3 is
actually used for such servers. #803 - ExtractorHTML:
<link rel=preload>is now treated as an embed andrel=prefetchandrel=prerenderas
navlinks. When a page links to the same URL both as an embed and otherwise, the embed is what gets scheduled.
#825- A
>inside a quoted attribute value no longer ends the tag, and JerichoExtractorHTML no longer
treats markup inside attribute values as tags. #804
- ExtractorCSS: Same-document references like
url(#gradient)are now ignored. #812 - AMQPUrlReceiver / AMQPProducer: Improved recovery from dropped connections and broker-side consumer cancels,
added a liveness check, and stopped leaking connections and channels. The consumer prefetch is now 100.
#781 - SourceTags report: Fixed slow job teardown when
includeResCodeis enabled. #788 - Web UI: Job directory files are no longer cached by the browser for 10 minutes, which could cause the config
editor to show and save over a stalecrawler-beans.cxml. #782 - BrowserProcessor: Firefox site isolation is turned off, as it was killing tabs mid-load, and a crashed tab is no
longer mistaken for a download. #819
Documentation
- Added 24 missing processors to the bean reference. #815
- Fixed factual errors in the user documentation. #787
Dependency upgrades
- amqp-client: 5.36.0 → 5.37.1
- codemirror__autocomplete: 6.20.3 → 6.20.4
- codemirror__commands: 6.11.0 → 6.11.1
- codemirror__language: 6.12.4 → 6.13.1
- codemirror__legacy-modes: 6.5.1 → 6.5.5
- codemirror__search: 6.5.11 → 6.7.2
- codemirror__state: 6.7.4 → 6.7.6
- codemirror__streamparser: 6.0.0 (new)
- codemirror__view: 6.43.11 → 6.43.15
- commons-codec: 1.22.1 → 1.23.0
- dnsjava: 3.6.5 → 3.7.1
- gson: 2.14.0 → 2.15.0
- jackson-bom: 2.22.3 → 2.23.0
- jetty (jetty-bom, jetty-ee10-bom): 12.0.39 → 12.0.41
- jsch: 2.28.7 → 2.28.8
- json: 20250517 → 20260814
- kafka-clients: 4.3.1 → 4.4.0
- kryo: 5.6.2 → 5.7.1
- lezer__common: 1.5.2 → 1.5.5
- lezer__highlight: 1.2.3 → 1.2.5
- lezer__lr: 1.4.10 → 1.4.12
- lz4-java: 1.12.0 → 1.14.1
- mina-core: 2.2.9 → 2.2.10
- restlet: 2.6.0 → 2.6.1
- snappy-java: 1.1.10.7 → 1.1.10.12
- spring: 7.0.9 → 7.0.10
- style-mod: 4.1.3 → 4.1.4
- webarchive-commons: 3.0.5 → 3.0.6