stormcrawler-3.7.0

Summary

Apache StormCrawler 3.7.0 introduces a queue stream for the URLFrontier module (#1974, #1981) which allows the crawler to back off from rate-limited hosts (#1984), hold queues on Retry-After (#1973) and enforce robots.txt Crawl-delay directives (#2000). The Selenium protocol (#2019), the CloudSearch indexer (#2018) and the Apache HttpClient-based protocol (#2020) are deprecated and will be removed in the next major release.

This release also adds a pluggable robots.txt rules parser (#1987), per-URL proxy selection via metadata (#1935), an IP address filter for the OkHttp protocol (#1942), support for using a metadata value as document ID in indexer bolts (#2012), gzip/bz2 input for FileSpout (#1952), and resolution of relative outlinks against <base href> (#2011). The AWS module was migrated to the AWS SDK v2 (#1939).

Additionally, the project was upgraded to Apache Storm 2.8.9 (#2032) and multiple dependencies were updated. This is the final release of the 3.x line, as Apache Storm 2.x has reached end of life. The journey continues with StormCrawler 4.0, where StormCrawler meets Apache Storm 3.

What's Changed

• StormCrawler 3.6.0 by @rzo1 in #1919
• Bump langchain4j.version from 1.15.0 to 1.15.1 by @dependabot[bot] in #1922
• Bump tika.version from 3.3.0 to 3.3.1 by @dependabot[bot] in #1920
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.5.5 to 3.5.6 by @dependabot[bot] in #1921
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1924
• Bump actions/checkout from 6.0.2 to 6.0.3 by @dependabot[bot] in #1927
• Fix missing space between block and inline sibling elements in JSoupTextExtractor by @nicoscandolo in #1926
• Add AGENTS.md + SECURITY.md pointing to the security model (scanner discoverability) by @potiuk in #1932
• Improve fault tolerance: handle parse errors, fix logging, and minor cleanup by @dpol1 in #1931
• Fix thread safety: replace shared Matcher and SimpleDateFormat instances by @dpol1 in #1929
• Fix resource leaks: use try-with-resources and add cleanup hooks by @dpol1 in #1930
• Bump org.jacoco:jacoco-maven-plugin from 0.8.14 to 0.8.15 by @dependabot[bot] in #1934
• Bump langchain4j.version from 1.15.1 to 1.16.1 by @dependabot[bot] in #1933
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1937
• 1143 Add per-URL proxy metadata selection by @preko-p in #1935
• docs: fix StormCrawler documentation links by @SSDWGG in #1940
• #1938 - Migrate AWS Java SDK from 1.x to 2.x (#1938) by @rzo1 in #1939
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1941
• Bump software.amazon.awssdk:bom from 2.46.7 to 2.46.10 by @dependabot[bot] in #1947
• Bump langchain4j.version from 1.16.1 to 1.16.2 by @dependabot[bot] in #1945
• Bump okhttp.version from 5.3.2 to 5.4.0 by @dependabot[bot] in #1946
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1949
• Bump org.opensearch.client:opensearch-java from 3.8.0 to 3.9.0 by @dependabot[bot] in #1948
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1950
• FileSpout: add support for gzip and bz2 files by @jnioche in #1952
• #1107 - Add okhttp IP address filter by @rzo1 in #1942
• Bump actions/setup-java from 5.2.0 to 5.3.0 by @dependabot[bot] in #1957
• #1951 refactor: replace linear proxy scan with lookup map in MultiProxyManager by @leedongkyu0407 in #1954
• #1955 - Make FileSpout work in distributed mode by @dpol1 in #1956
• build(deps): bump actions/checkout from 6.0.3 to 7.0.0 by @dependabot[bot] in #1958
• build(deps): bump langchain4j.version from 1.16.2 to 1.16.3 by @dependabot[bot] in #1959
• build(deps): bump software.amazon.awssdk:bom from 2.46.10 to 2.46.15 by @dependabot[bot] in #1961
• build(deps): bump selenium.version from 4.44.0 to 4.45.0 by @dependabot[bot] in #1960
• #466 - Handle text/plain content in JSoupParserBolt by @rzo1 in #1943
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1962
• use placeholder instead of older version of archetypes in README by @jnioche in #1963
• build(deps): bump actions/cache from 5.0.5 to 6.0.0 by @dependabot[bot] in #1964
• build(deps): bump actions/setup-java from 5.3.0 to 5.4.0 by @dependabot[bot] in #1965
• build(deps): bump software.amazon.awssdk:bom from 2.46.15 to 2.46.17 by @dependabot[bot] in #1967
• build(deps): bump junit.version from 6.1.0 to 6.1.1 by @dependabot[bot] in #1966
• build(deps): bump actions/cache from 6.0.0 to 6.1.0 by @dependabot[bot] in #1970
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1971
• build(deps): bump org.apache:apache from 38 to 39 by @dependabot[bot] in #1969
• build(deps): bump langchain4j.version from 1.16.3 to 1.17.0 by @dependabot[bot] in #1968
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1972
• Queue stream by @jnioche in #1974
• Block the URLFrontier queue on Retry-After via the queue stream by @dpol1 in #1973
• build(deps): bump software.amazon.awssdk:bom from 2.46.17 to 2.46.21 by @dependabot[bot] in #1978
• build(deps): bump com.microsoft.playwright:playwright from 1.60.0 to 1.61.0 by @dependabot[bot] in #1977
• build(deps): bump opensearch.version from 2.19.5 to 2.19.6 by @dependabot[bot] in #1976
• build(deps): bump langchain4j.version from 1.17.0 to 1.17.1 by @dependabot[bot] in #1975
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1980
• Port QueueBolt (queue stream) to the opensearch-java module by @rzo1 in #1981
• build(deps): bump actions/setup-java from 5.4.0 to 5.5.0 by @dependabot[bot] in #1983
• Adding Green Web Foundation to Powered By by @jnioche in #1985
• Don't trigger mvn workflow for changes to the docs by @jnioche in #1986
• Pluggable robot rule parser by @jnioche in #1987
• Adaptive back-off for rate-limited hosts via the queue stream by @dpol1 in #1984
• Workaround to activate LogViewer in docker compose by @jnioche in #1992
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1993
• Verbatim headers for Playwright by @jnioche in #1996
• build(deps): bump langchain4j.version from 1.17.1 to 1.17.2 by @dependabot[bot] in #1988
• build(deps): bump junit.version from 6.1.1 to 6.1.2 by @dependabot[bot] in #1989
• build(deps): bump selenium.version from 4.45.0 to 4.46.0 by @dependabot[bot] in #1990
• build(deps): bump software.amazon.awssdk:bom from 2.46.21 to 2.47.5 by @dependabot[bot] in #1991
• Regenerated License file after dependency upgrades by @github-actions[bot] in #1999
• [WARC] Format WARC-IP-Address header values following the WARC community recommendations by @sebastian-nagel in #1997
• #2001 Document adaptive scheduler metadata by @Yurii201811 in #2002
• bugfix - max content not applied for robots.txt when redirecting by @jnioche in #2004
• build(deps): bump actions/setup-java from 5.5.0 to 5.6.0 by @dependabot[bot] in #2003
• build(deps): bump software.amazon.awssdk:bom from 2.47.5 to 2.48.3 by @dependabot[bot] in #2008
• build(deps): bump langchain4j.version from 1.17.2 to 1.18.0 by @dependabot[bot] in #2007
• Regenerated License file after dependency upgrades by @github-actions[bot] in #2009
• build(deps): bump actions/checkout from 7.0.0 to 7.0.1 by @dependabot[bot] in #2010
• Fix #304: Use HTML test document instead of sitemap XML in SubDocumentsFilterTest by @AsifQureshi04 in #2005
• Resolve relative outlinks against in JSoupParserBolt by @jnioche in #2011
• Mark Selenium module as deprecated by @jnioche in #2019
• Mark CloudSearch as deprecated by @jnioche in #2018
• Allow indexer bolts to use a metadata value as the document ID by @sigee in #2012
• Mark HTTPClient as deprecated by @jnioche in #2020
• build(deps): bump org.apache.maven.plugins:maven-jar-plugin from 3.5.0 to 3.5.1 by @dependabot[bot] in #2025
• build(deps): bump tika.version from 3.3.1 to 3.3.2 by @dependabot[bot] in #2023
• build(deps): bump software.amazon.awssdk:bom from 2.48.3 to 2.49.3 by @dependabot[bot] in #2022
• Regenerated License file after dependency upgrades by @github-actions[bot] in #2028
• Regenerated License file after dependency upgrades by @github-actions[bot] in #2029
• Regenerated License file after dependency upgrades by @github-actions[bot] in #2030
• #2026 - http.agent.name is dropped from the robots.txt match set when http.robots.agents is set by @rzo1 in #2027
• #2021 Fix URLFilterBolt to emit anchored tuples by @lfoppiano in #2031
• Enforce robots.txt Crawl-delay in URLFrontier queues by @dpol1 in #2000
• Upgrade to Storm 2.8.9 by @dpol1 in #2032
• Regenerated License file after dependency upgrades by @github-actions[bot] in #2033

New Contributors

• @nicoscandolo made their first contribution in #1926
• @potiuk made their first contribution in #1932
• @preko-p made their first contribution in #1935
• @SSDWGG made their first contribution in #1940
• @leedongkyu0407 made their first contribution in #1954
• @Yurii201811 made their first contribution in #2002
• @AsifQureshi04 made their first contribution in #2005
• @lfoppiano made their first contribution in #2031

Full Changelog: stormcrawler-3.6.0...stormcrawler-3.7.0