Help us test StormCrawler 4.0 before the release

[rzo1]

Hi StormCrawlers!

@jnioche suggested this on the dev@ list, and we're posting it here as well to reach a broader audience.

We're close to releasing StormCrawler 4.0. The last remaining items, removing the DOM from the parse filters (#2262) and dropping HttpClient from core (#2211), are now done.

Before we cut the release, it would be great if you could run crawls with the code on the main branch, in particular with the OpenSearch module, which has seen some serious refactoring.

We've made a lot of changes since the last release, and since this is a major release, some of them are quite radical. We can always patch issues afterwards, but it's better to catch as much as possible now.

Running the new SpoutBenchmark and StatusUpdaterBenchmark against different backends is also a good way to test things.

The URLFrontier module has changed as well. If you want to benchmark that backend, build URLFrontier from source, as some recent performance improvements haven't been released yet.

Making results comparable

To be able to compare results between setups, please seed your crawls with the same list of URLs: the Tranco top 1M list with the ID GQ32K (or the first N entries of it), so everyone crawls the same hosts.

When you post results here, please include:

• the commit of main you built from
• the backend and its version (OpenSearch, Solr, URLFrontier, ...)
• your docker-compose.yml, or a description of how the services run
• hardware: CPU (model and cores), RAM, disk type, and network if relevant
• Java version and the worker JVM options
• the relevant parts of your crawler-conf.yaml (threads, topology.max.spout.pending, batching settings, ...)
• what you measured (pages fetched per second, benchmark output, ...) and over how long
• anything that broke or looked odd, with logs if possible

Happy crawling, and thanks for helping!

[jnioche]

The repo stormcrawler-benchmark contains resources and instructions to run the benchmarks with results for OpenSearch and URLFrontier. This can be used as a starting point to try different configurations and / or extend to backends not covered in the tests like SOLR and SQL.

[rzo1]

I ran a couple of short crawls of main at 2aa44dc (2aa44dc) with the WARC setup from https://github.com/rzo1/sc-warc-demo (branch sc-4.0-testing). No long run, I'm on wifi only.

Setup

• main at 2aa44dc, built from source
• Storm 3.1.0 (official storm:3.1.0 images: zookeeper, nimbus, supervisor, ui), URLFrontier 2.6 (crawlercommons/url-frontier:2.6, default -Xms2G -Xmx2G), see the docker-compose.yml in the branch
• Topology: URLFrontier spout -> partitioner -> fetcher -> sitemap -> feed -> JSoup parser -> WARCHdfsBolt, plus the status updater. Parallelism 1 everywhere.
• Apple M5 Max (18 cores), 128 GB RAM, internal SSD, Docker on macOS, Wi-Fi
• Java 25 (25.0.4.1 in the Storm image), 1 worker with -Xmx2g -Djava.net.preferIPv4Stack=true -Djdk.xml.maxGeneralEntitySizeLimit=0 -Djdk.xml.totalEntitySizeLimit=0
• fetcher.threads.number: 50, topology.max.spout.pending: 100, urlfrontier.max.buckets: 50, urlfrontier.max.urls.per.bucket: 10, http.content.limit: 65536, fetcher.thread.timeout: 60, sitemap.discovery: true, http.allow.redirects: true, okhttp protocol
• Seeds: first 1000 hosts of Tranco GQ32K

Results

Run	Duration	Fetched	Pages/s
1	7 min	8,720	~21 avg, 30 peak
2	12 min	17,483	25-28 after ramp-up, 30 peak

Run 2: 14,339 x 200, 1,901 redirects, 428 x 403, 419 x 404, 140 x 429, 102 fetch timeouts, ~690 MB fetched, no ERROR in the worker log. WARC files rotate at 50 MB and are valid after killing the topology. The first minutes are slower (~9-15 pages/s) while robots.txt is fetched for the 1000 hosts. On wifi the network is likely the limit, so these numbers say more about stability than throughput.

Found

• SitemapFilter drops redirect targets on sites that declare a sitemap in robots.txt #2282 SitemapFilter drops redirect targets on hosts that declare a sitemap in robots.txt. Run 2 logged ~2,000 "Redirect target ... rejected by the URL filters" (not all of them are this case, self-redirects are rejected too). The bolt-level part is already in 3.7.0; with http.allow.redirects: true it is new since Run redirect hops through the URL filters #2128. Fix: #2282 Don't apply SitemapFilter to redirect targets #2285
• Archetype: use URLFrontier 2.6 #2283 the archetype still used URLFrontier 2.4
• WARC: log replaced request lines at debug level #2284 WARCRequestRecordFormat logs every HTTP/2 request line at INFO
• The security page on the website still describes the 3.x defaults (http.trust.everything on, file in protocols, no IP filter). main is stricter now, so it should be updated with the release.
• The archetype sets the jdk.xml.* limits to 0 (unlimited). Maybe a large finite value would be better.

Not StormCrawler, but maybe useful for others: the WARC flux needs withConfigKey("warc") on the bolt, otherwise the RawLocalFileSystem config isn't used and the WARC file is truncated when the topology is killed. Also, URLFrontier 2.6 keeps its state in RocksDB inside the container, so docker restart doesn't give you an empty frontier.

I'll re-run this once the fixes (and Tika 4.2.0) are in, this time on ethernet.