2.6

URL Frontier 2.6

19 August 2026 — 125 commits since 2.5 (23 October 2025).

This release is about ingestion throughput and correctness under concurrency. The API gains a batched endpoint for discovered URLs and a way of purging old ones; the distributed mode now behaves like a cluster instead of a set of independent nodes; and a long series of concurrency bugs in both backends has been fixed.

Breaking changes

• Java 17 is now the minimum version for building and running the Frontier (#139). 2.5 required Java 11.
• The control RPCs act on the whole cluster in distributed mode. SetActive, GetActive, SetDelay, BlockQueueUntil and SetCrawlLimit used to apply only to the node receiving them; they are now broadcast, or routed to the node owning the queue. Set the local field on the request for the previous behaviour (#146, #149, #154).
• The server applies backpressure to PutURLs. A stream may have at most putURLs.max.inflight URLs (1024) received but not acked before the server stops reading from it, the excess staying on the wire where HTTP/2 flow control pushes back onto the sender. Set it to 0 for the historical behaviour of letting clients send at will.
• Stats.numberOfQueues counts the queues of the crawl whether they are active or not — the schema claimed "active" but no implementation ever did that. QueueList.total is now documented as not set by the reference implementation: counting the matching queues would mean scanning the whole frontier.

API additions

The schema stays wire-compatible with 2.5 apart from the semantics noted above.

• PutDiscovered(stream DiscoveredBatch) returns (stream BatchAck) — pushes discovered URLs in batches rather than one message per URL. The outlinks of a page form a natural batch, and since the per-message cost is what caps the ingestion rate, batching the discovered URLs — the bulk of what a crawl writes — is where the gain is. A batch is acked as a whole, with one status per URL in the order sent. Fetched URLs keep going through PutURLs.
• PurgeURLs(PurgeUrlParams) returns (Long) — deletes the URLs older than N days, optionally restricted to a queue or a crawl (#143).
• URLItem.creation_date — set by the service when a URL is first stored, returned by GetURLStatus and ListURLs, ignored when sent by a client. This is the date PurgeURLs compares its cutoff against (#143).
• CrawlLimitParams.local — restricts SetCrawlLimit to the node receiving it.

Performance

Writes. PutURLs now uses gRPC flow control instead of polling, with a bound on the URLs in flight per stream. Each PutDiscovered batch is written as a single RocksDB WriteBatch, and the existence of the batched URLs is checked before the locks are taken. A single WriteOptions instance is reused across writes, and strings are no longer interned in putURLItem (#140).

RocksDB tuning. Bloom filters are enabled by default, which spares the lookup done for every incoming URL from reading a data block when the URL is not yet known — the common case while crawling. They are kept out of the block cache, since they are not partitioned and caching them would mean re-reading several MB at a time. New options:

Option	Default	Effect
rocksdb.bloom.filters	true	set to false to disable the filters
rocksdb.bloom.filters.bits_per_key	10	~1.25 bytes per URL, false positive rate around 1%
rocksdb.wal.disable	false	roughly halves the bytes written per URL; whatever is in the memtables is lost if the service dies without closing cleanly, and those URLs simply get rediscovered by the crawl
rocksdb.memtable_memory_budget	RocksDB default	larger memtables mean fewer flushes and compactions, more memory and a longer recovery

Reads. GetStats iterates over the queues lazily instead of copying them, which used to run a large frontier out of memory. GetURLs picks its candidates by polling rather than sorting the whole queue. Streams are completed asynchronously instead of busy-waiting (#170).

Distributed mode

• SetActive is broadcast to every node, and GetActive aggregates across the cluster with a shared deadline: it reports true only if every node is active, and an unreachable node is an error rather than a false (#149).
• SetDelay and BlockQueueUntil are routed to the node owning the queue (#146), as is SetCrawlLimit (#154), which also sends the Empty response it used to omit (#153).
• GetURLs takes its per-queue eligibility decisions — blocked state, crawl limit, politeness delay, in-process cap — atomically, so concurrent clients can no longer be served the same queue inside its delay window. A send returning zero URLs restores the previous state, so an empty queue is not penalised (#147, #152).
• Forwarded blocking calls are bounded instead of being able to hang indefinitely (#174), and the shared stream forwarding putURLs to the owner node is now safe for concurrent clients (#69), closed rather than dropped (#207), and paced on the transport's readiness with the new forward.ready.timeout.ms (#208).
• The semantics are documented in service/README.md: what each control RPC does with and without local, the reservation guarantees, and the fact that control and lease state lives in memory and is lost on restart — after a restart, clients must re-assert the delays, blocks and limits they rely on.

Bug fixes

Pagination and listing. ListQueues counted its offset over the queue map rather than over the matching results, so on a frontier holding more than one crawl — or any inactive queue — paging silently skipped queues and returned others twice; it now counts only the entries that matched and echoes start, size and crawlID back. listQueues, listURLs and countURLs also iterated the insertion-ordered view that getURLs rotates, which could yield a queue twice or miss it entirely, and now use the rotation-immune view. ListQueues also returned maxQueues + 1 entries (#167), and a large start offset caused a StackOverflowError (#176).

RocksDB backend. The whole queueInfos table is cleared on start-up (#163). Deleting a queue now covers its entire key range including the last key, and no longer NPEs while logging a failed range deletion. The iterator is checked for validity before its key is read (#168), and a missing entry in deleteURLItem is treated as already deleted.

Memory backend. Per-queue locking replaces the queues-map monitor, the URL iterator snapshots the queue, queue rotation in getURLs is null-safe, creation dates are scoped to their queue (#166), the remaining queue-content readers take the lock (#194), and the code no longer relies on PriorityQueue.iterator() returning entries in sort order (#162).

Streams and lifecycle. PutURLs no longer hangs when the stream fails (#169), GetURLStatus no longer continues after onError (#164), queuesBeingDeleted uses an atomic claim (#173), GetStats no longer throws NoSuchElementException (#145), and the NPEs on the deleteQueue and deleteCrawl paths are gone (#165).

Configuration. addToConfig was fixed and a ParamHelper class introduced to centralise the retrieval and checking of parameters (#177); the sharded service is now passed the configuration it was missing; and the version is reported from the POM instead of being hardcoded.

Client

• DumpURLs exports every URL of a Frontier — queue key, crawl ID, metadata and scheduling information — as one JSON object per line, in the exact format PutURLs reads. This makes it possible to migrate the data to another backend or another major version. -t/--threads lists the queues first and dumps them in parallel. The dump covers only the node the client connects to, so in a cluster dump every node and concatenate; note that creation dates are not preserved across a dump / re-import cycle, which affects PurgeURLs.

  java -jar urlfrontier-client-2.6.jar DumpURLs -t 8 -o frontier.jsonl.gz
  java -jar urlfrontier-client-2.6.jar -t newhost PutURLs -f frontier.jsonl.gz
  

• PurgeURLs exposes the new endpoint from the command line.

• PutURLs gained three options that matter for a large injection: -t/--threads sends on several streams at once, -b/--batch groups that many discovered URLs into one PutDiscovered message (falling back to individual sends against a server which does not implement it), and -w/--in-flight caps the URLs sent but not confirmed per thread (10000 by default). It also reads its input line by line so it can handle large files, decompresses .gz input on the fly, and reports the OPS every 30 seconds while it runs.

• SetCrawlLimit and GetURLStatus, which already existed, are finally listed in the README.

Documentation

service/README.md now carries a full table of the configuration parameters — including which ones are flag-style and switched on by the mere presence of the key — plus the distributed mode, restart and deployment sections. client/README.md documents the injection options and the export / re-import workflow. A number of comments in urlfrontier.proto were brought back in line with what the implementation actually does, among them GetParams.delay_requestable, whose default of 0 does not mean "no limit" but "let the service pick" — 30 seconds in the reference implementation.

Build and CI

Java 17 baseline with Maven plugins and test dependencies updated (#139); a duplicate central-publishing-maven-plugin entry removed from the parent POM (#138); artifacts published to Sonatype Central by a workflow triggered on release; the Docker image published when a release is published rather than on a tag push; and the stale GitHub Actions updated to their current majors.

The multi-arch Docker build was fixed (#215): ForwardDeadlineTest closed a listening ServerSocket out from under a thread parked in a native accept(), and the JDK's pthread_kill wakeup is not delivered reliably under QEMU user-mode emulation, so the linux/arm64 leg failed in teardown even though every test passed. The close is now best-effort like the ones next to it. The build stage is also pinned to $BUILDPLATFORM — the shaded jar is platform-independent, since neither rocksdbjni nor grpc-netty-shaded uses native classifiers — so the suite runs once on the runner instead of a second time under emulation. The runtime stage stays per-target and the published manifest is still multi-arch.

The release procedure is written down in RELEASE.md.

Contributors

Julien Nioche, Davide Polato and Laurent Klock.

---

Full changelog: 2.5...2.6