Skip to content

v0.6.0.0

Latest
Latest

Choose a tag to compare

View all tags
@astubbs astubbs released this 11 Sep 21:51
· 6 commits to master since this release
Immutable release. Only release title and notes can be modified.
v0.6.0.0
8730ff0

First release of the community fork of confluentinc/parallel-consumer, which is no longer maintained. Published to Maven Central as bz.stub.parallelconsumer.

Maven Central coordinates. The groupId is new with the fork, so this is the one release where they belong at the top:

<dependency>
    <groupId>bz.stub.parallelconsumer</groupId>
    <artifactId>parallel-consumer-core</artifactId>
    <version>0.6.0.0</version>
</dependency>

The Vert.x, Reactor and Mutiny modules are parallel-consumer-vertx, parallel-consumer-reactor and parallel-consumer-mutiny under the same groupId and version. The Java package is bz.stub.parallelconsumer too, so imports change with it - one command rewrites them (GNU sed; on macOS use sed -i ''):

find . -name '*.java' -exec sed -i 's/io\.confluent\.parallelconsumer/bz.stub.parallelconsumer/g' {} +
  • Upgrading from upstream 0.5.x means changing the Maven groupId and the package your imports name. The Breaking section below is short; read it once.
  • Committed offsets are unchanged. An existing consumer group upgrades in place, with no reset and no migration.
  • Everything upstream merged after its last published release ships here for the first time. Upstream's last release on Maven Central is 0.5.3.2; its 0.5.3.3 section below was tagged but never published.

This is a stability release, and that is the point. The change set is the largest this codebase has shipped in one version, and almost all of it is fixes:

  • the commit-path deadlock behind the long-standing "consumption stops after a rebalance" reports
  • a family of torn reads
  • the metrics leak, offset accuracy on assignment, and an asynchronous commit recorded before the broker answered
  • the test lanes that now guard each of them

The bar was every known critical defect resolved, each with a named regression guard that passes. Every fix was reproduced deterministically and shown to fail before and pass after; the write-up behind each is in its linked PR. Two known critical defects sit outside that bar, both in the transactional producer mode only, and are named under Known limitations. New capabilities are queued behind this release; What comes next lists them by how far each has got.

Breaking

  • Coordinates and packages (#55, #294): the Maven groupId io.confluent.parallelconsumer is now bz.stub.parallelconsumer, and the Java packages io.confluent.parallelconsumer.* are now bz.stub.parallelconsumer.*, with the shared internal utilities under bz.stub.parallelconsumer.internal.utils.
    • For the rename itself, two changes are required: the dependency declaration, and the imports. Rewrite your imports with the one-line sed in the README's Upgrading section.
    • The rename changes no signature; the rest of this list is what else can touch your code.
  • A commit that exhausts its budget throws OffsetCommitBudgetExceededException (#204; #177, upstream #833). Public package, extends ParallelConsumerException, with Kafka's TimeoutException or SaslAuthenticationException as the cause.
    • Code catching the bare Kafka type around getFailureCause() or in a supervisor stops matching: catch the PC type, or unwrap getCause().
    • The message names the budget that ran out, its value, and the option to raise - and says when offsetCommitTimeout was below the consumer's default.api.timeout.ms, so no retry was reachable.
    • offsetCommitTimeout now bounds the whole commit, not each attempt: PC gives up where it used to retry forever.
    • saslAuthenticationRetryTimeout counts from the first SASL failure, not from the start of the call.
  • The JStream result Stream blocks until the consumer closes (#116; upstream #912). It used to end the first time its queue was momentarily empty, usually before the first result, so results piled up behind a caller that had already walked away.
    • Consume the stream on its own thread, as the Vert.x example now does. There is no compatibility path: the old shape never delivered the caller's results.
  • invalidOffsetMetadataPolicy governs every unreadable payload, and its default is IGNORE, not FAIL (#207). An unknown magic byte, a corrupt body, invalid base64 and Kafka Streams metadata all route through the policy; before, most of them threw on assignment whatever it was set to.
    • FAIL still stops the consumer, and now does so for every unreadable payload.
  • batchSize below one is rejected at construction (#496; #311). Zero, a negative or null now throws IllegalArgumentException from validate(), naming the option. Before, zero started a consumer that joined the group, polled, and processed nothing forever with no warning - or, with messageBufferSize set, died in a bare ArithmeticException. Nothing changes for any value of one or more.
    • A deployment whose batch size comes from a property that resolves to zero now fails to start instead of idling.
  • One exception renamed, one method removed (#267):
    • InternalRuntimeException is PCInternalRuntimeException - it is what getFailureCause() returns, and the old name read like a JDK type in a stack trace.
    • WorkManager.getSuccessfulWorkListeners() is gone; use addSuccessfulWorkListener. Mutating the returned list from another thread threw ConcurrentModificationException inside the control loop and stopped the consumer.
  • RecordContext equality is identity (#468). Two contexts built for the same record from different containers no longer compare equal as Set members or Map keys. It is the fix for a lost record, under Fixes.
  • Subclasses of the internal controller only (#296; #209) - a user of ParallelStreamProcessor is not affected:
    • setupWorkerPool must return a pool whose rejection handler is ThreadPoolExecutor.AbortPolicy; anything else throws IllegalArgumentException at construction. Other handlers were accepted and silently lost records.
    • setState is no longer callable from outside its package. A protected getState is added.
  • parallel-consumer-mutiny requires Java 17 (#214; #194, upstream #906) - its real floor, since SmallRye Mutiny 2.x is compiled for 17. Core, Vert.x and Reactor stay at Java 8.

Improvements

  • The project is renamed Parallel Consumer for Apache Kafka - the README title, and the <name> of the parent and every module (#276).
  • New parallel-consumer-mutiny module - SmallRye Mutiny (Quarkus) integration (upstream #891).
  • Your MDC reaches the threads that run your function. A trace_id or tenant set before poll*() is carried into the worker pool and into the Vert.x, Reactor and Mutiny engines, and whatever your function put in the MDC no longer leaks onto the next record's log lines. On by default; propagateMdc(false) restores the old behaviour exactly. The captured keys, never the values, are logged once at startup so a request-scoped value pinned at poll*() time is discoverable (#205; #195, upstream #907).
  • ParallelConsumerOptions.producerConfig(Map) builds PC's own producer from configuration, as an alternative to handing in an instance; supplying both fails validation, and a producer PC built is closed rather than leaked when start-up fails. First step towards surviving producer fencing, which is not in this release (#426; #225).
  • LongPollingMockConsumer ships in the main parallel-consumer-core artefact, so a broker-free test of your integration no longer needs the tests classifier jar (#202; #159, upstream #526).
  • New shards.max.size gauge: the largest number of records queued behind any single key or partition, so a hot key serialising the work behind it is visible where shards.size could not show it (upstream #905, via #57).
  • A commit-response timeout now tells you what happened. When the broker-poll thread dies, every committer waiting on it is released at once with the poll thread's own exception as the cause, rather than each waiting out offsetCommitTimeout and reporting "Timeout waiting for commit response" - and that message now reports the timeout actually configured, not a constant. When the poll thread is alive, the timeout line states a verdict on whether it is deadlocked, blocked, waiting or merely slow (#204, #354).
  • Vert.x, Reactor and Mutiny throughput. External engines get the pipelined work request back that a 0.4.0.0 change had removed - the source of a throughput regression against 0.3.x - bounded by a dispatch ceiling so maxConcurrency is never breached (#342).
  • Offset encoding on the commit path no longer walks every offset between the committed base and the highest succeeded one when only run-length encoding is in play - a multi-second scan when a single stuck record left a gap of millions (#106).
  • Log lines are bounded. The commit-failure ERROR names every partition and its offset (upstream #850) but summarises the encoded metadata to its length, the dropped-batch WARN and the user-function-failure ERROR log a summary instead of the whole batch with its keys and values, and each unabridged object moved to DEBUG (#203, #428; #168, #169, #170).
  • "Max loading factor steps reached" no longer warns on every control-loop pass. A fixed buffer (messageBufferSize set) reports its ceiling once at DEBUG; a dynamic factor that hits its cap warns at most every thirty seconds and says what to raise (#201; #155, upstream #402).
  • A new consumer group, or one whose committed offset aged out of the offsets topic, no longer logs "Truncating state" at WARN while truncating nothing; it logs at INFO that no committed offset was found and where it starts. The WARN is unchanged for genuine truncation (#494; #162, upstream #546).
  • The runtime error a user meets when configuring a batch size but calling a non-batch poll method links to this fork's documentation rather than upstream's (#289).

Fixes

Priority 1: correctness

Several of these were present in every released 0.5.x line and left no evidence behind: the committed offsets looked correct. Each needs narrow preconditions, and each is persistent once triggered.

  • A commit could describe a partition state that never existed, because the offset to commit and the encoded note of outstanding offsets above it were read separately, and a completion landing between the reads shifted every encoded offset. The loud form is the reported one - auto.offset.reset firing under frequent rebalancing; the quiet form is silent record loss - at traffic at or above the payload width the committed offset tracks the log end exactly, and real records are dismissed as already processed and never retried. Carries upstream's never-merged #893 (#337; #121, upstream #894).
  • The offset encoder read its range top after its snapshot, so a completion landing between the two reads encoded every still-incomplete offset in the widened range as complete; on restore they were skipped. Specific to the default commit mode, PERIODIC_CONSUMER_ASYNCHRONOUS (#344).
  • An asynchronous commit counted as committed when it was sent, not when the broker answered - the default commit mode again. A failed or dropped acknowledgement left the broker's committed offset behind PC's belief, the pc.partition.latest.committed.offset gauge read an offset the broker never had, and the partition's next owner resumed from a position the broker never recorded. Offsets are now marked clean only on the acknowledgement of the exact request that carried them; a transient failure is logged at WARN and re-sent, a permanent one at ERROR (#470; #248, upstream #203).
  • A record polled after a rebalance was dropped if a stale container still occupied its offset. The stale entry from the previous assignment was swept later, but the fresh record was already gone, and Kafka does not redeliver it while the consumer stays up. This is the shape of the "one offset never received, lag climbs, a restart processes it normally" reports (#31; upstream #909, #183, upstream #875).
  • Two sweeps removed work by offset, so either could evict a fresh replacement racing in from the controller. The fresh record then sat marked incomplete with nothing to complete it, pinning the partition's commit. WorkContainer equality is now identity (see Breaking), so a removal evicts the container it inspected or nothing (#468, #492).
  • A record could be selected and completed before its offset was registered, which tripped an internal assertion rather than losing anything today, and held only because registration and completion shared a thread (#450; #370).
  • A result from a container that had already been revoked could dirty the freshly assigned partition state, the one production route into a wrong commit, and its failure-path twin re-queued the stale record where nothing could sweep it (#346).
  • A commit rejected because this consumer had already left the group was recorded as done - no exception, no stall, just bookkeeping ahead of the broker, surfacing when the partition's new owner resumed behind records PC had marked complete (#108; relates to upstream #857).
  • Corrupt or foreign offset metadata produced answers instead of errors: a bitset header claiming more bits than the body held fabricated thousands of incomplete offsets, a huge declared length exhausted the heap, a negative run length placed the highest-seen offset below the committed one. A payload is now checked against itself, and a map whose run or bitset extends past the partition's log end offset - which no legitimate map can - is refused at the first batch rather than making PC skip every real record in the claimed range. Both routes take invalidOffsetMetadataPolicy (see Breaking). The IGNORE recovery also resumed one offset too far, skipping the first record (#207, #480).
  • Transactional mode: exactly-once now holds across a rebalance. The revocation-time commit ran on the poll thread without draining the controller's mailbox, so it could publish a transaction whose committed offsets omitted records the transaction contained; the partition's next owner reprocessed the input and produced the output again. The revoke now hands its commit to the control thread, bounded by commitLockAcquisitionTimeout, and a partition being revoked is fenced from further produces (#466).
  • Transactional mode: a terminally failed send no longer publishes half a result set. The producer callback threw before the client could move the transaction into an abortable state, so the records already accepted were committed and visible to a read_committed consumer while the rest never appeared (#261).
  • Transactional mode with batchSize of two or more redelivered records that had succeeded, with an ERROR blaming your function on every batch. The produce lock was released once per record for a resource taken once per batch (#257).
  • Transactional mode: an InvalidPidMappingException on the produce path marked the whole batch succeeded - output never produced, offset advanced - and a fenced or poisoned producer's abort on close leaked the KafkaProducer. The batch now fails and the close always closes the producer (#429; #423, upstream #830).
  • A worker pool that silently discarded rejected work no longer passes construction - see Breaking (#296).
  • A record PC could not return to its mailbox after a failure is now fatal to the instance rather than skipped. Before, it was logged and PC carried on to commit past work that was never done (#267).

Consumption stopped after a rebalance (upstream #857)

Upstream's most-reported symptom - the group goes quiet after a rebalance, with no error, until a restart - was not one defect. Splitting the reports by commit mode, and building a probe that forces the deadlock window open instead of waiting for a randomised run to find it, turned the symptom into a list of mechanisms. Each is closed with its own regression guard:

  • The poll/control commit deadlock, in PERIODIC_CONSUMER_SYNC only: the revocation callback took the commit monitor on the poll thread while the control thread held it and waited for the poll thread. The revoke path now declines the lock instead of blocking on it; partitions are revoked promptly and the offsets re-derived after the rebalance. A fix had existed for four months that nothing could prove, because the test meant to prove it ran a commit mode in which the deadlock cannot occur; the deterministic probe is what settled it, and it runs on every build. The same change removed a per-pass shard scan hidden inside a trace-level log argument, which cost real throughput under KEY ordering (#29; #119).
  • A commit landing during a rebalance killed the broker-poll thread, and the instance died later with a misleading commit timeout. Most likely under the cooperative-sticky assignor, whose members keep committing during rebalances (#100).
  • A closing consumer wedged its group: it stopped polling while draining, so it burned a core and stayed a rebalance-unresponsive member holding its whole assignment (#80).
  • A dead broker-poll thread left the Kafka consumer open in the consumer-commit modes - the shipped default among them - so no LeaveGroup was sent and the dead member kept its partitions until the broker evicted it after max.poll.interval.ms: a silent partial outage of about five minutes per instance that died this way. The consumer is now closed in every mode, and the warnings on that path name max.poll.interval.ms rather than session.timeout.ms as the timeout that actually evicts (#477).
  • A rebalance callback could wait, unbounded, on the retry queue's lock while the control loop scanned it - inside consumer.poll(), with the whole group waiting, spent out of max.poll.interval.ms. The poll thread no longer touches the retry queue; the controller collects what the callbacks leave (#481).
  • The record-intake gate was fed by counters that drifted: revoking a record parked in retry back-off, or a stale replacement at one offset, left phantom counts that kept the broker poller paused until a restart cleared them. The figure is now derived by conservation from what the maps actually did, with no clamp to hide drift, and the shard's own count is owned by a compare-and-set (#336, #373).
  • A retry entry orphaned by a rebalance kept "work is waiting" true forever with nothing assigned, and a sibling window left an orphan that held a draining close open to its timeout (#346, #437).
  • Under KEY ordering, a success landing mid-revocation could throw out of the rebalance listener with a NullPointerException, killing the poll thread - the third distinct null on that path, after upstream #757 (#345). A revoke after a failed assignment threw the same way and left later partitions unswept; a missing partition epoch now fails closed with a message naming the invariant (#451).
  • Two partition flags crossed threads unfenced. A stale read of the dirty flag burnt a commit cycle, and on a partition that then went idle the committed offset waited for the next rebalance; a stale read of the back-pressure flag admitted nothing, so nothing succeeded and the encode that would clear it never ran again. Measured on real hardware before being fixed: one is a volatile, the other is redesigned as a monotone completion count so nothing clears a flag any more (#349, #469).
  • Reactor: a Publisher that completed empty never retired its record, and one emitting several elements retired it once per element; with the new dispatch ceiling the first would have wedged the engine. Records now retire on the terminal signal (#342).
  • The back-pressure pause state is read from Kafka rather than mirrored, because the eager and cooperative protocols do opposite things with pause state across a rebalance, and a mirror that resets on assignment leaves a cooperative member paused for good. A regression guard, not a live defect on 0.5.x (#376).

What was measured and ruled out, so the list above is honest about its edges: the residual stall in very large fleets is the consumer-group protocol's join phase held open by churn, during which every member's poll returns nothing, and PC holds nothing during it (#444, #486); the "eager-mode stall" was a timing bound that flips with the processor count, withdrawn (#478); the instance-stall sightings under load are worker saturation with the member busy in user code, not a wedge (#458, #488); and the lag-stagnation detector measures speed rather than liveness, so it now reports instead of failing a run (#354). One arm is still unattributed: a member that stops answering the coordinator during a churn storm has been seen only in the randomised chaos suite, never fired under the load experiment, and cannot be told apart from the coordinator's own join phase without the closing members' thread dumps. Its seeds are recorded and #119 stays open for it.

Other fixes

  • PCMetrics grew without bound - one duplicate meter id per commit for the life of the consumer - and an exception from the meter registry during teardown propagated into the poll thread. Registration is deduplicated, revocation removes what it registered, and teardown never throws; OffsetMapCodecManager is kept instantiated so meters are no longer recreated on every commit (#57; #120, upstream #859, upstream #892).
  • Registering a loop-end callback or a success listener from another thread could stop the consumer with a ConcurrentModificationException the handler never reported; both registries are copy-on-write. The same change closed a class: user code that throws where PC's bookkeeping could not survive it. A retryDelayProvider that throws, returns null or returns a negative delay now falls back to the configured default instead of leaving its record stuck forever, and a user throwable is rendered without running the user's getCause() under the logger (#267).
  • A record could be redelivered immediately with its configured back-off skipped, when a claim decided before a renewed retry delay won a compare-and-set against a state that had cycled back to the same value. The claim is one atomic transition on a per-attempt identity (#335).
  • The retry queue read its index without the lock on one path, and could leave in-flight work queued for retry (#354).
  • Transactional mode: an explicit commitInterval equal to the default constant was treated as unset and silently replaced with the 100ms transactional default - fifty times the broker load configured. An explicit interval is now always kept (#427; #422).
  • Offset metadata PC did not write no longer kills the consumer on assignment - a group previously owned by a Kafka Streams application, another framework or operator tooling crashed the rebalance callback with "Unexpected magic"; it is now routed through the metadata policy above (#217; #118, upstream #326).
  • A close racing work distribution no longer kills the control thread: work is not taken for a pool that is already shut down, a rejection from such a pool is tolerated and the batch left for redelivery, and an instance whose pool died while running closes itself with a recorded failure cause instead of spinning (#296).
  • Vert.x: every close entry point now releases the WebClient and the Vert.x instance PC built - only the close(Duration, DrainingMode) form did, so a plain close() or try-with-resources stranded the event-loop group. A caller-supplied Vert.x runtime is never closed, and the caller's duration bounds the wait again (#453).
  • The last metrics counter map on the encode path is concurrent, like its siblings (#452).
  • Records polled for a partition whose assignment callback has not yet run are skipped and redelivered rather than raising a NullPointerException on a null epoch (afde8c5e).

Known limitations

Stated so the release does not claim more than it can show. None is new in this release.

  • Transactional producer mode: a revocation can wait on the transaction lock past the poll interval. When a second instance joins while a long commit holds the lock, the revocation callback waits for it - since #466 bounded by commitLockAcquisitionTimeout rather than unbounded, but not yet declined - and a wait longer than max.poll.interval.ms evicts the member. This is upstream's one verified-bug report. The fix depends on producer-fencing recovery and follows this release (#44, upstream #803; fix in #408).
  • Transactional producer mode: a poisoned transaction is not aborted while the instance runs. One terminal send failure - typically a record the broker rejects as too large - leaves the transaction open, so that partition stops for the life of the process. No partial result set is published; the data guarantee holds. The fix is on the same stack and follows this release (#434).
  • Records that never succeed pin the committed offset, and what sits above it has to fit in the commit metadata.
    • The committed offset is the lowest incomplete one. Everything completed above it is encoded into Kafka's 4 KB commit metadata, as a run-length or a bitset, whichever is smaller.
    • Regular patterns compress to almost nothing: a contiguous run of failures, or every second offset failing, costs next to no space however long it runs.
    • It takes an irregular scatter of permanent failures among successes, over a wide range, to outgrow the limit. When it does, back-pressure stops admitting that partition's records, and under retry-forever nothing clears it: the base can never move past a record that never completes.
    • Separately, records held for retry count toward the in-flight target (maxConcurrency x batchSize, times the load factor). A small target with a steady supply of failing records pauses the broker poller the way any full buffer does; a larger messageBufferSize raises that ceiling.
    • This release makes the gate's latched state visible: a WARN when it has stayed loaded with nothing retiring (#497). Before it, the only signal was the paused-partition gauge.
    • Workarounds today: handle terminal failures inside your function so the record retires, as the README's Skipping Records section describes, or a retryDelayProvider with a growing back-off.
    • The fix for records that cannot succeed is the dead-letter queue, under What comes next (#487; #149, upstream #310).
  • A fenced producer still terminates the instance. Recovery - abort, rebuild the producer, rejoin - is designed and follows this release (#225).
  • The test suite cannot see a single wedged key-order shard inside an otherwise healthy partition. A key whose work will never be dispatched again pins that partition's commit and fails no detector; a per-partition commit-liveness gate now exists (#491), the per-shard half does not, and no reproduction of the shape has been found.
  • The commit-response timeout reports were never reproduced (upstream #809, #833; #175). Every known cause of that message is fixed above, the message now says which it was, and a soak built to hunt the timeout found the intake stall instead.
  • A revocation redelivers work that was in flight - by design. PC commits only completed work, so records mid-flight when a partition moves are processed again by the new owner. The README section on reducing duplicate replay - choose the cooperative assignor, and drain on close - is the guidance (upstream #777).
  • The JStream result queue has no capacity bound, so a consumer of the stream slower than the producer grows it; what this release fixed is the consumer that walked away (#216).

What comes next

This release ships fixes. The capabilities are behind it in a queue, and this is that queue's state at the tag, grouped by how far each item has got. The roadmap data is docs/data/roadmap.yaml, rendered into the README. Nothing here is a date, and everything marked preview ships behind an opt-in.

Implemented, in the merge queue - the code exists on an open pull request, awaiting review and merge:

  • Producer-fencing recovery: abort, rebuild the producer and carry on instead of dying (#410, on #472 and #474; #225). With it, the two transactional-mode limitations above close: a revocation declines the transaction lock (#408) and a poisoned transaction is aborted (#434).
  • Virtual threads for the user function, opt-in, JDK 21 and later (#360; #190).
  • Self-tuning concurrency: the engine discovers its own sustainable admission target instead of asking you to guess maxConcurrency (#333; #227).
  • Global rate limiting across a whole consumer group, so a downstream budget is respected by every instance together: the partition-share allocator (#456) and its first cut (#392; #228).
  • Kafka Streams on Parallel Consumer, preview: a Streams topology run with per-key concurrency behind an opt-in switch (#271, #388), with worker failures and buffer pressure delivered into Streams' own control paths (#395), task ownership carried through rebalance (#394), stream time (#396), a refusal of the surface it cannot run safely (#389), an evidence suite (#398) and a runnable example (#391); #255.
  • Kafka Connect sinks on Parallel Consumer, preview (#269; #240).
  • Other languages: key-ordered concurrency for non-JVM runtimes through a sidecar, with clients for Python, Go, Rust, Ruby, .NET, TypeScript, Swift and C++ - the frozen wire contract (#383), the server (#384), a GraalVM native image (#385), the Java reference client (#386), one conformance suite every client must pass (#387), the per-language clients (#390) and demos (#331), and an in-process path for Go, Python and C with no sidecar at all (#340); #242.
  • A commit-failure seam: the application decides what happens when a commit fails, instead of PC always terminating (#352; #317).
  • A health-check surface on the consumer interface (#226).
  • Observability: record residence time, how long a record spends inside PC end to end including retries (#359); an embedded web dashboard showing offset encoding and in-flight state (#268; #215); denser offset metadata (#306; #192).
  • Engine performance: a direct-pull engine with a constant-time scan (#361) and the benchmark harness and results behind it (#362).
  • Documentation: a versioned documentation site (#302, #316; #208), a README per module (#303), and industry-grounded examples for every module (#266).
  • An API-compatibility gate, so the published Java API cannot change unnoticed (#315).

Designed, not yet built - requirements or a design exist, and no implementation does:

  • A dead-letter queue for records that cannot succeed - the most-asked feature, and the proper fix for the retry-forever limitation above (#149, brainstorm #313, upstream #310).
  • Batch failure attribution: a batch function reports which records failed, so only those retry (#189).
  • An error seam on the poll path, so a record that cannot be deserialised can be skipped or diverted rather than killing the instance (#163, #148, #153).
  • Micro-batching by size and time rather than by whatever is available, once the batch-composition decision it depends on is taken (upstream #915 is waiting on the same decision).
  • Bounded internal buffers with visible depth, and delivered-value metrics - the measures the library exists to improve, emitted directly.
  • A Java 17 baseline with Kafka 4 support, for the 0.7 line (#53).

Toward 1.0: settle the public API so it stops moving, including a stated thread-safety contract (#139).

Dependencies

  • Dependencies and build plugins refreshed to their latest non-major versions (#73): JUnit 5.10.2 → 5.14.4, Mockito 5.12.0 → 5.23.0, Testcontainers 1.19.8 → 1.21.4, AssertJ 3.24.2 → 3.27.7, SLF4J 2.0.13 → 2.0.18, Project Reactor 3.6.2 → 3.8.7 (the last step closing two advisories), SmallRye Mutiny 2.9.4 → 2.9.5, Vert.x 4.5.7 → 4.5.31, Lombok 1.18.28 → 1.18.46, Logback 1.5.x → 1.6.1, plus Guava, commons-lang3 and the PostgreSQL driver.
  • The Kafka client moves 3.9.1 → 3.9.2, a patch bump within the same 3.9 line - no API or behaviour change for callers. lz4-java is pinned ahead of the version kafka-clients 3.9.2 ships, which carries an advisory in code this library never calls; the pin drops once the client ships the fix itself.
  • Netty is held at 4.1.137.Final through netty-bom, closing an advisory in the version Vert.x resolves on its own; this reaches users of parallel-consumer-vertx transitively.
  • Micrometer is held on the 1.13 line (1.13.15), with five advisories excluded in the pom - three against it, two against its transitive HdrHistogram and LatencyUtils. Each is argued unreachable from this library in the pom itself - the affected emitters and instrumentation are not on the classpath, and no tag PC emits can carry the tainted input - and none has a public fix on the 1.13 line. The move to 1.16.7 or later, which retires all of them, waits on the Prometheus registry package rename in the metrics example. A user who hands PC a StatsD or logging registry together with attacker-controlled common tags is in the advisory's own antipattern and should move to 1.16.7 themselves. The reasoning per advisory: docs/inflight/deps-cve-backlog.md (#281, #430, #445, #493).
  • Kafka 4.x, JUnit 6, Testcontainers 2, Vert.x 5, Mutiny 3 and WireMock 3 were deliberately deferred to keep the release low-risk; each has its blocker recorded in docs/inflight/deps-deferred-majors.md.

Examples

  • parallel-consumer-example-streams: StreamsApp now takes bootstrapServers in its constructor and is actually runnable - its consumer, producer and server methods were stubs that only the test filled in.
  • The README's custom retry-delay example halved the wait on every failed attempt instead of growing it; the multiplier is corrected in the source the README is generated from, so upstream's docs-only patch cannot be lost again (#196; #167, upstream #622).
  • The Vert.x example consumes the JStream result stream on its own thread, as the API now requires (#116).

Build & CI

Test and CI infrastructure was rebuilt while preparing this release. The parts that say something about how carefully the library is tested:

  • A flake fails the build. The surefire retry that turned a failed-then-passed test into a green run is gone; what it had been hiding is tracked in a ledger, and a test that is genuinely under diagnosis is quarantined - annotated, registered, and tracked to an owning fix PR, with CI enforcing the registry in both directions and a release blocked while any test is quarantined. The registry is empty at this release: its last entries were cleared by fixing what they were about (#224, #84, #473).
  • Chaos Pain Suite - seeded, replayable scenarios that drive consumer fleets through rebalance storms under no-loss, bounded-duplicate and progress SLOs, with an ambient probe that autopsies every failing broker test, built to catch the silent stalls behind upstream #857. Several of the fixes above were found or confirmed this way (#83, #86).
  • Concurrency testing under a controlled scheduler, as standing lanes. Lincheck harnesses over the shard, retry-queue and partition-state seams, calibrated by refinding most of the torn-read defects above unaided - and one nobody had named - and a standalone jcstress module that measured the memory-model residuals on real hardware before they were fixed (#347, #404, #348). Next in this lane is Fray, which explores thread interleavings deterministically over the whole engine rather than a seam at a time.
  • A machine-checked register of every documented transactional guarantee, each with the test that proves it, which refuses to certify a run whose filters deselected its proofs (#262, #443).
  • No silently uncollected tests. An ArchUnit rule fails the build when a class holding tests is named something surefire never runs - dormant classes were found this way and re-enabled - and a second keeps Docker-dependent tests out of the fast unit suite. The tests inherited disabled from upstream are re-enabled or deleted, so no test in the tree is @Disabled (#101, #264).
  • Static analysis on by default, as a required check. SpotBugs over every module including tests, with fb-contrib, find-sec-bugs and the SLF4J detectors; Error Prone with NullAway and its @GuardedBy lock-discipline check at error level; Infer with every Java checker on - RacerD's race analysis among them - as a ratchet against a known-findings set, so a new finding fails and an old one cannot silently return; javac's own lint; and forbidden-apis. Every disabled rule is registered with a reason and a re-enable trigger (#356).
  • GitHub code scanning (CodeQL) over the Java, the workflows and the tooling on every PR, and dependency review on every dependency change.
  • Mutation testing on the classes a PR changes, as its own non-blocking check, plus a nightly whole-repo sweep; and forked-per-broker integration tests that removed a long-standing flake class (#111, #463, #68).
  • Coverage per suite, with history. Unit and integration coverage are uploaded under separate flags with patch and project checks on every PR, every master push uploads its base, and a query tool reads that history to say whether a given test has ever failed and at which commit - a record that outlives any CI log.
  • Throughput measured against master, not against a number in a file. Every PR's performance test is compared with recent master runs read from their artefacts, the run-to-run spread is reported beside the ratio so a reading is not mistaken for a result, and a master-side baseline series exists for the first time (#401).
  • Duplicate-code reports on every PR from two engines cross-validating each other, gated on the increase against the base branch rather than an absolute; and a file-similarity report for near-copies the line-level engines miss.
  • A whole-tree CVE scan that can prove it ran, on every PR and weekly, as its own check that goes red on a finding without blocking a merge; every exclusion carries its reasoning and a retirement condition in the pom (#279, #489).
  • Repository hygiene as a required check. Every issue reference must name its repository, because the fork's issue numbers overlap upstream's; every cited file path must exist; copyright headers, the documentation data's structure, shell lint and the PR checklist are all gated; and a stacked PR cannot merge before its parents.
Assets 3
Loading