feat(java): add batch consume support for PushConsumer#1249
Conversation
Add batch message consumption capability to the Java PushConsumer with: - BatchMessageListener interface returning a single ConsumeResult for the entire batch - BatchPolicy with Builder pattern and sensible defaults (32 msgs, 4MB, 5s) - BatchConsumeService for Standard mode with precise batch splitting (extractBatch) respecting both maxBatchSize and maxBatchBytes, with forward-progress guarantee - FifoBatchConsumeService for FIFO mode via inheritance with whole-batch retry semantics and single-batch-in-flight constraint - ConsumeService parent class enhanced with close(), protected constructor, and getters - Config validation in builder (cache limits >= batch limits) - Comprehensive tests using awaitility (BatchConsumeServiceTest, BatchConsumeTaskTest, BatchPolicyTest) covering Standard/FIFO modes, retry, DLQ, concurrency, and edge cases - BatchPushConsumerExample demonstrating usage Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolved conflicts in PushConsumerBuilder, PushConsumerBuilderImpl, and PushConsumerImpl to integrate batch consume support with upstream's new features (fifo consume accelerator, message interceptor filtering, lite consumer support). Co-Authored-By: Claude <noreply@anthropic.com>
…merImpl signature The batch consume feature added BatchMessageListener and BatchPolicy parameters to PushConsumerImpl constructors, but LitePushConsumerImpl was still calling the old signature. Pass null for batch parameters and false for enableMessageInterceptorFiltering to restore compilation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1249 +/- ##
============================================
+ Coverage 53.26% 53.87% +0.60%
- Complexity 651 834 +183
============================================
Files 208 224 +16
Lines 14303 14734 +431
Branches 5845 5600 -245
============================================
+ Hits 7619 7938 +319
- Misses 6308 6390 +82
- Partials 376 406 +30
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
1. Performance: Replace ArrayList with ArrayDeque to fix O(n) removal issue - buffer.remove(0) is O(n) for ArrayList, causing O(n*m) complexity - ArrayDeque.removeFirst() is O(1), improving overall performance 2. Semantics: Fix firstArrivalNanos reset logic to track actual arrival time - Previously reset to System.nanoTime() after partial extraction - Now uses actual message arrival time (BufferedMessage.arrivalNanos) - Prevents buffer drift under high load 3. Consistency: Cache bodySize in BufferedMessage to avoid repeated calculation - ByteBuffer.remaining() called twice could return different values - Cache once at buffer entry, reuse throughout lifecycle - Ensures bufferBytes counter stays accurate 4. Reliability: Make close() synchronously wait for pending batches - submitBatch now returns ListenableFuture for completion tracking - close() collects all futures and waits with 30s timeout - Prevents message loss during shutdown 5. Documentation: Add warning about FIFO global buffer breaking messageGroup isolation - BatchMessageListener JavaDoc warns about cross-group batching - FifoBatchConsumeService JavaDoc clarifies single batch in-flight semantics - Users understand retry affects all messageGroups in batch All tests pass (13/13 BatchConsumeServiceTest). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ifoBatchConsumeService.close() Replace fully qualified names (java.util.List, java.util.ArrayList, etc.) with short names since the imports are already declared at the top of the file. This improves code readability and follows Java coding conventions.
Review Suggestions
|
- Add dedicated FifoBatchConsumeServiceTest covering FIFO ordering, retry, DLQ, corrupted message handling, metrics, and graceful shutdown - Change submitBatch() callback executor from directExecutor() to listeningDecorator(getConsumptionExecutor()) for consistency with the task submission executor Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
# Conflicts: # java/client/src/main/java/org/apache/rocketmq/client/java/impl/consumer/PushConsumerImpl.java
Summary
Add batch message consumption capability to the Java PushConsumer, allowing messages to be accumulated in a buffer and delivered in batches based on configurable policies.
List<MessageView>and returns a singleConsumeResultfor the entire batchextractBatchrespects both count and bytes limits simultaneously) and forward-progress guarantee (oversized single message still flushes)close()lifecycle method, protected constructor for batch services (no NOOP listener hack), and accessor methodsmaxCacheMessageCount >= maxBatchSize,maxCacheMessageSizeInBytes >= maxBatchBytesKey Design Decisions
ConsumeResultper batchFifoBatchConsumeService extends BatchConsumeServiceTest plan
BatchPolicyTest— 15 cases: constructor validation, Builder defaults, Builder custom values, invalid paramsBatchConsumeTaskTest— 6 cases: success, failure, exception, null return, unmodifiable list, batch sizeBatchConsumeServiceTest— 13 cases: maxBatchSize flush, maxBatchBytes flush, maxWaitTime flush, Standard failure, FIFO success, FIFO retry+DLQ, FIFO single-batch-in-flight, graceful shutdown, concurrent PQ submission, corrupted message discard, precise batch splitting, forward-progress, FIFO corrupted discardmvn -B packagepasses locally (JDK 11, 223 tests, 0 failures)🤖 Generated with Claude Code