Skip to content

CAMEL-20227: Fix Pausable EIP losing messages due to Kafka offset advancement#23805

Merged
davsclaus merged 2 commits into
mainfrom
fix/CAMEL-20227
Jun 6, 2026
Merged

CAMEL-20227: Fix Pausable EIP losing messages due to Kafka offset advancement#23805
davsclaus merged 2 commits into
mainfrom
fix/CAMEL-20227

Conversation

@davsclaus
Copy link
Copy Markdown
Contributor

Summary

Claude Code on behalf of Claus Ibsen

Fixes the Pausable EIP with Kafka consumer losing messages when the consumer is paused and later resumed. The root cause was that KafkaConsumerListener.afterConsume() never called consumer.pause() or seeked back to committed offsets, so consumer.poll() kept advancing offsets for records that were never processed.

  • Rewrite afterConsume() to always evaluate the resume predicate, call consumer.pause() + seekConsumer() when pausing, and consumer.resume() when resuming
  • Remove the paused boolean flag that prevented proper predicate evaluation
  • Extract seekConsumer() helper shared between afterConsume and afterProcess
  • Use Kafka's native consumer.pause() so poll() returns empty records during pause (no Thread.sleep needed)

Prior partial fix (869e870952c2) only added a paused flag without addressing the offset advancement. A full fix existed on the unmerged origin/kafka-pause branch — this PR adapts that fix to current main with the cleaner consumer.pause() approach.

Test plan

  • Existing KafkaPausableConsumerIT passes — verifies all messages are received after pause/resume
  • Existing KafkaPausableConsumerCircuitBreakerIT passes
  • Verify no messages are lost when autoCommitEnable=true during pause/resume cycle

🤖 Generated with Claude Code

…fset advancement

The Pausable EIP with Kafka consumer was losing messages because:
1. afterConsume() only evaluated the resume predicate when already paused,
   but never called consumer.pause() or seeked back to committed offsets
2. The poll loop continued immediately without pausing, causing auto-commit
   to advance offsets for unprocessed records

Fix: Rewrite afterConsume() to always evaluate the predicate, call
consumer.pause() and seek to the correct offset when pausing, and
consumer.resume() when resuming. This ensures poll() returns empty
records during pause and offsets are not advanced for unprocessed messages.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
@davsclaus davsclaus requested review from orpiske and oscerd June 6, 2026 05:43
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 6, 2026

🌟 Thank you for your contribution to the Apache Camel project! 🌟
🤖 CI automation will test this PR automatically.

🐫 Apache Camel Committers, please review the following items:

  • First-time contributors require MANUAL approval for the GitHub Actions to run
  • You can use the command /component-test (camel-)component-name1 (camel-)component-name2.. to request a test from the test bot although they are normally detected and executed by CI.
  • You can label PRs using skip-tests and test-dependents to fine-tune the checks executed by this PR.
  • Build and test logs are available in the summary page. Only Apache Camel committers have access to the summary.

⚠️ Be careful when sharing logs. Review their contents before sharing them publicly.

…le consumers

Add a note recommending autoCommitEnable=false with allowManualCommit=true
when using pausable consumers, to avoid message loss from Kafka auto-committing
offsets for unprocessed records during pause/resume cycles.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
@github-actions github-actions Bot added the docs label Jun 6, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 6, 2026

🧪 CI tested the following changed modules:

  • components/camel-kafka

⚠️ Some tests are disabled on GitHub Actions (@DisabledIfSystemProperty(named = "ci.env.name")) and require manual verification:

  • components/camel-kafka: 2 test(s) disabled on GitHub Actions
All tested modules (8 modules)
  • Camel :: JBang :: MCP
  • Camel :: JBang :: Plugin :: Route Parser
  • Camel :: JBang :: Plugin :: TUI
  • Camel :: JBang :: Plugin :: Validate
  • Camel :: Kafka
  • Camel :: Launcher :: Container
  • Camel :: YAML DSL :: Validator
  • Camel :: YAML DSL :: Validator Maven Plugin

⚙️ View full build and test results

@davsclaus davsclaus merged commit 5836d58 into main Jun 6, 2026
6 checks passed
@davsclaus davsclaus deleted the fix/CAMEL-20227 branch June 6, 2026 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant