ci: add live-broker integration test harness (Kafka in CI)#706
Merged
Conversation
faust had no way to exercise a real broker: tests/integration only shells out to the CLI, and CI ran no Kafka at all, so broker-dependent bugs (offset-commit gaps, exactly-once duplicates, recovery races) couldn't be verified anywhere. Add the foundation: - tests/integration/broker/: a broker_url fixture that skips when no Kafka is reachable (so the suite stays green on dev machines and the normal broker-less jobs), plus fixtures for bootstrap servers and unique topic/group names. - test_smoke.py: an aiokafka produce->consume round-trip (robust anchor proving the CI broker is up) and a full faust App round-trip (send -> agent consumes) exercising the real stack end to end. - A new advisory 'Integration (Kafka ...)' CI job that starts an apache/kafka KRaft service, waits for it, and runs the broker tests. It is continue-on-error and intentionally NOT in the required 'check' gate, so broker flakiness can't block the merge queue while we stabilise it. This unblocks writing regression tests for the broker-only issues. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #706 +/- ##
=======================================
Coverage 94.14% 94.15%
=======================================
Files 104 104
Lines 11136 11136
Branches 1201 1201
=======================================
+ Hits 10484 10485 +1
+ Misses 551 550 -1
Partials 101 101 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…t broker - conftest.py: drop the ';' from the skip message. On Python 3.12 (which the lint job uses) PEP 701 f-string tokenisation exposes the semicolon to the older pycodestyle, which flags it as E702; local flake8 on 3.11 did not. - test_smoke.py: relax ResourceWarning to a non-error for these tests. The suite escalates ResourceWarning (pyproject), but aiokafka/faust leave sockets/tasks that can emit it during live-broker teardown, which masked the real result. - workflow: set KAFKA_LOG4J_ROOT_LOGLEVEL=WARN so the job log shows pytest output instead of thousands of lines of broker logging. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
Two problems on the integration job: 1) Observability: as a GH services container the broker log floods the job log and hides the pytest output. Run Kafka via 'docker run -d' instead so the broker log stays in the container (dumped via 'docker logs' only on failure) and the job log shows the test results. 2) The faust App round-trip timed out (assignment received, then LeaveGroup exactly RECV_TIMEOUT later): an embedded app.start() leaves flow control suspended, so the agent's stream never pulls. Resume it explicitly after assignment, matching what faust's own worker / test helpers do. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
- Add confluent-kafka coverage: a raw confluent round-trip anchor and parametrise the faust App round-trip over both transports (kafka:// = aiokafka, confluent:// = confluent-kafka). confluent-kafka is optional, so those tests importorskip when it's absent. Install it in the CI job via requirements/extras/ckafka.txt. - Stop dumping the full broker log on failure (it drowned the pytest output in the job log); tail it to 40 lines and add -ra --tb=short so the actual test result is visible. - Shorten the round-trip timeouts so a hang fails fast. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
Keep the foundation focused and reliably green: raw aiokafka and confluent-kafka produce->consume round-trips that prove the CI Kafka broker is up and each client library works. Drop the in-process faust App end-to-end test for now -- embedding the app lifecycle (start, drive an agent, shut down) in pytest hangs on teardown and needs its own hardening, which shouldn't block landing the harness. It will return as a follow-up with a proper timeout guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
Both raw round-trips pass, but the repo's autouse threads_not_lingering guard failed them at teardown: talking to the broker makes asyncio resolve localhost via getaddrinfo in its default ThreadPoolExecutor, and that resolver thread (asyncio_0) outlives the test. That is expected for live-network tests and not a leak we can act on, so shadow the guard with a no-op in the broker package's conftest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
This was referenced Jul 19, 2026
wbarnha
added a commit
that referenced
this pull request
Jul 19, 2026
Add tests/integration/broker/test_offset_commit.py, which drives faust's real aiokafka consumer against a live Kafka broker (the #706 CI harness): - test_does_not_commit_past_inflight_message reproduces #606 -- offset 2 is in-flight (tracked, unacked) while 3 and 4 are acked out of order -- and asserts the broker's committed offset never advances past 2. Verified to fail without the fix (the buggy _new_offset returns 5 and commits it). - test_commits_contiguous_acks is a positive control proving the real commit path does persist offset 5 to the broker when nothing is in-flight. Both skip when no broker is reachable and run in the dedicated CI integration job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
faust had no way to exercise a real broker:
tests/integration/only shelled out to the CLI, and CI ran no Kafka at all. This adds the foundation so broker-dependent bugs — offset-commit gaps (#606), exactly-once duplicates (#517/#323), recovery races — can eventually get real regression tests.How
tests/integration/broker/(new):conftest.py— abroker_urlfixture that skips when no Kafka is reachable (quick TCP probe), so the suite stays green on developer machines and the normal broker-less jobs, and runs for real only in the dedicated job. Point it anywhere withFAUST_TEST_BROKER(defaultkafka://localhost:9092). Pluskafka_bootstrap,unique_topic,unique_groupfixtures.test_smoke.py— one round-trip per transport-client library faust ships a driver for:test_aiokafka_roundtrip— produce→consume through aiokafka.test_confluent_roundtrip— produce→consume through confluent-kafka (optional dep;importorskips when absent).CI — a new advisory
Integration (Kafka <version>)job that:apache/kafkaKRaft container (single node, no Zookeeper),confluent-kafkaextra,pytest tests/integration/broker.It is
continue-on-error: trueand not in the requiredcheckgate, so broker flakiness can't block the required checks or the merge queue.Scope
This is deliberately the foundation: the raw round-trip anchors prove Kafka-in-CI works end to end for both client libraries. A full in-process faust
Append-to-end test (start an app, drive an agent, shut it down) is left to a follow-up — embedding the app lifecycle in pytest hangs on teardown and needs its own hardening (a hard timeout guard + clean-shutdown fix), which shouldn't block landing the harness.Locally and in the broker-less jobs the two tests skip (verified).
🤖 Generated with Claude Code