Skip to content

test(connectors): add random source liveness helper#3377

Merged
hubcio merged 10 commits into
apache:masterfrom
arunsingh:feat/2892-pareto-source-suite
Jun 5, 2026
Merged

test(connectors): add random source liveness helper#3377
hubcio merged 10 commits into
apache:masterfrom
arunsingh:feat/2892-pareto-source-suite

Conversation

@arunsingh
Copy link
Copy Markdown
Contributor

@arunsingh arunsingh commented May 29, 2026

Summary

Part of #2892.

Adds a small random source liveness helper and wires the random source smoke test through it.

Changes:

  • Add connectors::random_source_liveness for bounded random source message polling.
  • Replace random source polling with the liveness helper.
  • Drop random restart coverage because random has no stable id/cursor to prove resume.

The broader behavior suite should start from sources with observable cursors/ids, such as postgres or influxdb, with #2940 scope kept separate for failed-send semantics.

Local validation

  • cargo fmt --all --check
  • git diff --check

Local blockers:

  • cargo test -p integration --no-run is blocked on Windows before reaching these tests by existing Unix-only message_bus imports.
  • WSL validation is blocked because cargo is not installed in WSL.

@arunsingh arunsingh changed the title test(connectors): add source connector suite helper test(connectors): add source connector pareto suite May 29, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented May 29, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 46.41%. Comparing base (10ca430) to head (06c7948).

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3377       +/-   ##
=============================================
- Coverage     74.49%   46.41%   -28.09%     
  Complexity      943      943               
=============================================
  Files          1245     1243        -2     
  Lines        121477   106184    -15293     
  Branches      97599    82306    -15293     
=============================================
- Hits          90498    49286    -41212     
- Misses        28026    54264    +26238     
+ Partials       2953     2634      -319     
Components Coverage Δ
Rust Core 39.13% <ø> (-36.46%) ⬇️
Java SDK 58.44% <ø> (ø)
C# SDK 69.92% <ø> (ø)
Python SDK 81.06% <ø> (ø)
PHP SDK 83.57% <ø> (ø)
Node SDK 91.53% <ø> (ø)
Go SDK 40.20% <ø> (ø)
see 350 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@arunsingh arunsingh marked this pull request as ready for review May 29, 2026 20:18
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label May 29, 2026
Comment thread core/integration/tests/connectors/source_suite.md Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 1, 2026
@hubcio
Copy link
Copy Markdown
Contributor

hubcio commented Jun 1, 2026

overall change is OK, but what about the rest of connectors? you marked that #2892 will be closed, yet this issue it only touching the random source connector which is essentially a pure-test connector type.

@arunsingh
Copy link
Copy Markdown
Contributor Author

arunsingh commented Jun 1, 2026

overall change is OK, but what about the rest of connectors? you marked that #2892 will be closed, yet this issue it only touching the random source connector which is essentially a pure-test connector type.

@hubcio I have changed this to Part of #2892; this PR won't close the issue, and real source connectors can be wired in follow-ups. Let me know if you want this merged as the first step or want one real source connector wired before merge, likely, wire one Docker-backed source connector, probably Postgres or Elasticsearch, into source_suite.

Copy link
Copy Markdown
Contributor

@hubcio hubcio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for PR, I finally had some time to properly review it.

two blockers inline (the restart test can't actually fail, and poll_until_min_messages drops messages when min_messages is above one batch), then a note on direction.

on direction: #2892 is really about behaviour - progress-before-send, retry-on-fail, restart-resume, mark/delete-after-commit, replay window. a poll-until-non-empty helper proves liveness, which already works everywhere; it doesn't give a seam for any of those, and two structural things stop it growing into the suite as-is.

first, the random source can't host 4 of the 5 checks - no seed, the produced ProducedMessage goes out with envelope id: None (only a random, non-monotonic Uuid lives inside the json payload as Record.id), no consumer-visible cursor (just an in-process messages_produced counter that's already unit-tested in the crate), and no external system to mark/delete. so it can only ever be a liveness smoke. the sources that already have an observable, gated cursor are postgres and influxdb - those are the honest first wires for a real behaviour test, and the postgres source/restart tests from #2579 are the pattern to copy.

second, heads up that behaviours 1 and 2 aren't guaranteed by the SDK today: the plugin advances its cursor inside poll() before returning (random does messages_produced +=, postgres marks/deletes rows and bumps tracking offsets), and the send path is one-way - SendCallback returns (), and the runtime's failed-send branch only logs/metrics/sets-error, nothing re-offers the batch. so on a failed send the un-sent batch is dropped (at-most-once on the runtime->iggy hop), and #2892 item 2 can't pass against any source without an SDK/FFI change. this is the same contract question already open in #2940 (partial writes / replay-safe progress / deterministic write identity) - worth linking the suite scope to that and rescoping, rather than writing a test that asserts behaviour the code doesn't have. the one invariant the runtime does enforce is on-disk state only advances on send success (the state save sits entirely in the success arm), which is testable now.

suggested shape if you want this to become the real suite: a small SourceUnderTest trait the runner is generic over (config_toml, seed(n) with a monotonic stamped id, expected()), plus one accumulating poll_until_n that replaces the inline loops and unifies the retry budget - today that's two independent per-module copies of POLL_ATTEMPTS/POLL_INTERVAL_MS (postgres + elasticsearch) plus hardcoded literals in influxdb, so there's no single shared constant to point at yet. that's basically the existing postgres fixture made generic. with the connector roadmap (#2753) and new sources still landing (mongodb #3285, meilisearch #3404), a real per-source baseline pays off. random then stays as an explicit liveness smoke and the restart check lives on postgres.

Comment thread core/integration/tests/connectors/random/random_source.rs Outdated
Comment thread core/integration/tests/connectors/random/random_source.rs Outdated
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
@arunsingh arunsingh changed the title test(connectors): add source connector pareto suite test(connectors): add source liveness helper Jun 2, 2026
@arunsingh
Copy link
Copy Markdown
Contributor Author

Updated this to liveness-only: random restart is dropped, polling now accumulates/retries, and postgres/influxdb behavior coverage can follow with #2940 scope in mind.

hubcio
hubcio previously approved these changes Jun 3, 2026
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
Comment thread core/integration/tests/connectors/source_suite.rs Outdated
@hubcio
Copy link
Copy Markdown
Contributor

hubcio commented Jun 3, 2026

I approved, above comments are just nits. feel free to fix them now or in future PRs. let me know :)

@arunsingh arunsingh changed the title test(connectors): add source liveness helper test(connectors): add random source liveness helper Jun 4, 2026
@arunsingh
Copy link
Copy Markdown
Contributor Author

Addressed the nits in 08d2287 and kept the PR scoped to random source liveness.

@hubcio hubcio merged commit da75706 into apache:master Jun 5, 2026
57 checks passed
@github-actions github-actions Bot removed the S-waiting-on-author PR is waiting on author response label Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants