Skip to content

feat(connectors): add Apache Fluss sink - #3782

Open
adamrecsko wants to merge 1 commit into
apache:masterfrom
adamrecsko:feat/connectors-fluss-connector
Open

feat(connectors): add Apache Fluss sink#3782
adamrecsko wants to merge 1 commit into
apache:masterfrom
adamrecsko:feat/connectors-fluss-connector

Conversation

@adamrecsko

Copy link
Copy Markdown

Which issue does this PR address?

Relates to #3689

Rationale

This is phase 1 of Apache Fluss connector support, enabling Apache Iggy streams to write messages into Fluss.

What changed?

Apache Iggy previously had no sink for writing messages to Apache Fluss. The new sink writes binary, JSON, or text payloads to append-only Fluss log tables, with optional Iggy metadata and
automatic table creation.

Writer buffering, retries, idempotence, SASL authentication, startup validation, and Docker-backed integration tests are included.

Local Execution

  • Local checks passed except for full workspace Clippy
  • Pre-commit hooks ran and passed
  • Passed cargo fmt --all
  • Passed cargo sort --check --no-format --workspace
  • Passed focused Clippy for iggy_connector_fluss_sink
  • Passed all 19 Fluss sink unit tests
  • Passed the focused Docker-backed Fluss integration test
  • Full workspace Clippy did not pass because of six unrelated existing errors in core/sdk and one in core/server

AI Usage

  1. OpenAI Codex.
  2. Used for implementation assistance, review remediation, test execution, and PR wording.
  3. Verified with formatting, dependency sorting, focused Clippy, unit tests, Docker-backed integration testing, and pre-commit hooks.
  4. Yes, I can explain every line of the code if asked.

@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 31, 2026
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.74%. Comparing base (01e7210) to head (af17922).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3782      +/-   ##
============================================
- Coverage     75.78%   75.74%   -0.05%     
  Complexity      969      969              
============================================
  Files          1323     1323              
  Lines        159751   159673      -78     
  Branches     133139   133138       -1     
============================================
- Hits         121071   120943     -128     
+ Misses        35045    35026      -19     
- Partials       3635     3704      +69     
Components Coverage Δ
Rust Core 75.68% <ø> (-0.01%) ⬇️
Java SDK 62.71% <ø> (ø)
C# SDK 71.13% <ø> (-1.16%) ⬇️
Python SDK 93.10% <ø> (ø)
PHP SDK 84.52% <ø> (ø)
Node SDK 96.30% <ø> (+0.15%) ⬆️
Go SDK 43.08% <ø> (ø)
see 34 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@adamrecsko
adamrecsko force-pushed the feat/connectors-fluss-connector branch 2 times, most recently from 3140c92 to c131ae6 Compare August 1, 2026 07:42
@krishvishal

Copy link
Copy Markdown
Contributor

Thanks for the PR! The CI is failing because Cargo.lock in this branch is internally inconsistent.

This should fix it:

git merge origin/master
cargo check --workspace

@adamrecsko
adamrecsko force-pushed the feat/connectors-fluss-connector branch 2 times, most recently from 2d74b3d to 905186b Compare August 3, 2026 17:10
@adamrecsko

Copy link
Copy Markdown
Author

Thanks for the PR! The CI is failing because Cargo.lock in this branch is internally inconsistent.

This should fix it:

git merge origin/master
cargo check --workspace

Hi @krishvishal !
Done, sync and rebased onto master and run cargo check --workspace

@adamrecsko
adamrecsko force-pushed the feat/connectors-fluss-connector branch from 905186b to af17922 Compare August 3, 2026 18:05
@adamrecsko

Copy link
Copy Markdown
Author

Small update, I've the moved fluss-rs version management to the workspace dependencies, the source connector does the same, I wanted to be align with that

@krishvishal

krishvishal commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Hey @adamrecsko the CI is failing.Seems its related to fluss version. Please look into it.

Also please rebase with latest master.

You can also join Iggy's discord so we can converse there.

@hubcio

hubcio commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

it might be related to protoc being required, check discussion under #3688

@adamrecsko

Copy link
Copy Markdown
Author

It is most probably the same issue as in the source connector, but I will try to reproduce locally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants