Skip to content

[build] Add a local development kit for contributors - #3851

Open
Gezi-lzq wants to merge 1 commit into
apache:mainfrom
Gezi-lzq:build/devkit
Open

[build] Add a local development kit for contributors#3851
Gezi-lzq wants to merge 1 commit into
apache:mainfrom
Gezi-lzq:build/devkit

Conversation

@Gezi-lzq

@Gezi-lzq Gezi-lzq commented Aug 5, 2026

Copy link
Copy Markdown

Purpose

Closes #3850.

This draft provides a concrete implementation of a contributor-oriented DevKit for running and validating locally built Fluss artifacts. It is opened as a draft to collect feedback on its scope, dependencies, and maintenance boundary.

  • No generative AI tools used
  • Yes, OpenAI Codex was used to assist design, implementation, testing, and review

Brief change log

  • Add a Docker Compose and just based local development environment
  • Mount the local build-target without building a Fluss image
  • Include Flink in the core environment while keeping Lake Tiering profile-specific
  • Support one or three TabletServers and Core, Iceberg, Paimon, Lance, and Hudi profiles
  • Add composable commands for startup, logs, host SQL execution, tiering validation, and cleanup
  • Keep filesystem plugins isolated from Lake plugin classpaths
  • Run Fluss processes under Docker init and expose JDWP and Prometheus metrics endpoints
  • Propagate configured Hadoop settings to the Hudi write client so the S3A-backed profile retains its endpoint and credentials

Tests

The DevKit was exercised manually with both single-node and three-node core clusters. All profile Compose configurations rendered successfully, just --list exposed the expected public commands, and host SQL files were executed through just run-sql.

After the final dependency and profile simplification, a clean-volume regression started the core Fluss and Flink services to healthy state and confirmed that the core profile does not submit a Tiering Job. A host SQL file connected through the Flink SQL Client and returned the Fluss database. The Paimon profile then completed table creation, a three-row write, a running Tiering Job, and a Lake query returning three rows with ID sum 6 and payload range alpha to gamma.

Clean core clusters reached healthy state with JDWP reachable for every Fluss process and each Prometheus endpoint returning fluss_* metrics. A dedicated runtime probe confirmed that Docker init runs as PID 1 with the Fluss Java process as its direct child, and normal Compose shutdown completed successfully. Iceberg, Paimon, and Hudi each completed the create-table, write, tier, and lake-query workflow. Lance was validated through table creation, writes, and a running tiering job. Maven validation, Checkstyle, Spotless, and RAT checks also passed.

Full mvn verify has not yet completed in the current local environment, so this PR remains a draft.

API and Format

No public API or storage format changes.

Documentation

Add devkit/README.md covering setup, profiles, operations, validation, debugging, metrics, and profile extension.

@Gezi-lzq
Gezi-lzq force-pushed the build/devkit branch 2 times, most recently from c60d11e to de731d7 Compare August 5, 2026 03:17
Add a Docker Compose and just based development kit that runs locally built Fluss artifacts and supports core, Iceberg, Paimon, Lance, and Hudi profiles with manual tiering validation commands.

Run Fluss processes under Docker init, enable JDWP and Prometheus metrics, and propagate configured Hadoop settings to the Hudi write client so the S3A-backed Hudi profile retains its endpoint and credentials.
@Gezi-lzq
Gezi-lzq marked this pull request as ready for review August 5, 2026 10:24
@Gezi-lzq
Gezi-lzq marked this pull request as draft August 5, 2026 16:03
@Gezi-lzq
Gezi-lzq marked this pull request as ready for review August 5, 2026 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add a local development kit for contributors

1 participant