[build] Add a local development kit for contributors

devkit/.gitignore

@@ -0,0 +1 @@
+.deps/

devkit/README.md

@@ -0,0 +1,140 @@
+# Fluss DevKit
+
+Fluss DevKit starts a local Fluss environment with Docker Compose and `just`. It uses upstream
+runtime images and mounts the local `../build-target` directory read-only, so testing source changes
+never requires building a Docker image.
+
+## Requirements
+
+- JDK 11 or later
+- Docker with Docker Compose
+- [just](https://github.com/casey/just)
+
+## Quick Start
+
+Build and start the core development environment:
+
+```bash
+cd devkit
+just build
+just up
+```
+
+For a lake profile, build the tiering artifacts and select a format:
+
+```bash
+just build-tiering
+just up iceberg
+```
+
+`just up` waits for the Fluss and Flink clusters and, for lake profiles, the tiering job to become
+ready. Run `just --list` to see all available commands.
+
+## Profiles
+
+| Profile | Services |
+|---|---|
+| `core` | ZooKeeper, Fluss, and Flink |
+| `iceberg` | Core, RustFS, PostgreSQL JDBC Catalog, Flink, and Iceberg tiering |
+| `paimon` | Core, RustFS, Flink, and Paimon tiering |
+| `lance` | Core, RustFS, Flink, and Lance tiering |
+| `hudi` | Core, RustFS, Flink, and Hudi tiering |
+
+Profiles start one TabletServer by default. Pass `3` as the second argument to start three:
+
+```bash
+just up core 3
+just up paimon 3
+```
+
+## Validate Lake Tiering
+
+The validation workflow is intentionally split into commands that can be run and inspected one at
+a time:
+
+```bash
+just build-tiering
+just up paimon
+
+format=paimon
+table="${format}_manual_$(date +%s)"
+
+just create-table "$format" "$table"
+just write-data "$format" "$table"
+just tiering-status
+just query-lake "$format" "$table"
+```
+
+Use `iceberg`, `paimon`, or `hudi` for a complete create, write, tier, and lake-query workflow.
+Tiering is asynchronous, so repeat `query-lake` if the first query runs before the lake commit is
+visible.
+
+The sample writes three rows. A successful lake query reports row count `3`, ID sum `6`, three
+distinct payloads, and the payload range `alpha` to `gamma`. Use a new table name for each run:
+dropping a non-empty Fluss table does not remove its lake table.
+
+Lance supports `create-table`, `write-data`, and tiering, but the DevKit does not bundle a Flink SQL
+reader for `query-lake`. Inspect the generated objects in RustFS when validating Lance.
+
+## Operations
+
+```bash
+just status
+just logs
+just logs tablet-server-0 200
+just exec tablet-server-0 java -version
+just run-sql ./my-query.sql
+just down
+just clean
+```
+
+Start a profile with `just up` before using `run-sql`. SQL files are read from the host and passed to
+the Flink SQL Client. Relative paths are resolved from the current working directory.
+
+`just up` replaces containers from the previous profile but preserves named volumes. `down` removes
+containers and keeps data; `clean` also removes Compose volumes. Downloaded JARs remain in the
+ignored `.deps` cache.
+
+Default endpoints:
+
+| Service | Address |
+|---|---|
+| CoordinatorServer | `localhost:9123` |
+| TabletServer 0 | `localhost:9124` |
+| TabletServers 1 and 2 | `localhost:9125` and `localhost:9126` (three-node mode) |
+| CoordinatorServer JDWP | `localhost:15005` |
+| TabletServer JDWP | `localhost:15006` to `localhost:15008` |
+| CoordinatorServer metrics | `http://localhost:9249/metrics` |
+| TabletServer metrics | `http://localhost:9250/metrics` to `http://localhost:9252/metrics` |
+| Flink UI | `http://localhost:8083` |
+| RustFS S3 API | `http://localhost:9000` |
+| RustFS Console | `http://localhost:9001` |
+
+JDWP is enabled for every Fluss process with `suspend=n`, so a remote debugger can attach without
+blocking startup. Prometheus export is also enabled by default and can be checked directly, for
+example with `curl http://localhost:9249/metrics`.
+
+The default Fluss runtime is `eclipse-temurin:17-jre-noble`; set `FLUSS_DEVKIT_IMAGE` to use
+another Java 17 runtime. Port environment variables in the Compose files can override the default
+addresses, for example `COORDINATOR_DEBUG_PORT=5005 just up` or
+`TABLET_SERVER_0_METRICS_PORT=9300 just up`.
+
+## Adding a Profile
+
+Create a directory under `profiles/`. The directory contains one required file and up to four
+optional files:
+
+| File | Purpose |
+|---|---|
+| `server.yaml` | Fluss and lake configuration |
+| `jars.urls` | JARs used by both Fluss Server and Flink |
+| `server.urls` | Additional Fluss Server-only JARs |
+| `flink.urls` | Additional Flink-only JARs |
+| `compose.files` | Compose overlays, relative to `devkit/`, for extra local services |
+
+Put one URL or Compose path on each line; empty lines and `#` comments are ignored. If `server.yaml`
+contains `datalake.format`, `just up` automatically stages the matching locally built lake plugin
+and starts Flink tiering. Use `compose.files` only when the profile needs additional local services
+or runtime overrides.
+
+Start the new profile with `just up <profile>`. No `justfile` change is required.

devkit/compose/iceberg.yml

@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  postgres:
+    image: postgres:17
+    restart: unless-stopped
+    environment:
+      POSTGRES_USER: iceberg
+      POSTGRES_PASSWORD: iceberg
+      POSTGRES_DB: iceberg
+    volumes:
+      - iceberg-postgres-data:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U iceberg -d iceberg"]
+      interval: 3s
+      timeout: 3s
+      retries: 20
+      start_period: 5s
+
+  coordinator-server:
+    depends_on:
+      postgres:
+        condition: service_healthy
+
+volumes:
+  iceberg-postgres-data:

devkit/compose/lake-common.yml

@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  rustfs:
+    image: rustfs/rustfs:1.0.0-alpha.83
+    restart: unless-stopped
+    command: /data
+    environment:
+      RUSTFS_ACCESS_KEY: rustfsadmin
+      RUSTFS_SECRET_KEY: rustfsadmin
+      RUSTFS_CONSOLE_ENABLE: "true"
+    ports:
+      - "${RUSTFS_API_PORT:-9000}:9000"
+      - "${RUSTFS_CONSOLE_PORT:-9001}:9001"
+    volumes:
+      - rustfs-data:/data
+    healthcheck:
+      test: ["CMD-SHELL", "nc -z 127.0.0.1 9000"]
+      interval: 5s
+      timeout: 5s
+      retries: 20
+      start_period: 5s
+
+  rustfs-init:
+    image: minio/mc
+    depends_on:
+      rustfs:
+        condition: service_healthy
+    entrypoint:
+      - /bin/sh
+      - -c
+      - |
+        until mc alias set rustfs http://rustfs:9000 rustfsadmin rustfsadmin; do sleep 2; done
+        mc mb --ignore-existing rustfs/fluss
+        printf '' | mc pipe rustfs/fluss/hudi/.keep
+
+  coordinator-server:
+    depends_on:
+      rustfs-init:
+        condition: service_completed_successfully
+
+  jobmanager:
+    depends_on:
+      rustfs-init:
+        condition: service_completed_successfully
+
+volumes:
+  rustfs-data:

devkit/compose/lance.yml

@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  taskmanager:
+    environment:
+      FLINK_PROPERTIES: |
+        jobmanager.rpc.address: jobmanager
+        taskmanager.memory.process.size: 2048m
+        taskmanager.memory.task.off-heap.size: 512m
+        taskmanager.numberOfTaskSlots: 4

devkit/compose/tiering.yml

@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-flink-common: &flink-common
+  image: flink:1.20.0-scala_2.12-java17
+  entrypoint:
+    - /bin/bash
+    - -c
+    - |
+      cp /opt/devkit-jars/lib/*.jar /opt/flink/lib/
+      chmod a+r /opt/flink/lib/*.jar
+      exec /docker-entrypoint.sh "$$@"
+    - --
+  volumes:
+    - ./.deps/flink/active:/opt/devkit-jars:ro
+    - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/devkit-config/server.yaml:ro
+    - ./tests:/opt/devkit-tests:ro
+    - shared-data:/tmp/fluss
+
+services:
+  jobmanager:
+    <<: *flink-common
+    profiles: [flink]
+    command: ["jobmanager"]
+    depends_on:
+      coordinator-server:
+        condition: service_healthy
+    environment:
+      FLINK_PROPERTIES: |
+        jobmanager.rpc.address: jobmanager
+        jobmanager.memory.process.size: 1024m
+        parallelism.default: 1
+        rest.address: jobmanager
+    ports:
+      - "${FLINK_REST_PORT:-8083}:8081"
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS http://localhost:8081/overview >/dev/null"]
+      interval: 5s
+      timeout: 5s
+      retries: 24
+      start_period: 10s
+
+  taskmanager:
+    <<: *flink-common
+    profiles: [flink]
+    command: ["taskmanager"]
+    depends_on:
+      jobmanager:
+        condition: service_healthy
+    environment:
+      FLINK_PROPERTIES: |
+        jobmanager.rpc.address: jobmanager
+        taskmanager.memory.process.size: 1536m
+        taskmanager.numberOfTaskSlots: 4
+    healthcheck:
+      test:
+        - CMD-SHELL
+        - curl -fsS http://jobmanager:8081/taskmanagers | grep -q '"id"'
+      interval: 5s
+      timeout: 5s
+      retries: 24
+      start_period: 10s
+
+  tiering-submit:
+    <<: *flink-common
+    profiles: [tiering]
+    restart: "no"
+    depends_on:
+      taskmanager:
+        condition: service_healthy
+    environment:
+      FLINK_PROPERTIES: |
+        jobmanager.rpc.address: jobmanager
+        rest.address: jobmanager
+    command:
+      - /bin/bash
+      - -c
+      - |
+        ARGS=()
+        HAS_BOOTSTRAP=false
+        while IFS= read -r LINE || [ -n "$$LINE" ]; do
+          LINE="$${LINE%%#*}"
+          [[ "$$LINE" == *": "* ]] || continue
+          KEY="$${LINE%%: *}"
+          VALUE="$${LINE#*: }"
+          KEY="$${KEY#"$${KEY%%[![:space:]]*}"}"
+          KEY="$${KEY%"$${KEY##*[![:space:]]}"}"
+          VALUE="$${VALUE#"$${VALUE%%[![:space:]]*}"}"
+          VALUE="$${VALUE%"$${VALUE##*[![:space:]]}"}"
+          case "$$KEY" in
+            datalake.*|lake.tiering.*|config.providers*|fluss.*)
+              ARGS+=("--$$KEY" "$$VALUE")
+              [ "$$KEY" = "fluss.bootstrap.servers" ] && HAS_BOOTSTRAP=true
+              ;;
+          esac
+        done < /opt/devkit-config/server.yaml
+        if [ "$$HAS_BOOTSTRAP" = false ]; then
+          ARGS+=(--fluss.bootstrap.servers coordinator-server:19123)
+        fi
+        exec /opt/flink/bin/flink run -d \
+          /opt/devkit-jars/jobs/fluss-flink-tiering.jar "$${ARGS[@]}"