v1.1.0

Every service is now reachable from your machine only, and the docs have a Roadmap and a Changelog.

• Every published port binds to 127.0.0.1, on macOS, Linux and WSL2, so other machines on your network cannot reach the services. How you reach a service says how to open a port when you need to.
• Kafka no longer publishes ports 29092, 29093 and 29094, which were listeners for a minikube cluster.
• An odctl skill and llms.txt help an AI assistant choose a profile and reach its services. See Use it with an AI assistant.
• Eight guides show in a diagram how data moves between their services, and How it works shows what odctl up checks and the order it starts services in. The end-to-end tests run every command the guides show.
• The tables of odctl ps and odctl explain draw straight right borders on rows with port mappings.
• odctl down --help says what --volumes removes.

Upgrading

• To reach a service from another machine, run odctl init, remove 127.0.0.1: from the service's port line in .odctl/compose-*.yml, and run odctl recreate <profile>.
• A Kafka client in minikube cannot use host.minikube.internal:29092 any more. Add a listener for it to the Kafka service in .odctl/compose-kafka.yml.
• A workspace made by odctl init with an earlier release keeps its old compose files, which publish on all interfaces and include port 29092. Run odctl init --force to copy the new files; it replaces your edits.

Full Changelog: v1.0.2...v1.1.0

v1.0.2

odctl 1.0.2 adds Python 3.15 support and updates two images.

✨ Python 3.15

• odctl is tested on Python 3.15 and lists it on PyPI. CI now runs 3.10 to 3.15.

⬆️ Updates

• ZooKeeper 3.9.6 (Fluss profile).
• Metabase v0.63.19.3.

📝 Docs

• The README and Getting started show a short recording of a run.
• A "Why not plain Docker Compose?" section in the README and on the docs home page.
• The README says projects can pin odctl>=1.0,<2, in place of the Compatibility section.

🛠️ Infrastructure

• Renovate opens update PRs on Mondays only, grouping image updates by compose file (#102).

Full Changelog: v1.0.1...v1.0.2

v1.0.1

odctl 1.0.1 fixes two problems that appear after upgrading from an earlier version.

🐛 Bug Fixes

• Old jars stayed in the shared deps volume after an upgrade (#126). The deps container copied new jars into odctl-shared-deps but never removed old ones, so a service could load a jar from an earlier version. After upgrading to 1.0.0, Kafka Connect loaded Debezium 3.5.1 instead of 3.7.0. deps now removes files the image no longer ships. The first odctl up after upgrading cleans the volume, so no odctl down --volumes is needed.
• Without a workspace, a cached latest image kept running (#127). With no .odctl workspace, odctl's own images ran as latest, and Docker does not pull an image it already has. odctl now uses its own version as the tag, as odctl init does. A TAG set in the shell still takes precedence.
• odctl explain showed ${TAG:-latest} for odctl's own images. It now shows the tag that will run, for example mlflow:1.0.1.

⬆️ Updates

• Metabase v0.63.19.2.

🛠️ Infrastructure & Testing

• e2e: the deps smoke test plants a jar the image does not ship and checks that odctl up deps removes it.
• Docs:
  • the guides no longer repeat server versions, which the Profiles pages already list from the compose files;
  • a test checks that the client versions the guides tell readers to install (Evidently, Feast, MLflow, XGBoost) match the images.

Full Changelog: v1.0.0...v1.0.1

v1.0.0

odctl 1.0.0: a documentation site, a compatibility promise for 1.x, and a fix for odctl up straight after odctl recreate.

✨ Key Features & Improvements

• Documentation site at jaehyeon.me/odctl. It has:

  • a getting-started page;
  • How it works, with the workspace and tested examples for changing memory, ports and Python packages;
  • one profile page per area;
  • 18 guides with screenshots, one per technology the end-to-end tests cover;
  • the CLI reference and troubleshooting.

  The profile pages and the CLI reference are generated from registry.yml, the compose files and odctl --help, so they match the release.

• A compatibility promise. From 1.0.0, odctl follows semantic versioning. Within 1.x, command names, option names, profile names and the host ports a profile publishes stay the same. Minor releases add profiles, services and options.

• Every command warns about an old workspace. odctl list, explain, info and the rest now say when .odctl/.env sets an older TAG, as odctl up already did. Before, they read the old workspace silently. odctl init does not warn, because it is the fix.

⚠️ Breaking Changes

• capacities is removed from registry.yml. No command read it. A workspace registry.yml from 0.10.0 or earlier that still has it keeps working, because the key is ignored.

🐛 Bug Fixes

• odctl up straight after odctl recreate storage failed with "Failed to start compose-infra.yml". recreate returned while seaweed-init was still creating buckets, and docker compose up --wait fails when a container exits during the wait. up and recreate now wait for it to finish.
• The issue templates used the old CLI name dml and old profile names.

🛠️ Infrastructure & Testing

• Docs CI:
  • the site is built in strict mode on every run;
  • it is published from main with mike, one folder per version with latest as an alias;
  • canonical addresses point at latest.
• Docs tests check:
  • every profile is on exactly one page;
  • the home page areas come from the same list as the profile pages;
  • every technology appears in the README diagram;
  • local service addresses become links that search engines do not follow.
• CI warnings:
  • each test job has its own uv cache key;
  • pre-commit runs without pre-commit/action, which ran on the deprecated Node.js 20.
• Package metadata: the classifier is Production/Stable, and the source package leaves out the docs and their images.

Full Changelog: v0.10.0...v1.0.0

v0.10.0

Iceberg 1.12 and Flink 2.2 across the stack, a PostgreSQL with keyword and geospatial search, the new temporal and evidently profiles, and a Grafana dashboard for every service.

✨ Key Features & Improvements

• Iceberg 1.12.0, Flink 2.2, Fluss 1.0.0 and Hadoop 3.4.3. Flink 2.2 is the newest minor with Kafka and JDBC connector releases. The Flink client for Fluss is pinned to the same version as the apache/fluss image. Spark stays on 4.1.3, because Iceberg 1.12.0 publishes no Spark 4.2 runtime.
• The Iceberg REST catalog and the Kafka Connect Iceberg sink are built from the Iceberg source. Apache stopped publishing the apache/iceberg-rest-fixture image at 1.10.1, and the sink's runtime is not on Maven. The deps image builds both from the Iceberg release the stack uses, so the catalog, the sink and the engine clients always match.
• PostgreSQL 18 with pgvector, pg_textsearch and PostGIS. odctl now builds its own PostgreSQL image on the official one. pg_textsearch adds BM25 keyword search, for hybrid search with pgvector, and PostGIS adds geospatial queries. All three are created in the vector database.
• New profile temporal: the Temporal development server with its Web UI, for durable workflows such as one that waits for a person to approve an agent's action.
• New profile evidently: the Evidently UI for data drift, prediction drift, data quality and model quality reports, with everything stored in PostgreSQL.
• A Grafana dashboard for every service. 19 dashboards load into an odctl folder in the telemetry profile's Grafana.
• Version updates: Debezium 3.7.0.Final, the PostgreSQL JDBC driver 42.7.13, Trino 483, Metabase v0.63.19.1, Karapace 6.2.4, SeaweedFS 4.48 and otel-lgtm 0.35.0.

⚠️ Breaking Changes

• The Kafka Connect Iceberg sink is now Apache's. Its class is org.apache.iceberg.connect.IcebergSinkConnector, and it only appends rows. iceberg.tables.upsert-mode-enabled and iceberg.tables.cdc-field no longer exist. iceberg.table.write-props. is now iceberg.tables.write-props., and iceberg.control.group-id is now iceberg.control.group-id-prefix.
• The mlops image is renamed mlflow, which is what it runs. The mlops profile group keeps its name.
• An existing .odctl workspace keeps the old compose files. Run odctl init --force after upgrading; it resets local edits.

🐛 Bug Fixes

• Flink JDBC tables work. The Flink JDBC connector needs the OpenLineage Java client but does not bundle it, so every Flink JDBC source or sink failed with ClassNotFoundException: io.openlineage.client.utils.jdbc.JdbcExtractor. The deps image now ships openlineage-java.
• feast-ui no longer fails at start on a fresh database. Every Feast process creates the registry tables when it starts, so feast-ui and feast-serve could both try at once. feast-ui now starts once feast-serve is healthy.
• The Kafka Connect Iceberg sink commits in under a minute. A sink task reads its control topic only when offsets flush, so with Kafka Connect's 60 second default the first snapshot took about five minutes. Connect now flushes every 10 seconds.
• Airflow no longer logs export failures when telemetry is not running. Every Airflow process logged a retry and a failure for each metrics batch. Airflow now also uses the standard OTEL_* variables, which Airflow 3.3 asks for.
• odctl explain labels the catalog, PostgreSQL, MLflow and Evidently containers.

🛠️ Infrastructure & Testing

• Smoke tests check more of each profile:
  • the source-built Iceberg sink commits a snapshot;
  • Flink writes and reads an Iceberg table, and a Fluss table;
  • Flink reads Avro from Kafka through the schema registry and writes to PostgreSQL over JDBC;
  • PostgreSQL answers a BM25 query and a PostGIS distance;
  • every profile serves its metrics, and Grafana loads all 19 dashboards.
• Renovate:
  • tracks the Fluss, Debezium, PostgreSQL driver, pgvector and pg_textsearch versions;
  • waits three days after a release before proposing it;
  • no longer keeps a Dependency Dashboard issue.

Full Changelog: v0.9.0...v0.10.0

v0.9.0

Feast reads its feature repository from object storage, each ML tool starts from one profile, and host addresses work whether Docker's IPv6 is on or off.

✨ Key Features & Improvements

• The Feast repository comes from s3://feast/repo, as DAGs come from s3://airflow/dags. feast-ui and feast-serve check the prefix every 15 seconds and reload and restart Feast when a file is added, changed or removed, with no container restart. With nothing there, they use the bundled repository. A feast bucket is created with the others.
• One profile per tool.
  • odctl up feast starts the Feast UI and the online feature server.
  • odctl up mlflow starts the tracking server and the model server.
  • The model server waits idle and healthy while MODEL_URI is empty. It serves once MODEL_URI is set and odctl up mlflow runs again.
  • Feast now sits in the mlops group beside MLflow.
• Airflow knows the Feast registry. The container sets FEAST_REGISTRY, next to MLFLOW_TRACKING_URI and the PyIceberg variables, so a task that builds a FeatureStore needs no settings of its own.
• Host addresses use 127.0.0.1. With IPv6 enabled in Docker, localhost reaches a service's IPv6 address. Services that listen on IPv4 only then reset the connection: the Feast UI, MLflow and Airflow. Every address odctl prints is 127.0.0.1, and each odctl explain panel ends with a note on when localhost works.
• odctl explain lists every prerequisite that starts. The registry now names only direct prerequisites, and anything already implied is removed. Each profile starts exactly what it did before.

⚠️ Breaking Changes

• Profiles removed: feast-serve and mlflow-serve. odctl up feast and odctl up mlflow start those containers.
• FEAST_REPO is removed. Upload your feature repository to s3://feast/repo.
• An existing .odctl workspace keeps the old compose files. Run odctl init --force after upgrading; it resets local edits.

🐛 Bug Fixes

• _AIRFLOW_PIP_DEPS installs packages again. In 0.8.0 the Airflow service replaced the image's entrypoint, which is what installs them. The variable reached the container, and nothing was installed.
• PostgreSQL allows 300 connections. The default of 100 ran out when many profiles ran together, and new connections were refused with "too many clients already".
• Spark and Flink no longer read half-written jars. Every odctl up copied all jars into the shared volume again, so an engine reading one mid-copy failed with ClassNotFoundException. Now only a missing or changed jar is copied, under a temporary name, then renamed into place.
• odctl explain prints usage text in full. Square brackets such as feast[duckdb,iceberg,redis,postgres] were read as formatting and dropped.

🛠️ Infrastructure & Testing

• The MLflow smoke test checks that the model server waits idle, then serves a registered model through odctl up mlflow.
• The Feast smoke test also checks the online feature server.
• The pgvector and catalog smoke tests tolerate state an earlier failed run left behind.
• Unit tests pin the new shape:
  • the Feast repository sync;
  • the merged profiles;
  • Airflow using the image's entrypoint;
  • FEAST_REGISTRY;
  • the connection limit;
  • the jar copy;
  • the explain note.

Full Changelog: v0.8.0...v0.9.0

v0.8.0

Airflow and telemetry each run as one container, every service reports metrics, and the Airflow image works with the stack's own MLflow, Feast and Iceberg services.

✨ Key Features & Improvements

• Airflow runs as one container. airflow standalone replaces the api-server, scheduler, dag-processor, triggerer, init and aws-cli sync containers. DAGs are read straight from s3://airflow/dags through Airflow's S3DagBundle, refreshed every 15 seconds, so the sync sidecar is gone. A changed DAG showed up in 4 seconds in testing. Plugins are copied from s3://airflow/plugins at start; Airflow loads plugins only once, so odctl restart airflow picks up a change. LocalExecutor parallelism is 4, and the reserved memory falls from 6.25 GiB of limits to 3 GiB.
• The Airflow image works with the stack's own services. It adds the MLflow client (~=3.16.0), the Feast SDK (0.66.0, with the DuckDB, Iceberg, Postgres and Redis extras), pyiceberg[s3] 0.12.0, and the XGBoost, LightGBM and CPU PyTorch runtimes. The containers get MLFLOW_TRACKING_URI and the PYICEBERG_CATALOG__ODCTL__* variables Feast reads. XGBoost is held at 3.2 in both the airflow and mlops images: the mlops image runs Python 3.11, XGBoost 3.3 and later need 3.12, and a model trained in Airflow has to load in MLflow.
• Telemetry runs as one container. grafana/otel-lgtm replaces Prometheus, Alertmanager and Grafana, and adds logs, traces and an OpenTelemetry receiver on 4317 (gRPC) and 4318 (HTTP). Prometheus stays on host port 19090 and Grafana on 3004, and prometheus and grafana remain network aliases. Alertmanager had no alert rules to route.
• Every service reports metrics, with no exporter containers.
  • Native endpoints: Karapace, Trino, Metabase, OpenMetadata, Fluss, SeaweedFS, Spark (master, workers and applications), MLflow, feast-serve and Marquez, alongside Flink and ClickHouse.
  • Kafka, Kafka Connect and the Iceberg REST catalog, through the JMX exporter agent, now shipped in the deps image.
  • Airflow pushes over OTLP.
  • PostgreSQL and Valkey are collected by otel-lgtm's own OpenTelemetry Collector.
  • Elasticsearch, the Spark history server, mlflow-serve and feast-ui have no metrics option.
• odctl up warns about an out-of-date workspace. odctl init writes TAG from the CLI version once, so upgrading the CLI later silently kept starting the old images and compose files. odctl up now names the workspace's TAG and suggests odctl init --force.

⚠️ Breaking Changes

• Containers removed: airflow-webserver, airflow-scheduler, airflow-triggerer, airflow-dag-processor, airflow-sync, airflow-init, alertmanager, prometheus and grafana. Airflow is at airflow:8080. prometheus and grafana still resolve, as aliases for the telemetry container.
• Airflow's login uses SimpleAuthManager, which standalone always uses, instead of FAB. The credentials stay user / password.
• An existing .odctl workspace keeps the old compose files. Run odctl init --force after upgrading; it resets local edits.

🐛 Bug Fixes

• Kafka CLI tools run inside the broker container again with metrics on. The JMX agent goes to the broker and Connect worker processes only. Given to the whole container, it would load into every docker exec tool as well, fail to bind port 9404, and exit.

🛠️ Infrastructure & Testing

• The airflow smoke test drives a DAG from S3 to a successful run through the S3 DAG bundle, and checks the fixed login.
• The telemetry smoke test pushes an OTLP metric both to Prometheus and through the collector, and queries Grafana's Prometheus data source with a login.
• Unit tests pin the new shape: one service per profile for airflow and telemetry, the S3 bundle and execution API settings, each service's metrics setting, the collector config, and the workspace warning.
• Version updates: Airflow 3.3.2, MLflow 3.16.1, Spark 4.1.3, Valkey 9.1.3, Confluent 8.3.2, Metabase 0.63.18.2, Karapace 6.2.3 and SeaweedFS 4.47.

Full Changelog: v0.7.0...v0.8.0

v0.7.0

Minor release. Adds a feature store profile, and lets the stack receive OpenTelemetry metrics.

✨ Key Features & Improvements

• A feast profile puts a feature store over the Iceberg tables and Valkey already in the stack. Feast stores no feature values of its own: offline values stay in the Iceberg tables behind catalog and storage, online values go to valkey through feast materialize, and a SQL registry on postgres holds only definitions. odctl up feast starts the UI on 8890 and resolves postgres, storage, catalog and valkey itself; odctl up feast-serve adds the online feature server on 6566. Both run the published quay.io/feastdev/feature-server:0.66.0.
• Four constraints are configured for you, and none of them are in Feast's documentation. warehouse is empty on an IcebergSource, because Feast puts it in the URL path as a prefix and apache/iceberg-rest-fixture serves the specification without one, so a real warehouse returns HTTP 400 Ambiguous URI empty segment. SeaweedFS settings arrive as PYICEBERG_CATALOG__ODCTL__* variables, because the REST path ignores catalog_properties. Your own environment needs feast[duckdb,iceberg,redis,postgres], since the duckdb extra is required alongside iceberg. And feast teardown leaves rows in feature_view_version_history, so a repeated apply into one project collides.
• resources/feast/feature_store.yaml is a default feature repository, so the profile starts before you have written one. FEAST_REPO in the generated .env points it at your own, which you apply yourself: odctl ships no feature definitions.
• Prometheus receives OpenTelemetry metrics at /api/v1/otlp/v1/metrics, through --web.enable-otlp-receiver. Traces need nothing new, since MLflow already exposes an OTLP endpoint at /v1/traces configured entirely on the client.

🐛 Bug Fixes

• The MLflow tracking server moves to host port 5004. macOS binds 5000 to the AirPlay receiver in Control Center, so the server answered inside its container while looking dead from the host. --allowed-hosts carries both the container port and the published one: a missing entry returns 403 on the UI while /health still answers 200, so the container reports healthy with a dead UI.

🛠️ Infrastructure & Testing

• The feast profile asserts what it does rather than that a port answers. Its smoke test builds an Iceberg table through the REST catalog on SeaweedFS, runs feast apply, checks the point-in-time join returns the right as-of values, materialises to Valkey and reads them back, then drops what it made. Each step writes to a log that prints on failure.
• The telemetry smoke test posts an OTLP metric and queries it back, rather than checking that the flag is set. Prometheus accepts the JSON encoding, so this needs curl and no SDK.
• Host port collisions are a unit test. No two profiles that can run together may publish the same port, with lite and full variants of one stack treated as the only exception. All 52 published ports are covered.
• The e2e matrix gains a feast group, and a change to compose-mlops.yml alone now selects mlflow and feast.

Closes #99, #100.

Full Changelog: v0.6.0...v0.7.0

v0.6.0

Minor release. Two profiles that had never been exercised past a listening port turned out to have been broken for months, and repairing them is most of this release.

✨ Key Features & Improvements

• OpenMetadata moves from 1.13.3 to 2.0.1, and Elasticsearch to 9.3.0, bringing the MCP server and Context Center. Elasticsearch 9 is not the price of 2.0: 1.13.6's own compose runs 9.3.0 too, so staying on the 1.x line would have moved it anyway. All 127 server environment variables in the 2.0.1 release compose were diffed against odctl's 30, and nothing was removed or renamed.
• Every profile now asserts what it does, not that a port answers. Nine were written or deepened: airflow, metadata, fluss, lineage, telemetry, postgres, storage, catalog and deps. postgres had shipped pgvector since the image was chosen and nothing had ever run a similarity query against it.
• A release cannot reach PyPI without its images. publish.yml runs docker manifest inspect for every ghcr.io/jaehyeon-kim/odctl/ image at the tag, with no skip flag. odctl up refuses earlier still, naming the expected tag rather than leaving Docker to report a missing manifest.

🐛 Bug Fixes

• The airflow profile could never run a task. Airflow 3 parses DAG files in a separate dag-processor that the profile never started, so airflow dags list stayed empty behind four healthy containers. Tasks then failed because the Execution API defaults to localhost:8080 while the api-server is a different container, and failed again because each container generated its own JWT secret. The DAG scan interval also drops from 300 seconds to 120.
• The fluss profile never started at all. It pointed at clickhouse-keeper, a hostname no container provides, after a commit replaced its dedicated ZooKeeper with ClickHouse Keeper. ClickHouse Keeper cannot serve Fluss regardless: it rejects opcode 19, createContainer, and dropped the session every eight seconds. A zookeeper:3.9.5 container is restored.
• ON CLUSTER '{cluster}' on ch-lite blocked for 181 seconds and then failed, because {cluster} named a two-shard cluster whose second shard is an alpine sleep infinity container. It now names the single-shard cluster that was already defined and unused. Measured at 0.19 seconds after the fix, with ch-full unchanged.
• The MLflow tracking server was killed on startup. Each uvicorn worker imports the ML runtimes in the mlops image, so memory is about 1.1G fixed plus 0.35G per worker, and 2.58G at the default four. Now two workers and a 2500m limit.
• Grafana had no datasource. Its provisioning file sat at the root of the mount rather than in datasources/.
• ClickHouse metrics were unscrapeable twice over: the four Prometheus targets named clickhouse-11 through clickhouse-22 where the containers are ch-11 through ch-22, and port 9363 was not enabled at all.
• Setting Airflow extra packages in .env installed nothing. The CLI wrote _AIRFLOW_PIP_DEPS and the compose file read _PIP_ADDITIONAL_REQUIREMENTS.
• mem_limit on openmetadata-ingestion was nested inside environment:, where compose ignores it, so it ran uncapped. JAVA_OPTS on the server was never read either: the start script reads OPENMETADATA_HEAP_OPTS.
• odctl up hid Docker's own error behind four lines of boilerplate ending in "it wasn't captured", and pointed at docker ps -a, a dead end when a pull fails and no container exists.
• registry.yml gave the wrong Alertmanager port and the wrong Grafana credentials, and Trino advertised a row filter and column mask that rules.json does not contain.

🛠️ Infrastructure & Testing

• Six classes of misconfiguration are now unit tests, each proved by reintroducing the original fault: hostnames no container provides, in mounted config, in compose environment and in bare *_HOST variables; mem_limit nested where compose ignores it; floating image tags; and .env variables nothing reads.
• registry.yml is checked against the compose files, so a documented port or hostname cannot drift. Nothing had ever compared them, which is how the Alertmanager port stayed wrong from the day it was written.
• A service declaring a profile owned by another compose file is now reported. That silence is why ch-keeper listed the fluss profile for months without ever starting for it.
• Health gates on four depends_on edges that previously waited only for a container to exist: marquez-web, grafana, fluss-coordinator and fluss-tablet-1.
• The e2e matrix is derived from the changed files rather than hardcoded, and deps gained an entry it never had. It still runs on tags and manual dispatch only, because TAG follows the CLI version and a pull request asks for images that do not exist yet.
• Every workflow moved to the current major of each action. setup-uv is pinned to a full version because that project stopped publishing moving major tags at v8.

Closes #3, #8, #13, #14, #19, #20, #23, #68, #71, #74, #75, #90, #91.

Full Changelog: v0.5.1...v0.6.0

v0.5.1

Patch release. One fix, in the ACL the Valkey profile ships.

🐛 Bug Fixes

• The Valkey user account can now use Pub/Sub. compose-store.yml created it with ~* and +@all. Those grant all keys and all commands, and neither grants channels, so Valkey applied its default of resetchannels and refused PUBLISH, SUBSCRIBE and PSUBSCRIBE with NOPERM. The account now also carries &*.

Key operations were unaffected, which is what made this hard to see. Streams, strings, hashes and pipelines all worked, because streams are keys and ~* covers them. Only Pub/Sub clients failed, and redis-py's async subscribe does not surface the permission error, so an application logs that it subscribed while PUBSUB CHANNELS stays empty and no message is ever delivered.

Anyone who worked around this with ACL SETUSER user allchannels can drop it.

🛠️ Verification

• Both configurations run on valkey/valkey-bundle:9.1.2-alpine, one container each. With ~* +@all, ACL LIST reports resetchannels and PUBLISH returns NOPERM No permissions to access a channel. With &* added, it reports ~* &* +@all and PUBLISH returns 0.
• 51 unit tests pass. A new test in test_compose_profiles.py asserts the grant is present, and it fails when &* is removed.
• Confirmed downstream: the dynamic-des Redis integration test, which exercises a subscriber, passes against this release on a fresh container with no workaround.

Found while moving dynamic-des example infrastructure onto odctl.

Full Changelog: v0.5.0...v0.5.1