Skip to content

docs: per-command Docker requirements for flowctl (#2968)#3184

Open
0xfandom wants to merge 1 commit into
estuary:masterfrom
0xfandom:docs/flowctl-docker-requirements
Open

docs: per-command Docker requirements for flowctl (#2968)#3184
0xfandom wants to merge 1 commit into
estuary:masterfrom
0xfandom:docs/flowctl-docker-requirements

Conversation

@0xfandom

Copy link
Copy Markdown

Description:

docs/concepts/flowctl.md said only that "some flowctl commands, such as flowctl generate, require a Docker daemon", which understates the requirement. A user writing their first TypeScript or Python derivation hits hangs and timeouts in flowctl preview / flowctl catalog test with nothing pointing at Docker as the missing prerequisite — while SQL-only walkthroughs work without Docker and set the opposite expectation.

This adds a Docker requirements section with a per-command matrix, plus a troubleshooting entry for the hang/timeout symptom. Each cell is derived from the actual dispatch paths rather than the issue's proposed table (one correction — see notes):

  • SQLite derivations dispatch in-process via derive_sqlite::connector() (crates/runtime/src/derive/connector.rs) — never need Docker. TypeScript/Python derivations run as connector images (ghcr.io/estuary/derive-{typescript,python}).
  • preview uses local_specs::load_and_validate_full — all connectors run locally.
  • catalog test and catalog publish both use local_specs::load_and_validate, which validates derivation connectors locally but not capture/materialization connectors.
  • generate needs a capture/materialization connector image only to spec a missing endpoint config stub.
  • Podman is supported via the DOCKER_CLI env var (crates/runtime/src/container.rs), with no CLI flag.

Fixes #2968

Workflow steps:

No behavior changes — documentation only. Readers of concepts/flowctl now get:

  1. Installation step 2 links to the new section instead of the vague one-liner.
  2. A "Docker requirements" section: three orienting bullets, the command × entity matrix, a hang/timeout admonition, and the podman note.
  3. A troubleshooting entry for "preview / catalog test hangs or times out on a TypeScript or Python derivation".

Documentation links affected:

  • concepts/flowctl — new #docker-requirements anchor; installation step 2 and troubleshooting both link to it. No existing anchors moved or removed.

Notes for reviewers:

  • One divergence from the issue's proposed table, in favor of what the code does: the issue marks flowctl catalog publish as never needing Docker ("control-plane handles execution"). But catalog/publish.rs calls the same local_specs::load_and_validate as catalog test, which runs derivation connectors locally — so publishing a TS/Python derivation does require Docker, even though execution happens in the data plane. The table and footnote ² reflect that.
  • Verified with a full docusaurus build: the section renders with its table, footnotes, and admonition, and the #docker-requirements anchor resolves. (The build then fails post-render in the redirect plugin on hubspot-real-time — a macOS case-insensitivity collision between the docusaurus.config.js:113 redirect and HubSpot-real-time.md that exists on master and doesn't occur on case-sensitive CI filesystems; happy to file it separately.)
  • No overlap with Improve Docker daemon diagnostics for local flowctl runs #3001/Improve Docker daemon error for flowctl runtime #2976 (Docker daemon error improvements for flowctl: clearer error when Docker daemon isn't running #2969) — those touch container.rs and the flowctl README; this is the user-facing docs side.

The flowctl concepts page said only that "some flowctl commands, such
as `flowctl generate`, require a Docker daemon", which understates the
requirement. Users writing their first TypeScript or Python derivation
hit hangs and timeouts in `preview` and `catalog test` with no signal
that Docker is the missing prerequisite, while SQL-only walkthroughs
work without it and set the opposite expectation.

Add a "Docker requirements" section with a per-command matrix, derived
from the actual dispatch paths:

- SQLite derivations run in-process (derive_sqlite::connector); never
  need Docker. TypeScript and Python derivations run as connector
  images (ghcr.io/estuary/derive-{typescript,python}).
- `preview` validates all connectors locally (load_and_validate_full).
- `catalog test` AND `catalog publish` both use load_and_validate,
  which runs derivation connectors locally but not capture or
  materialization connectors - so publishing a TS/Python derivation
  needs Docker even though execution happens in the data plane.
- `generate` needs a capture/materialization connector image only to
  spec a missing endpoint config stub.
- Podman works via the DOCKER_CLI environment variable.

Also add a troubleshooting entry for the hang/timeout symptom.

Fixes estuary#2968
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: per-language Docker requirement for flowctl preview / catalog test

1 participant