docs: per-command Docker requirements for flowctl (#2968)#3184
Open
0xfandom wants to merge 1 commit into
Open
Conversation
The flowctl concepts page said only that "some flowctl commands, such
as `flowctl generate`, require a Docker daemon", which understates the
requirement. Users writing their first TypeScript or Python derivation
hit hangs and timeouts in `preview` and `catalog test` with no signal
that Docker is the missing prerequisite, while SQL-only walkthroughs
work without it and set the opposite expectation.
Add a "Docker requirements" section with a per-command matrix, derived
from the actual dispatch paths:
- SQLite derivations run in-process (derive_sqlite::connector); never
need Docker. TypeScript and Python derivations run as connector
images (ghcr.io/estuary/derive-{typescript,python}).
- `preview` validates all connectors locally (load_and_validate_full).
- `catalog test` AND `catalog publish` both use load_and_validate,
which runs derivation connectors locally but not capture or
materialization connectors - so publishing a TS/Python derivation
needs Docker even though execution happens in the data plane.
- `generate` needs a capture/materialization connector image only to
spec a missing endpoint config stub.
- Podman works via the DOCKER_CLI environment variable.
Also add a troubleshooting entry for the hang/timeout symptom.
Fixes estuary#2968
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
docs/concepts/flowctl.mdsaid only that "some flowctl commands, such asflowctl generate, require a Docker daemon", which understates the requirement. A user writing their first TypeScript or Python derivation hits hangs and timeouts inflowctl preview/flowctl catalog testwith nothing pointing at Docker as the missing prerequisite — while SQL-only walkthroughs work without Docker and set the opposite expectation.This adds a Docker requirements section with a per-command matrix, plus a troubleshooting entry for the hang/timeout symptom. Each cell is derived from the actual dispatch paths rather than the issue's proposed table (one correction — see notes):
derive_sqlite::connector()(crates/runtime/src/derive/connector.rs) — never need Docker. TypeScript/Python derivations run as connector images (ghcr.io/estuary/derive-{typescript,python}).previewuseslocal_specs::load_and_validate_full— all connectors run locally.catalog testandcatalog publishboth uselocal_specs::load_and_validate, which validates derivation connectors locally but not capture/materialization connectors.generateneeds a capture/materialization connector image only to spec a missing endpoint config stub.DOCKER_CLIenv var (crates/runtime/src/container.rs), with no CLI flag.Fixes #2968
Workflow steps:
No behavior changes — documentation only. Readers of concepts/flowctl now get:
Documentation links affected:
#docker-requirementsanchor; installation step 2 and troubleshooting both link to it. No existing anchors moved or removed.Notes for reviewers:
flowctl catalog publishas never needing Docker ("control-plane handles execution"). Butcatalog/publish.rscalls the samelocal_specs::load_and_validateascatalog test, which runs derivation connectors locally — so publishing a TS/Python derivation does require Docker, even though execution happens in the data plane. The table and footnote ² reflect that.docusaurus build: the section renders with its table, footnotes, and admonition, and the#docker-requirementsanchor resolves. (The build then fails post-render in the redirect plugin onhubspot-real-time— a macOS case-insensitivity collision between thedocusaurus.config.js:113redirect andHubSpot-real-time.mdthat exists on master and doesn't occur on case-sensitive CI filesystems; happy to file it separately.)container.rsand the flowctl README; this is the user-facing docs side.