Split images/ into a separate repo now that #407 makes them optional at deploy time?

[RonaldHensbergen]

Context

Issue #407 proposes a per-module image.source: build | registry config
option: when source: registry, a module's Compose service pulls the
scanned/signed image already built and published by publish-images.yml
(to GHCR and Docker Hub) instead of rebuilding from the local Dockerfile
under images/.

Once that lands, a production profile (source: registry everywhere)
no longer needs the images/ directory, its Dockerfiles, or the
build-images/publish-images.yml/image-security-scan machinery at
checkout time at all — it only needs the compiled profiles/ +
modules/ tree and network access to pull already-published, signed
images.

Question

Would it make sense to eventually split this repository into:

1. An "images" repo — owns images/**, the Dockerfiles, image-specific
   requirements files, publish-images.yml, image-security-scan.yml,
   and the vulnerability-remediation SLA process (docs/image-scanning.md,
   .trivyignore). This repo's only externally-visible output is a set of
   signed, versioned images on GHCR/Docker Hub.
2. A "profiles" repo (this repo, minus images/) — owns the CDS CLI
   itself (cli/**), modules/**, profiles/**, shared/**,
   workdirs/**, and consumes the published images by digest/tag via
   image.source: registry module config.

Why this might be worth it

• Faster CI for the common case. Most PRs touch cli/, modules/, or
  profiles/ and currently still run through image-build-adjacent jobs
  (build-images, smoke-test, Boot/E2E) even when no Dockerfile changed.
  Splitting removes that coupling for the CLI/config-only repo.
• Independent release cadence. Images already have their own weekly
  rebuild + SLA-driven patch cycle (docs/image-scanning.md) that's
  decoupled from CLI/profile feature work; a separate repo would make that
  boundary explicit instead of implicit.
• Smaller blast radius for vulnerability response. A HIGH/CRITICAL
  finding today opens a vuln-scan issue in this repo and requires a PR
  here even though it's purely an OS/dependency patch unrelated to CDS
  features (see [vuln-scan] HIGH/CRITICAL findings in cds-dagster #526–[vuln-scan] HIGH/CRITICAL findings in cds-dbt #528, just fixed). A dedicated images repo would keep
  that traffic separate from CLI/feature PRs and issues.
• Clearer ownership/access boundaries, if this project ever gains more
  contributors — image-security patches and CLI feature work have very
  different review needs.

Why this might not be worth it (or not yet)

• Local dev still wants source: build (the default). Local dev/test
  needs the Dockerfiles + images/ present in the same checkout as the
  profile being iterated on, or dev workflows get slower/more awkward
  (submodule/multi-repo checkout, or cds get-style fetching of the images
  repo).
• Coordinated changes. A module + its image sometimes change together
  (e.g. new module config that requires an image rebuild, like Add Alpine-based hardened variant for Dagster image (and later Superset) #372/Add selectable Dagster image variant (base/hardened) with restructured image layout #376's
  image.variant split). Two repos means two PRs + a version-pin bump in
  between, adding coordination overhead for a project this size.
• Small team/single maintainer. The main benefits (independent release
  cadence, access boundaries) matter more with multiple contributors/teams;
  for now the split may just add process overhead (two CI configs, two
  issue trackers, cross-repo version pinning) without much practical
  benefit.
• cds get/cds init already fetch profiles+modules independently of a
  full checkout (see the extends-aware fetch logic added for Profile composition: support layered/extended profiles to reduce duplication #175), so
  the "modules repo doesn't need images" use case is already partially
  served without a full repo split — worth asking whether that alone is
  "good enough."

Possible middle ground

• Keep one repo, but treat images/** as an independently-tagged/released
  subtree (e.g. its own GitHub Release cadence and CODEOWNERS boundary)
  without a full repo split, revisiting a real split only if contributor
  count or CI time actually become a pain point.

Ask

Does a split make sense now that #407 makes images/ optional at
deploy time, or is the middle-ground (independent versioning, same repo)
the better first step? Feedback and alternative approaches welcome.

[RonaldHensbergen]

@SemTiOne your thoughts please!

[SemTiOne]

Hmmm... Makes sense to split eventually, but not now. I'd do the middle ground first, keep images/** in this repo with independent versioning via its own release cadence and CODEOWNERS boundary.

The reasons are:

• Two repos means two CI configs, two issue trackers, cross-repo version pinning. We're a two-person team, the coordination overhead isn't worth it yet.
• Module + image changes still couple often. Dagster variant split (Add selectable Dagster image variant (base/hardened) with restructured image layout #376) needed both module.yaml and Dockerfile changes in the same PR. Two repos means two PRs + version bump in between.
• cds get/cds init already handle fetching modules without a full checkout.

publish-images.yml path filter already scopes image CI to images/** changes, so CLI-only PRs skip image builds today.

I'd sequence it as:

1. Land Allow modules to pull published images instead of always building locally #407 (image.source: build | registry), makes images/ optional at deploy time
2. Add independent tagging to images/** subtree
3. Split only if contributor count or CI time actually becomes a bottleneck

The signed-images fixture and update-fixture job already track digests per-image. That infrastructure carries over cleanly if we split later.