diff --git a/internal/mirror/PROXY-REGISTRY.md b/internal/mirror/PROXY-REGISTRY.md new file mode 100644 index 00000000..31da955c --- /dev/null +++ b/internal/mirror/PROXY-REGISTRY.md @@ -0,0 +1,276 @@ +# `d8 mirror pull --proxy-registry` + +This document describes `--proxy-registry`, a `d8 mirror pull` mode that lets you pull a Deckhouse Kubernetes Platform bundle from a source registry that does **not** implement the registry catalog API (`/v2/_catalog`, `/v2//tags/list`) — typically a caching/transparent proxy in front of an upstream Deckhouse registry. + +For everything else about `d8 mirror pull` (authentication, bundle layout, push command, troubleshooting, etc.) see [README.MD](./README.MD). + +--- + +## Why this mode exists + +The default `d8 mirror pull` discovers tags by calling the registry's `ListTags` endpoint once per repository: once for the platform release-channel repo, once for the modules root, and once per module. That works against `registry.deckhouse.ru` and any registry that implements the catalog API. + +Caching/proxy registries usually refuse the catalog API outright — they only serve the per-tag manifest and blob endpoints. Against such a registry, the default discovery path returns an empty tag list and the resulting bundle is empty too, even when the cache already holds the exact tags you asked for. + +`--proxy-registry` swaps the catalog calls for a deterministic forward-probe walk over semver tags, seeded from the version anchors you supply via `--include-platform` and `--include-module`. Every tag the registry actually serves ends up in the bundle; every tag that returns `404 Not Found` is treated as "not in the cache" and skipped. + +--- + +## When to use + +| Goal | Recommended flag | +|------|-----------------| +| Pull from `registry.deckhouse.ru` directly | omit `--proxy-registry` | +| Pull from a caching/proxy registry that has already cached the desired versions | `--proxy-registry` + `--include-platform` + `--include-module` | +| Pull from a registry that supports the catalog API but you still want range-based filtering | omit `--proxy-registry`, use `--include-platform` alone | + +--- + +## End-to-end flow + +When you run `d8 mirror pull --proxy-registry` against a caching registry, the pipeline takes the following branches relative to a regular pull. Steps marked **(probe)** are the only ones that differ from the default flow; everything else (image pulling, layout writing, bundle packing, GOST digests) is identical. + +``` + ┌──────────────────────────────────────┐ +1. CLI parses flags │ --proxy-registry on? │ + │ yes → require --include-platform │ + │ and/or --include-module │ + │ each with @ │ + │ reject --deckhouse-tag and │ + │ --since-version │ + └──────────────────────────────────────┘ + │ + ▼ +2. Connect to source registry — same authn as normal pull. + +3. Platform component (skipped if --no-platform): + a. Fetch release-channel snapshots (alpha/beta/early-access/stable/ + rock-solid, plus LTS if it exists). These are direct per-tag + GET requests, so they work against a proxy registry. + b. (probe) Instead of ListTags on the release-channel repo, run the + forward-probe walk seeded from --include-platform's lower bound. + Each candidate version becomes a HEAD against + /release-channel:vX.Y.Z. + c. Apply the normal latest-patch-per-minor and inclusive-anchor + rules to the probe result; merge with channel snapshots; drop + channels that no longer fall inside the constraint window. + d. Pull installer, standalone installer, and Deckhouse platform + images for the resolved version set (per-tag GETs, no listing). + e. Pack into platform.tar. + +4. Installer component (skipped if --no-installer): + Pull /install:<--installer-tag> directly (single tag, no + listing needed; works on any registry). + +5. Security databases component (skipped if --no-security-db): + Pull security DB images by their well-known tags (per-tag GETs). + +6. Modules component (skipped if --no-modules and not --only-extra-images): + For each module name pulled from the --include-module whitelist + (proxy mode never asks the registry "what modules exist?"): + + a. Walk release channels per module — same per-tag GET pattern as + step 3a. Missing channels are tolerated. + b. (probe) Instead of Module(name).ListTags, run the forward-probe + walk seeded from the module's --include-module@. + Each candidate becomes a HEAD against + /modules/:vX.Y.Z. + c. Apply the filter's latest-patch-per-minor / anchor rules; merge + with channel-snapshot versions. + d. Pull module images, release version images, and any internal + digest images referenced by images_digests.json (per-digest GETs). + e. Pull extra images discovered via extra_images.json. + f. Pull VEX attestations (unless --skip-vex-images) by deriving the + .att tag from each image digest and checking if it exists. + g. Pack into module-.tar. + +7. GOST digests (if --gost-digest): + Compute STREEBOG checksums next to each .tar / .chunk. + +8. Cleanup tmp directory. +``` + +The "(probe)" steps are the only network behaviour that changes — they're the ones a proxy registry rejects by returning an empty tag list. Every other step uses per-tag GET / HEAD requests, which is exactly what a proxy registry is good at. + +--- + +## Walk algorithm + +1. Start from the lowest semver literal named in the constraint (e.g. `^1.64.0` starts at `1.64.0`, `>=1.64 <=1.68` starts at `1.64.0`). +2. Increment the patch component by 1 and re-probe. Keep going as long as the registry returns the manifest **and** the version still satisfies the constraint. +3. When a patch step fails, advance to `(major, minor+1, 0)` and probe once. If it exists, resume step 2 from there. +4. When the new-minor probe also fails, advance to `(major+1, 0, 0)` and probe once. If it exists, resume step 2. +5. When the new-major probe also fails (or falls outside the constraint), stop. The bundle contains every tag confirmed during the walk. + +The walk never invents tags: only versions that the registry confirmed are written to the bundle, and the latest-patch-per-minor / inclusive-anchor rules described in [README.MD: Platform Version Filtering](./README.MD#platform-version-filtering) and [README.MD: Module Filtering](./README.MD#module-filtering) are applied to the result the same way they would be after a normal `ListTags`. + +--- + +## Worked example + +Assume the source proxy registry has already cached the following platform release tags (every other tag returns `404 Not Found`): + +``` +v1.64.0, v1.64.1, v1.64.2 +v1.65.0 +v1.66.0, v1.66.1 +``` + +You run: + +```bash +d8 mirror pull /tmp/d8-bundle \ + --source proxy.internal.company.com/deckhouse/ee \ + --license $LICENSE_TOKEN \ + --proxy-registry \ + --include-platform ">=1.64.0 <=1.68.0" \ + --no-modules +``` + +The platform probe issues the following HEAD requests in order. The right column shows the state transition the algorithm makes after each result. + +| # | HEAD request | Result | Probe state after | +|---|------------------------------------------------|----------|------------------------------------------------------| +| 1 | `release-channel:v1.64.0` | 200 OK | append v1.64.0, continue patch loop at 1.64.1 | +| 2 | `release-channel:v1.64.1` | 200 OK | append v1.64.1, continue patch loop at 1.64.2 | +| 3 | `release-channel:v1.64.2` | 200 OK | append v1.64.2, continue patch loop at 1.64.3 | +| 4 | `release-channel:v1.64.3` | 404 | patch loop ends; jump to next minor (1.65.0) | +| 5 | `release-channel:v1.65.0` | 200 OK | append v1.65.0, resume patch loop at 1.65.1 | +| 6 | `release-channel:v1.65.1` | 404 | patch loop ends; jump to next minor (1.66.0) | +| 7 | `release-channel:v1.66.0` | 200 OK | append v1.66.0, resume patch loop at 1.66.1 | +| 8 | `release-channel:v1.66.1` | 200 OK | append v1.66.1, continue patch loop at 1.66.2 | +| 9 | `release-channel:v1.66.2` | 404 | patch loop ends; jump to next minor (1.67.0) | +| 10 | `release-channel:v1.67.0` | 404 | new-minor probe failed; jump to next major (2.0.0) | +| 11 | (skipped) `v2.0.0` is outside `<=1.68.0` | n/a | constraint excludes 2.0.0; **probe terminates** | + +After the probe finishes, the downstream pipeline keeps only the highest patch per `(major, minor)` (so `v1.64.0` and `v1.64.1` are dropped because `v1.64.2` is newer in the same minor). The final platform set written to `platform.tar` is therefore: + +``` +v1.64.2, v1.65.0, v1.66.1 +``` + +…plus any version pinned by an existing release channel snapshot (alpha/beta/etc.) that also satisfies the constraint. + +Note that `v1.67.x` and `v1.68.x` would have been pulled too if the proxy registry had cached them — the probe asks about `v1.67.0` in step 10 and stops only because the answer is `404`. This is intentional: the proxy registry is the source of truth about which tags it can actually serve, and the probe's job is to faithfully reproduce that subset. + +--- + +## What "exists" and "not found" mean on the wire + +The probe relies on the standard registry-v2 manifest endpoint: + +``` +HEAD /v2//manifests/ +``` + +The mapping from HTTP response to probe action is: + +| HTTP response | Treated as | Probe action | +|----------------|------------|---------------------------------------------------| +| `200 OK` | Tag exists | Append to bundle, advance patch by 1 | +| `404 Not Found` (incl. `MANIFEST_UNKNOWN` body) | Tag absent | End patch loop, fall through to next minor / major | +| `401 Unauthorized`, `403 Forbidden` | Auth failure | Abort the entire pull with the error | +| `5xx`, network error, timeout | Real failure | Abort the entire pull with the error | + +In other words: only an unambiguous "the registry does not have this tag" stops the probe — everything else is propagated so a transient network blip never gets silently mistaken for "release series ended". This is the same error policy used by `CheckImageExists` in the rest of the pull pipeline. + +If a proxy registry returns `200 OK` for tags it later refuses to serve the manifest of, the per-tag GET in the normal pull step (step 3d / 6d of the flow) will surface a clear error against that exact tag. + +--- + +## Per-component behaviour in proxy mode + +| Component | Default mode | `--proxy-registry` mode | +|-----------|--------------|--------------------------| +| Source `_catalog` | Not used (CLI pulls per-repo) | Not used | +| Modules root `ListTags` (used to enumerate the module catalog) | One catalog call lists every module | **Skipped.** Module names come from `--include-module` directly | +| Platform release-channel `ListTags` | One catalog call lists every release tag | **Skipped.** Tags come from the forward-probe walk seeded by `--include-platform` | +| Per-module `ListTags` | One catalog call per module lists tag history | **Skipped.** Tags come from the forward-probe walk seeded by `--include-module@` | +| Per-channel `GetImage` / `GetMetadata` | Per-tag GET, works on proxies | **Unchanged** — still used | +| Per-tag manifest `GET` (during actual pull) | Per-tag GET, works on proxies | **Unchanged** — still used | +| Per-tag manifest `HEAD` (CheckImageExists) | Used opportunistically (e.g. LTS-channel existence, VEX detection) | **Used as the probe primitive** | +| Internal digests (`images_digests.json`) | Pulled by digest reference | **Unchanged** — proxy registries serve digest pulls fine | +| Extra images (`extra_images.json`) | Per-tag GETs after parsing the JSON | **Unchanged** | +| VEX attestations (`.att` tags) | Existence check + per-tag GET | **Unchanged** | + +The reason this works: every operation other than the three `ListTags` calls is already a per-resource HTTP request, which a proxy registry handles by either serving from its cache or forwarding to the upstream registry on demand. + +--- + +## Required flag combinations + +| Other flags | Required with `--proxy-registry` | +|------------|----------------------------------| +| platform is being pulled (default) | `--include-platform ` | +| `--no-platform` is set | `--include-platform` is **not** required | +| modules are being pulled (default) or `--only-extra-images` | At least one `--include-module @`. Every entry **must** include `@` — `--include-module foo` alone is rejected because the probe would otherwise start at `v0.0.0` and silently miss every real tag | +| `--no-modules` is set | `--include-module` is **not** required | +| `--exclude-module` | Honoured (subtracts from the include list) | +| `--deckhouse-tag` | **Conflict**: a single pinned tag is already a direct check; do not combine with `--proxy-registry` | +| `--since-version` | **Conflict**: `--since-version` has no upper bound and the probe cannot terminate. Use `--include-platform` with an explicit range instead | +| `--dry-run` | Honoured — runs the probe and prints the plan without downloading any blobs | + +--- + +## Performance characteristics + +Each probe step is one `HEAD` to the source registry, costing on the order of one round-trip. The total number of probe requests for a single component is therefore roughly: + +``` +probe_requests ≈ (# of patches actually present) + (# of "patch series ended" gaps) + (# of "minor series ended" gaps) + 1 +``` + +In practice, for a constraint like `>=1.64.0 <=1.68.0` against a registry that holds all five minors with a handful of patches each, the platform probe issues somewhere between 20 and 40 HEAD requests before terminating. Per-module probes are similar in shape but smaller — most modules have far fewer historical versions than the platform itself. + +If you have lots of modules listed in `--include-module`, the per-module probes run sequentially in the existing loop (one module at a time, same as the regular pull). The dominant cost of the pull is still the actual image data transfer, not the probe. + +--- + +## Limitations and known caveats + +1. **Sparse patch ranges:** the probe stops at the first missing patch in a series. If a registry's cache has `v1.64.0`, `v1.64.2`, `v1.64.5` but not `v1.64.1`, the probe will capture `v1.64.0` and then jump to `v1.65.0` because `v1.64.1` is missing. Patches `v1.64.2` and `v1.64.5` are not retried in proxy mode. Pre-warm the cache before pulling, or use `--include-platform "=v1.64.5"` (exact-tag form) to mirror a specific patch unconditionally. +2. **Sparse minor ranges:** the same logic applies to minors. If the cache has `v1.64.x` and `v1.66.x` but no `v1.65.x` at all, the new-minor probe at `v1.65.0` fails, the new-major probe at `v2.0.0` fails, and the walk terminates — `v1.66.x` is missed. Use a tighter constraint range per minor, or warm the missing minor in the cache first. +3. **Pre-release versions (`v1.65.0-rc.1`) are never probed:** the walk increments only the `(major, minor, patch)` triple. If your proxy holds only `-rc.x` tags for a particular series, pull them directly via `--include-platform "=v1.65.0-rc.1"`. +4. **Custom non-semver tags are not probed:** the probe is semver-only by construction. Use the exact-tag form (`--include-platform "=customtag"` or `--include-module name@=customtag`) to pull non-semver tags from a proxy. +5. **The probe cost grows with constraint width:** a very wide constraint like `>=0.0.0 <100.0.0` will probe a lot of empty (major, minor) combinations before terminating. Always supply a realistic lower bound in `--include-platform` / `--include-module`. +6. **No fallback to `ListTags`:** if `--proxy-registry` is set and a particular registry path actually supports the catalog API, the catalog is still ignored. Drop the flag for that pull if you want to use catalog-based discovery. + +--- + +## Examples + +```bash +# Pull a platform window and one module from a proxy registry +d8 mirror pull /tmp/d8-bundle \ + --source proxy.internal.company.com/deckhouse/ee \ + --license $LICENSE_TOKEN \ + --proxy-registry \ + --include-platform ">=1.64.0 <=1.68.0" \ + --include-module prometheus@^1.0.0 + +# Modules only — skip the platform probe entirely +d8 mirror pull /tmp/d8-bundle \ + --source proxy.internal.company.com/deckhouse/ee \ + --license $LICENSE_TOKEN \ + --proxy-registry \ + --no-platform \ + --include-module prometheus@^1.0.0 \ + --include-module ingress-nginx@^1.5.0 + +# Platform only — skip modules entirely +d8 mirror pull /tmp/d8-bundle \ + --source proxy.internal.company.com/deckhouse/ee \ + --license $LICENSE_TOKEN \ + --proxy-registry \ + --no-modules \ + --include-platform "^1.64.0" + +# Validate the proxy-mode plan before doing any actual pulling +d8 mirror pull /tmp/d8-bundle \ + --source proxy.internal.company.com/deckhouse/ee \ + --license $LICENSE_TOKEN \ + --proxy-registry \ + --include-platform ">=1.64.0 <=1.68.0" \ + --include-module prometheus@^1.0.0 \ + --dry-run +``` diff --git a/internal/mirror/README.MD b/internal/mirror/README.MD index b0181bdd..49b0923a 100644 --- a/internal/mirror/README.MD +++ b/internal/mirror/README.MD @@ -84,6 +84,12 @@ d8 mirror pull [flags] | `--force` | | Overwrite existing bundle packages if they conflict with current pull operation | | `--no-pull-resume` | | Do not continue last unfinished pull operation; start from scratch | +#### Proxy/Cache Registry Discovery + +| Flag | Description | +|------|-------------| +| `--proxy-registry` | Pull from a proxy/caching registry that does not implement the registry catalog API. Discovers tags by probing individual semver versions instead of listing. Requires `--include-platform` and/or `--include-module`. See [Proxy Registry Mode](#proxy-registry-mode) for a summary and [PROXY-REGISTRY.md](./PROXY-REGISTRY.md) for the full reference. Cannot be combined with `--deckhouse-tag` or `--since-version` | + #### Connection Options | Flag | Description | @@ -139,6 +145,27 @@ d8 mirror pull /tmp/d8-bundle \ --- +### Proxy Registry Mode + +`--proxy-registry` adapts the pull command to source registries that act as a transparent proxy or cache in front of another registry and do **not** implement the registry catalog API (`/v2/_catalog`, `/v2//tags/list`). Instead of calling `ListTags`, the pull walks individual semver tags forward from a starting point you supply via `--include-platform` / `--include-module` and probes each one with a single manifest `HEAD` request. + +Minimal example: + +```bash +d8 mirror pull /tmp/d8-bundle \ + --source proxy.internal.company.com/deckhouse/ee \ + --license $LICENSE_TOKEN \ + --proxy-registry \ + --include-platform ">=1.64.0 <=1.68.0" \ + --include-module prometheus@^1.0.0 +``` + +Full documentation — end-to-end pipeline diagram, walk algorithm, worked example with a step-by-step HEAD-request trace, HTTP response handling, per-component behaviour table, required flag combinations, performance characteristics, known limitations and more examples — lives in a dedicated document: + +**→ [PROXY-REGISTRY.md](./PROXY-REGISTRY.md)** + +--- + ### Module Filtering The `--include-module` and `--exclude-module` flags support version constraints for fine-grained control over which module versions to include. @@ -494,6 +521,8 @@ By default, the platform service discovers releases between the current `rock-so `--include-platform` replaces this window with a user-supplied semver constraint, applying the same latest-patch-per-minor and inclusive-anchor rules as module version filtering. Channel snapshots outside the constraint are pruned so the bundle stays internally consistent. +`--proxy-registry` replaces the catalog-based `ListTags` call entirely with a sequential forward-probe walk seeded from the constraint's lower bound. The latest-patch-per-minor and inclusive-anchor rules still apply to the result; only the way the candidate tag list is obtained changes. Full reference: [PROXY-REGISTRY.md](./PROXY-REGISTRY.md). + ### Module Filtering Logic - **Whitelist Mode:** Only specified modules are included diff --git a/internal/mirror/cmd/pull/flags/flags.go b/internal/mirror/cmd/pull/flags/flags.go index 7959523e..45e322be 100644 --- a/internal/mirror/cmd/pull/flags/flags.go +++ b/internal/mirror/cmd/pull/flags/flags.go @@ -74,6 +74,16 @@ var ( OnlyExtraImages bool SkipVexImages bool + // ProxyRegistry switches platform/module release discovery from the + // catalog-based ListTags path to a sequential probe of explicit + // version tags. It exists for proxy/caching registries that do NOT + // implement the registry catalog API but DO serve manifests for tags + // they have cached. Requires --include-platform and/or --include-module + // so the probe has a defined entry point — without those flags the + // probe would have to start from 0.0.0 and the bundle would always + // come back empty. + ProxyRegistry bool + DryRun bool MirrorTimeout time.Duration = -1 @@ -257,6 +267,18 @@ module-name@=v1.3.0+stable → exact tag match: include only v1.3.0 and and publ false, "Do not pull VEX images.", ) + flagSet.BoolVar( + &ProxyRegistry, + "proxy-registry", + false, + `Pull from a proxy/caching registry that does not implement the registry catalog API. + +Instead of calling the registry's "list tags" endpoint (which proxy registries typically return empty), this mode probes individual tags by incrementing patch -> minor -> major from the version explicitly named via --include-platform / --include-module. The probe stops once both a new patch and a new minor of the current major fail to resolve, then attempts the next major; if that also fails the probe terminates and downloads what was discovered. + +Requires --include-platform when platform is not skipped via --no-platform, and at least one --include-module when modules are not skipped via --no-modules. --exclude-module and --no-platform are respected. + +Cannot be combined with --deckhouse-tag or --since-version (use --include-platform's lower bound instead).`, + ) flagSet.BoolVar( &DryRun, "dry-run", diff --git a/internal/mirror/cmd/pull/pull.go b/internal/mirror/cmd/pull/pull.go index 2e77c1aa..78a983f9 100644 --- a/internal/mirror/cmd/pull/pull.go +++ b/internal/mirror/cmd/pull/pull.go @@ -101,6 +101,8 @@ func NewCommand() *cobra.Command { pullCmd.MarkFlagsMutuallyExclusive("include-module", "exclude-module") pullCmd.MarkFlagsMutuallyExclusive("include-platform", "deckhouse-tag") pullCmd.MarkFlagsMutuallyExclusive("include-platform", "since-version") + pullCmd.MarkFlagsMutuallyExclusive("proxy-registry", "deckhouse-tag") + pullCmd.MarkFlagsMutuallyExclusive("proxy-registry", "since-version") pullflags.ParseEnvironmentVariables() return pullCmd @@ -297,6 +299,7 @@ func (p *Puller) Execute(ctx context.Context) error { BundleChunkSize: pullflags.ImagesBundleChunkSizeGB * 1000 * 1000 * 1000, Timeout: pullflags.MirrorTimeout, DryRun: pullflags.DryRun, + ProxyRegistry: pullflags.ProxyRegistry, }, logger.Named("pull"), p.logger, diff --git a/internal/mirror/cmd/pull/validation.go b/internal/mirror/cmd/pull/validation.go index 9ee3e3b1..13ec2bb3 100644 --- a/internal/mirror/cmd/pull/validation.go +++ b/internal/mirror/cmd/pull/validation.go @@ -22,6 +22,7 @@ import ( "net/url" "os" "path/filepath" + "strings" "github.com/Masterminds/semver/v3" "github.com/google/go-containerregistry/pkg/name" @@ -39,6 +40,9 @@ func parseAndValidateParameters(_ *cobra.Command, args []string) error { if err = parseAndValidateVersionFlags(); err != nil { return err } + if err = validateProxyRegistryFlag(); err != nil { + return err + } if err = validateImagesBundlePathArg(args); err != nil { return err } @@ -143,6 +147,67 @@ func parseAndValidateVersionFlags() error { return nil } +// validateProxyRegistryFlag enforces the combinations the proxy-registry +// probe needs to work: each component that is actually being pulled (i.e. +// not switched off via --no-platform / --no-modules / --only-extra-images) +// must come with an explicit lower bound, because the probe cannot rely +// on the registry's tag catalog and has to be told where to start +// incrementing from. +// +// Notes: +// - --deckhouse-tag / --since-version conflicts: --deckhouse-tag asks +// for one tag (a tag-existence check is enough, no probe needed) and +// --since-version asks for "everything from X upward without an upper +// bound", which a probe cannot terminate safely. Both should use the +// non-proxy path instead. +// - --exclude-module is allowed but only meaningful when at least one +// --include-module remains, because the probe still needs concrete +// module names to walk. +func validateProxyRegistryFlag() error { + if !pullflags.ProxyRegistry { + return nil + } + + if pullflags.DeckhouseTag != "" { + return errors.New("--proxy-registry cannot be combined with --deckhouse-tag: pulling a single tag does not need a list-based discovery and uses the direct check-tag-exists path already") + } + if pullflags.SinceVersionString != "" { + return errors.New("--proxy-registry cannot be combined with --since-version: --since-version has no upper bound, so the probe cannot terminate. Use --include-platform with an explicit lower bound (and optional upper bound) instead") + } + + needPlatform := !pullflags.NoPlatform + needModules := !pullflags.NoModules || pullflags.OnlyExtraImages + + // At least one component must actually be pulled — otherwise the + // flag is a no-op against a registry that probably already failed + // to satisfy the user. + if !needPlatform && !needModules { + return errors.New("--proxy-registry has nothing to do: both --no-platform and --no-modules are set") + } + + if needPlatform && pullflags.PlatformConstraintString == "" { + return errors.New("--proxy-registry requires --include-platform (or --no-platform to skip platform mirroring): the probe needs an explicit lower bound to start incrementing from") + } + if needModules { + if len(pullflags.ModulesWhitelist) == 0 { + return errors.New("--proxy-registry requires --include-module (or --no-modules to skip module mirroring): the probe needs explicit module names and version anchors to start incrementing from") + } + // Every --include-module entry must come with an explicit + // version part. The implicit ">=0.0.0" fallback used by the + // regular pull mode is poisonous for the probe: it starts at + // v0.0.0 and stops on the first not-found, silently skipping + // any module whose lowest tag is above v0.0.0 / v0.1.0 / + // v1.0.0. Bail out loudly so the user picks a real anchor. + for _, entry := range pullflags.ModulesWhitelist { + if !strings.Contains(entry, "@") { + return fmt.Errorf("--proxy-registry requires every --include-module entry to specify an explicit version constraint (e.g. %q@^1.0.0); without it the probe would start at v0.0.0 and miss everything", strings.TrimSpace(entry)) + } + } + } + + return nil +} + func validateChunkSizeFlag() error { if pullflags.ImagesBundleChunkSizeGB < 0 { return errors.New("Chunk size cannot be less than zero GB") diff --git a/internal/mirror/cmd/pull/validation_test.go b/internal/mirror/cmd/pull/validation_test.go index 7833826b..9cd32218 100644 --- a/internal/mirror/cmd/pull/validation_test.go +++ b/internal/mirror/cmd/pull/validation_test.go @@ -478,6 +478,165 @@ func TestValidationParseAndValidateVersionFlags(t *testing.T) { } } +func TestValidationValidateProxyRegistryFlag(t *testing.T) { + tests := []struct { + name string + proxyRegistry bool + platformConstraintString string + deckhouseTag string + sinceVersionString string + modulesWhitelist []string + noPlatform bool + noModules bool + onlyExtraImages bool + expectError bool + errorMsg string + }{ + { + name: "flag off — every other combination is a no-op", + proxyRegistry: false, + expectError: false, + }, + { + name: "happy path: platform constraint + module whitelist with explicit version", + proxyRegistry: true, + platformConstraintString: ">=1.64.0 <=1.68.0", + modulesWhitelist: []string{"prometheus@^1.0.0"}, + expectError: false, + }, + { + name: "missing --include-platform when platform is on", + proxyRegistry: true, + platformConstraintString: "", + modulesWhitelist: []string{"prometheus@^1.0.0"}, + expectError: true, + errorMsg: "requires --include-platform", + }, + { + name: "missing --include-module when modules are on", + proxyRegistry: true, + platformConstraintString: "^1.64.0", + modulesWhitelist: nil, + expectError: true, + errorMsg: "requires --include-module", + }, + { + name: "--include-module without explicit version is rejected", + proxyRegistry: true, + platformConstraintString: "^1.64.0", + modulesWhitelist: []string{"prometheus"}, + expectError: true, + errorMsg: "explicit version constraint", + }, + { + name: "--no-platform skips the include-platform requirement", + proxyRegistry: true, + noPlatform: true, + modulesWhitelist: []string{"prometheus@^1.0.0"}, + expectError: false, + }, + { + name: "--no-modules skips the include-module requirement", + proxyRegistry: true, + platformConstraintString: "^1.64.0", + noModules: true, + modulesWhitelist: nil, + expectError: false, + }, + { + name: "--no-platform and --no-modules together is a no-op", + proxyRegistry: true, + noPlatform: true, + noModules: true, + expectError: true, + errorMsg: "nothing to do", + }, + { + name: "--only-extra-images still requires --include-module even with --no-modules", + proxyRegistry: true, + platformConstraintString: "^1.64.0", + noModules: true, + onlyExtraImages: true, + modulesWhitelist: nil, + expectError: true, + errorMsg: "requires --include-module", + }, + { + name: "conflicts with --deckhouse-tag", + proxyRegistry: true, + platformConstraintString: "^1.64.0", + modulesWhitelist: []string{"prometheus@^1.0.0"}, + deckhouseTag: "v1.65.0", + expectError: true, + errorMsg: "deckhouse-tag", + }, + { + name: "conflicts with --since-version", + proxyRegistry: true, + platformConstraintString: "^1.64.0", + modulesWhitelist: []string{"prometheus@^1.0.0"}, + sinceVersionString: "1.50.0", + expectError: true, + errorMsg: "since-version", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + // Snapshot global state — these flags are package-level. + originals := struct { + proxyRegistry bool + platformConstraintString string + deckhouseTag string + sinceVersionString string + modulesWhitelist []string + noPlatform bool + noModules bool + onlyExtraImages bool + }{ + proxyRegistry: pullflags.ProxyRegistry, + platformConstraintString: pullflags.PlatformConstraintString, + deckhouseTag: pullflags.DeckhouseTag, + sinceVersionString: pullflags.SinceVersionString, + modulesWhitelist: pullflags.ModulesWhitelist, + noPlatform: pullflags.NoPlatform, + noModules: pullflags.NoModules, + onlyExtraImages: pullflags.OnlyExtraImages, + } + defer func() { + pullflags.ProxyRegistry = originals.proxyRegistry + pullflags.PlatformConstraintString = originals.platformConstraintString + pullflags.DeckhouseTag = originals.deckhouseTag + pullflags.SinceVersionString = originals.sinceVersionString + pullflags.ModulesWhitelist = originals.modulesWhitelist + pullflags.NoPlatform = originals.noPlatform + pullflags.NoModules = originals.noModules + pullflags.OnlyExtraImages = originals.onlyExtraImages + }() + + pullflags.ProxyRegistry = tt.proxyRegistry + pullflags.PlatformConstraintString = tt.platformConstraintString + pullflags.DeckhouseTag = tt.deckhouseTag + pullflags.SinceVersionString = tt.sinceVersionString + pullflags.ModulesWhitelist = tt.modulesWhitelist + pullflags.NoPlatform = tt.noPlatform + pullflags.NoModules = tt.noModules + pullflags.OnlyExtraImages = tt.onlyExtraImages + + err := validateProxyRegistryFlag() + + if tt.expectError { + assert.Error(t, err) + if tt.errorMsg != "" { + assert.Contains(t, err.Error(), tt.errorMsg) + } + } else { + assert.NoError(t, err) + } + }) + } +} + func TestValidationValidateChunkSizeFlag(t *testing.T) { tests := []struct { name string diff --git a/internal/mirror/modules/constraints.go b/internal/mirror/modules/constraints.go index 114a9bdf..457f9e41 100644 --- a/internal/mirror/modules/constraints.go +++ b/internal/mirror/modules/constraints.go @@ -43,6 +43,17 @@ type SemanticVersionConstraint struct { // those operators are shorthand for a range and the patch filter is // free to collapse same-minor patches inside them. anchors []*semver.Version + // lowerBound is the smallest version literal that appears in the + // constraint string, regardless of which operator (^, ~, >=, =, etc.) + // preceded it. It is intended for callers that cannot list registry + // tags directly (proxy registries) and need a starting point to + // enumerate (major, minor, patch) by incrementing from a known + // version that the user has named. + // + // May be nil only when the constraint string contains no recognisable + // version literal, which is rejected by NewSemanticVersionConstraint + // before it reaches consumers. + lowerBound *semver.Version } // anchorOpRegex captures version literals that follow an inclusive boundary @@ -54,6 +65,14 @@ type SemanticVersionConstraint struct { // stored. var anchorOpRegex = regexp.MustCompile(`(?:>=|<=|=>|=<)\s*([^\s,]+)`) +// versionLiteralRegex captures any version literal in the constraint string, +// regardless of the operator preceding it (or absence thereof). Used by +// LowerBound to find a sensible probing entry point when callers can't list +// tags directly (e.g. proxy registries) and need to enumerate versions by +// incrementing patch/minor/major starting from the constraint's lowest +// explicitly-named version. +var versionLiteralRegex = regexp.MustCompile(`v?(\d+)(?:\.(\d+))?(?:\.(\d+))?(?:[-+][0-9A-Za-z.\-]+)?`) + func NewSemanticVersionConstraint(c string) (*SemanticVersionConstraint, error) { constraint, err := semver.NewConstraint(c) if err != nil { @@ -65,9 +84,15 @@ func NewSemanticVersionConstraint(c string) (*SemanticVersionConstraint, error) return nil, fmt.Errorf("invalid semantic version constraint %q: %w", c, err) } + lower, err := extractLowerBound(c) + if err != nil { + return nil, fmt.Errorf("invalid semantic version constraint %q: %w", c, err) + } + return &SemanticVersionConstraint{ constraint: constraint, anchors: anchors, + lowerBound: lower, }, nil } @@ -80,6 +105,62 @@ func (s *SemanticVersionConstraint) Anchors() []*semver.Version { return s.anchors } +// LowerBound returns the smallest version literal found in the constraint +// string. This is the natural starting point for callers that enumerate +// (major, minor, patch) by probing the registry one tag at a time (proxy +// registry mode) — incrementing forward from a version the user has +// explicitly named avoids scanning from v0.0.0. +// +// Returns nil only when the constraint string contained no recognisable +// version literal. NewSemanticVersionConstraint refuses such constraints +// so consumers can treat a nil result as a programming error. +func (s *SemanticVersionConstraint) LowerBound() *semver.Version { + return s.lowerBound +} + +// extractLowerBound walks every version literal that appears in the +// constraint string (regardless of the operator preceding it) and returns +// the smallest one. We deliberately ignore operators here: the goal is a +// safe starting point for forward enumeration, not constraint semantics. +// +// Notes on operator-specific behaviour: +// - `^X.Y.Z` / `~X.Y.Z` / implicit `X.Y.Z`: the only literal is X.Y.Z +// and it is returned as-is. +// - `>=A <=B` (or any range with two literals): the smaller of A and B +// is returned so probing starts at the constraint's lower edge even +// when the user wrote the bounds in a non-canonical order. +// - `>A`: returned as A; the strict-greater semantics belong to Match, +// not to the enumeration start point. +// +// Returns an error only on the (defensive) case where a literal that the +// regex extracted fails semver.NewVersion — which should be impossible +// because semver.NewConstraint already validated the string. +func extractLowerBound(constraintStr string) (*semver.Version, error) { + matches := versionLiteralRegex.FindAllString(constraintStr, -1) + if len(matches) == 0 { + return nil, fmt.Errorf("no version literal found in constraint") + } + + var lowest *semver.Version + for _, raw := range matches { + v, err := semver.NewVersion(raw) + if err != nil { + // Defensive: the regex is permissive but constraint parsing + // already accepted the string upstream. Surface a clear error + // instead of silently dropping the literal. + return nil, fmt.Errorf("version literal %q not parseable: %w", raw, err) + } + if lowest == nil || v.LessThan(lowest) { + lowest = v + } + } + + if lowest == nil { + return nil, fmt.Errorf("no parseable version literal in constraint") + } + return lowest, nil +} + // extractInclusiveAnchors finds every >=X / <=X literal in the constraint // string and parses X with semver.NewVersion. Duplicates are removed. // The returned slice is nil when no inclusive boundary literals are present. diff --git a/internal/mirror/modules/filter.go b/internal/mirror/modules/filter.go index e5950cc4..96884ca1 100644 --- a/internal/mirror/modules/filter.go +++ b/internal/mirror/modules/filter.go @@ -18,6 +18,7 @@ package modules import ( "fmt" + "sort" "strings" "github.com/Masterminds/semver/v3" @@ -103,6 +104,30 @@ func (f *Filter) GetConstraint(moduleName string) (VersionConstraint, bool) { return constraint, found } +// IsWhitelist reports whether the filter is operating in whitelist mode. +// It exists so the modules service can take alternate code paths that +// only make sense when a finite, user-supplied module list is available +// (e.g. proxy-registry probing, which has no module catalog to enumerate). +func (f *Filter) IsWhitelist() bool { + return f._type == FilterTypeWhitelist +} + +// ModuleNames returns the names registered with the filter in +// deterministic insertion-agnostic order (sorted). For a whitelist +// filter this is exactly the set of modules the user named with +// --include-module; for a blacklist filter it is the set the user +// asked to exclude. +// +// The slice is freshly allocated so callers may mutate it freely. +func (f *Filter) ModuleNames() []string { + names := make([]string, 0, len(f.modules)) + for name := range f.modules { + names = append(names, name) + } + sort.Strings(names) + return names +} + // ParseVersionConstraint turns a user-supplied constraint string into a // VersionConstraint. The syntax mirrors the `module-name@` body // accepted by --include-module so any consumer (modules filter, platform diff --git a/internal/mirror/modules/modules.go b/internal/mirror/modules/modules.go index 61b90124..665d4380 100644 --- a/internal/mirror/modules/modules.go +++ b/internal/mirror/modules/modules.go @@ -75,6 +75,17 @@ type Options struct { Timeout time.Duration // DryRun prints the pull plan without downloading any image blobs DryRun bool + // ProxyRegistry replaces catalog-based discovery (ListTags of the + // modules root and per-module tag listings) with a sequential probe + // of individual version tags derived from the user's --include-module + // version constraint. It exists for proxy/caching registries that do + // not implement the registry catalog API but DO serve manifests for + // tags they cache. + // + // The CLI requires --include-module with explicit version anchors + // when ProxyRegistry is set so the probe has both a module name list + // and a per-module lower bound to start incrementing from. + ProxyRegistry bool } type Service struct { @@ -160,6 +171,15 @@ func (svc *Service) PullModules(ctx context.Context) error { func (svc *Service) validateModulesAccess(ctx context.Context) error { svc.logger.Debug("Validating access to the modules registry") + // Proxy registries typically refuse the catalog API entirely. + // We deliberately skip the listing access check here — the CLI has + // already required --include-module so we know exactly which + // modules to probe, and per-tag CheckImageExists/GetImage calls + // (used downstream) work fine against a proxy. + if svc.options.ProxyRegistry { + return nil + } + // For specific tags, check if the tag exists _, err := svc.modulesService.ListTags(ctx) if errors.Is(err, client.ErrImageNotFound) { @@ -188,10 +208,15 @@ func (svc *Service) pullModules(ctx context.Context) error { // - stores intermediate pulled images; final bundle is packed later. tmpDir := filepath.Join(svc.workingDir, "modules") - // List all available modules - moduleNames, err := svc.modulesService.ListTags(ctx) + // Pick the module-name discovery path: + // - proxy registries can't enumerate the modules catalog, but the CLI + // has guaranteed --include-module is set, so we read the names + // directly from the whitelist filter and skip the listing call. + // - everywhere else, we discover names from the registry's tag list + // and let the filter prune them. + moduleNames, err := svc.discoverModuleNames(ctx) if err != nil { - return fmt.Errorf("list modules: %w", err) + return err } if len(moduleNames) == 0 { @@ -393,16 +418,60 @@ func (svc *Service) discoverChannelVersions(ctx context.Context, moduleName stri return svc.extractVersionsFromReleaseChannels(ctx, moduleName), nil } +// discoverModuleNames returns the list of module names this run should +// consider. The behaviour depends on whether the registry can be +// enumerated: +// +// - Default path: ListTags on the modules root returns every module +// the registry exposes. The filter (whitelist/blacklist) then prunes +// the result downstream. +// - Proxy-registry path: the catalog API is assumed unavailable. The +// CLI guarantees a whitelist filter is set, so the module names come +// straight from --include-module — we never call the registry here. +// +// Both paths return an empty slice (no error) when nothing is to be +// pulled; the caller prints the standard "modules were not found" +// warning so a misconfigured source still surfaces clearly. +func (svc *Service) discoverModuleNames(ctx context.Context) ([]string, error) { + if svc.options.ProxyRegistry { + if svc.options.Filter == nil || !svc.options.Filter.IsWhitelist() { + // Defensive: validation should have rejected this combination + // (--proxy-registry needs --include-module). Surface a clear + // error so a future refactor of the CLI layer doesn't silently + // degrade into a no-op pull. + return nil, fmt.Errorf("--proxy-registry requires a whitelist of modules (--include-module)") + } + return svc.options.Filter.ModuleNames(), nil + } + + moduleNames, err := svc.modulesService.ListTags(ctx) + if err != nil { + return nil, fmt.Errorf("list modules: %w", err) + } + return moduleNames, nil +} + // listTagsIfConstrained returns the module's tag list, but only when the filter // has a non-exact (semver) constraint - exact-tag and no-constraint paths don't // read Releases. ErrImageNotFound is logged and treated as no tags (same policy // as validateModulesAccess for missing module repos). +// +// In --proxy-registry mode the per-module catalog is unavailable, so we +// probe the tags one by one with a forward semver walk seeded from the +// constraint's lower bound (see ProbeAvailableVersions). The probe only +// surfaces tags the registry actually serves, so the downstream +// Filter.VersionsToMirror logic continues to apply its +// latest-patch-per-minor and inclusive-anchor rules unmodified. func (svc *Service) listTagsIfConstrained(ctx context.Context, moduleName string) ([]string, error) { constraint, hasConstraint := svc.options.Filter.GetConstraint(moduleName) if !hasConstraint || constraint.IsExact() { return nil, nil } + if svc.options.ProxyRegistry { + return svc.probeModuleTags(ctx, moduleName, constraint) + } + tags, err := svc.modulesService.Module(moduleName).ListTags(ctx) switch { case errors.Is(err, client.ErrImageNotFound): @@ -414,6 +483,48 @@ func (svc *Service) listTagsIfConstrained(ctx context.Context, moduleName string return tags, nil } +// probeModuleTags walks tags for a single module via HEAD requests +// instead of asking the registry to enumerate them. Only versions that +// the registry actually serves AND that satisfy the user's constraint +// are returned — the rest of the pull pipeline is unchanged. +// +// The constraint must be a SemanticVersionConstraint here: exact-tag +// constraints are filtered out by listTagsIfConstrained before reaching +// this function, and a future constraint type would need its own +// proxy-aware code path. +func (svc *Service) probeModuleTags(ctx context.Context, moduleName string, constraint VersionConstraint) ([]string, error) { + semverConstraint, ok := constraint.(*SemanticVersionConstraint) + if !ok { + // Be loud — silently falling back to ListTags here would defeat + // the whole purpose of --proxy-registry on a registry that + // refuses catalog access. + return nil, fmt.Errorf("module %s: --proxy-registry only supports semver-style constraints, got %T", moduleName, constraint) + } + + check := func(ctx context.Context, v *semver.Version) (bool, error) { + tag := "v" + v.String() + err := svc.modulesService.Module(moduleName).CheckImageExists(ctx, tag) + if err == nil { + return true, nil + } + if errors.Is(err, client.ErrImageNotFound) { + return false, nil + } + return false, fmt.Errorf("check module %s tag %q: %w", moduleName, tag, err) + } + + versions, err := ProbeAvailableVersions(ctx, semverConstraint, check) + if err != nil { + return nil, fmt.Errorf("probe tags for module %s: %w", moduleName, err) + } + + tags := make([]string, 0, len(versions)) + for _, v := range versions { + tags = append(tags, "v"+v.String()) + } + return tags, nil +} + // mergeAndDedupeVersions merges channel-derived versions with versions resolved // from filter constraints over the given tags, then deduplicates. func (svc *Service) mergeAndDedupeVersions(moduleName, registryPath string, channelVersions, tags []string) []string { diff --git a/internal/mirror/modules/probe.go b/internal/mirror/modules/probe.go new file mode 100644 index 00000000..4402eab9 --- /dev/null +++ b/internal/mirror/modules/probe.go @@ -0,0 +1,162 @@ +/* +Copyright 2026 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package modules + +import ( + "context" + "errors" + "fmt" + + "github.com/Masterminds/semver/v3" +) + +// ProbeChecker reports whether a specific semver tag is served by the +// registry. The boolean MUST be false (with a nil error) when the tag is +// definitively absent — this is the signal the probe uses to decide that a +// patch series has ended. A non-nil error indicates a real failure +// (network, auth, ...) and aborts the probe. +// +// Callers integrating with a registry client typically translate the +// client's "image not found" sentinel to (false, nil) and propagate all +// other errors. +type ProbeChecker func(ctx context.Context, version *semver.Version) (bool, error) + +// ErrProbeNoLowerBound is returned when a constraint without a parseable +// lower bound is handed to ProbeAvailableVersions. Callers should +// surface this to the user as "your constraint needs an explicit +// version anchor (e.g. ^1.65.0, ~1.65.0, or >=1.65.0)" rather than +// silently scanning from v0.0.0. +var ErrProbeNoLowerBound = errors.New("constraint has no parseable lower-bound version literal — probing requires an explicit starting point") + +// ProbeAvailableVersions enumerates registry tags by walking semver +// versions starting from the constraint's lower bound. It exists for +// proxy/caching registries that do NOT expose the registry catalog API +// but DO serve manifests for tags they cache. +// +// Walk rules (intentionally identical to the user-facing description): +// +// 1. From the lower-bound (M, m, p), increment patch one step at a time. +// Each step is tested with `check` and (when present and matching the +// constraint) appended to the result. The patch series ends as soon +// as a step does not exist OR does not satisfy the constraint. +// 2. When the patch series ends, advance one step to (M, m+1, 0) and +// retry. If that exists, resume rule 1 from there; if not, fall +// through to rule 3. +// 3. When the new-minor step also fails, advance to (M+1, 0, 0). If +// that exists, resume rule 1 from there; if not, terminate the probe. +// +// The probe never invents a tag — every appended version was confirmed +// by the registry. It also never widens past the constraint: a version +// that the constraint does not accept is treated the same as a missing +// tag for rule-1 purposes, and the rule-2 / rule-3 jump points are +// also constraint-gated so the probe can terminate cleanly inside a +// bounded range like ">=1.64 <=1.68". +// +// Context cancellation is honoured between every probe step so a +// proxy registry that hangs on a single HEAD request does not block +// shutdown indefinitely. +func ProbeAvailableVersions( + ctx context.Context, + constraint *SemanticVersionConstraint, + check ProbeChecker, +) ([]*semver.Version, error) { + if constraint == nil { + return nil, errors.New("probe requires a non-nil constraint") + } + if check == nil { + return nil, errors.New("probe requires a non-nil checker") + } + + start := constraint.LowerBound() + if start == nil { + return nil, ErrProbeNoLowerBound + } + + major, minor, patch := start.Major(), start.Minor(), start.Patch() + found := make([]*semver.Version, 0) + + // probeOne tests (major, minor, patch) against constraint+registry and, + // on success, appends the version and advances patch by 1. The boolean + // return ("advanced") tells the outer loop whether to keep walking + // patches or to fall through to the minor/major lookahead. + probeOne := func() (bool, error) { + if err := ctx.Err(); err != nil { + return false, err + } + + v := semver.MustParse(fmt.Sprintf("%d.%d.%d", major, minor, patch)) + if !constraint.Match(v) { + return false, nil + } + + exists, err := check(ctx, v) + if err != nil { + return false, err + } + if !exists { + return false, nil + } + + found = append(found, v) + patch++ + return true, nil + } + + for { + // Rule 1: walk patches until one fails (or context is cancelled). + for { + advanced, err := probeOne() + if err != nil { + return nil, err + } + if !advanced { + break + } + } + + // Rule 2: try the next minor with patch reset to 0. If it lands + // on an existing+matching version, we resume rule 1 from the + // patch right after it. + minor++ + patch = 0 + advanced, err := probeOne() + if err != nil { + return nil, err + } + if advanced { + continue + } + + // Rule 3: rule 2 didn't pan out. Try the next major. + major++ + minor = 0 + patch = 0 + advanced, err = probeOne() + if err != nil { + return nil, err + } + if advanced { + continue + } + + // Both lookaheads failed and we've already drained the patch + // loop above — terminate. + break + } + + return found, nil +} diff --git a/internal/mirror/modules/probe_test.go b/internal/mirror/modules/probe_test.go new file mode 100644 index 00000000..4175fac2 --- /dev/null +++ b/internal/mirror/modules/probe_test.go @@ -0,0 +1,238 @@ +/* +Copyright 2026 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package modules + +import ( + "context" + "errors" + "testing" + + "github.com/Masterminds/semver/v3" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestSemanticVersionConstraint_LowerBound(t *testing.T) { + tests := []struct { + name string + constraint string + wantLowerBound string + }{ + { + name: "implicit caret", + constraint: "^1.65.0", + wantLowerBound: "1.65.0", + }, + { + name: "tilde", + constraint: "~1.65.0", + wantLowerBound: "1.65.0", + }, + { + name: "range with two inclusive anchors", + constraint: ">=1.64.0 <=1.68.0", + wantLowerBound: "1.64.0", + }, + { + name: "range with greater-than (no equals) lower bound", + constraint: ">1.0.0 <=1.5.0", + wantLowerBound: "1.0.0", + }, + { + name: "lower bound smallest even when written in reversed order", + constraint: "<=1.68.0 >=1.64.0", + wantLowerBound: "1.64.0", + }, + { + name: "constraint with v-prefixed version literal", + constraint: ">=v1.50.0 <2.0.0", + wantLowerBound: "1.50.0", + }, + { + name: "major-only literal expands to .0.0", + constraint: ">=1.64 <=1.68", + wantLowerBound: "1.64.0", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + c, err := NewSemanticVersionConstraint(tt.constraint) + require.NoError(t, err) + require.NotNil(t, c.LowerBound()) + assert.Equal(t, tt.wantLowerBound, c.LowerBound().String()) + }) + } +} + +// fakeRegistry exposes a deterministic version set to the probe so test +// expectations can match exact tag traversals. +type fakeRegistry struct { + have map[string]struct{} + calls []string + err error +} + +func newFakeRegistry(versions ...string) *fakeRegistry { + have := make(map[string]struct{}, len(versions)) + for _, v := range versions { + have[v] = struct{}{} + } + return &fakeRegistry{have: have} +} + +func (r *fakeRegistry) check(_ context.Context, v *semver.Version) (bool, error) { + if r.err != nil { + return false, r.err + } + r.calls = append(r.calls, v.String()) + _, ok := r.have[v.String()] + return ok, nil +} + +func TestProbeAvailableVersions(t *testing.T) { + // wantCalls reflects ONLY versions that pass constraint.Match — the + // probe short-circuits out-of-constraint candidates before touching + // the registry, which is the whole point of having a bounded probe. + tests := []struct { + name string + constraint string + registryHas []string + wantVersions []string + wantCalls []string + }{ + { + name: "walk full patch series then stop on missing minor and out-of-range major", + constraint: "^1.64.0", + registryHas: []string{"1.64.0", "1.64.1", "1.64.2"}, + wantVersions: []string{"1.64.0", "1.64.1", "1.64.2"}, + wantCalls: []string{ + "1.64.0", "1.64.1", "1.64.2", "1.64.3", // patch ends on missing + "1.65.0", // next minor — missing + // next major 2.0.0 is outside ^1.64.0; no registry call made + }, + }, + { + name: "skip a missing starting patch but pick up the next minor", + constraint: ">=1.64.0 <2.0.0", + registryHas: []string{"1.65.0", "1.65.1"}, + wantVersions: []string{"1.65.0", "1.65.1"}, + wantCalls: []string{ + "1.64.0", // missing — falls into "try next minor" + "1.65.0", // next minor — found, resume patch + "1.65.1", // patch continues + "1.65.2", // patch ends on missing + "1.66.0", // next minor — missing + // 2.0.0 outside constraint; not probed + }, + }, + { + name: "respects upper bound from inclusive range", + constraint: ">=1.64.0 <=1.65.0", + registryHas: []string{"1.64.0", "1.64.1", "1.65.0", "1.65.1", "1.65.2"}, + wantVersions: []string{"1.64.0", "1.64.1", "1.65.0"}, + wantCalls: []string{ + "1.64.0", "1.64.1", "1.64.2", // patch ends on missing + "1.65.0", // next minor — found, resume patch + // 1.65.1 fails constraint (<=1.65.0); no registry call. + // next minor / major also outside constraint; no calls. + }, + }, + { + name: "advances across major boundary when minor lookahead fails", + constraint: ">=1.0.0 <3.0.0", + registryHas: []string{"1.0.0", "2.0.0", "2.0.1"}, + wantVersions: []string{"1.0.0", "2.0.0", "2.0.1"}, + wantCalls: []string{ + "1.0.0", "1.0.1", // patch ends + "1.1.0", // next minor — missing + "2.0.0", // next major — found, resume patch + "2.0.1", "2.0.2", // patch ends + "2.1.0", // next minor — missing + // 3.0.0 outside constraint; no probe call + }, + }, + { + name: "empty when nothing exists in either jump", + constraint: "^1.99.0", + registryHas: nil, + wantVersions: []string{}, + wantCalls: []string{ + "1.99.0", // patch ends on missing + "1.100.0", // next minor — missing (still in ^1.99.0 = <2.0.0) + // 2.0.0 outside ^1.99.0; no probe call + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + c, err := NewSemanticVersionConstraint(tt.constraint) + require.NoError(t, err) + + reg := newFakeRegistry(tt.registryHas...) + + got, err := ProbeAvailableVersions(context.Background(), c, reg.check) + require.NoError(t, err) + + gotStrs := make([]string, 0, len(got)) + for _, v := range got { + gotStrs = append(gotStrs, v.String()) + } + assert.Equal(t, tt.wantVersions, gotStrs, "discovered versions mismatch") + assert.Equal(t, tt.wantCalls, reg.calls, "probe call sequence mismatch") + }) + } +} + +func TestProbeAvailableVersions_PropagatesRegistryError(t *testing.T) { + c, err := NewSemanticVersionConstraint("^1.0.0") + require.NoError(t, err) + + sentinel := errors.New("registry down") + reg := &fakeRegistry{err: sentinel} + + _, err = ProbeAvailableVersions(context.Background(), c, reg.check) + require.ErrorIs(t, err, sentinel) +} + +func TestProbeAvailableVersions_RespectsContextCancellation(t *testing.T) { + c, err := NewSemanticVersionConstraint("^1.0.0") + require.NoError(t, err) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + _, err = ProbeAvailableVersions(ctx, c, func(_ context.Context, _ *semver.Version) (bool, error) { + // Should never reach the registry — context is already cancelled. + t.Fatal("checker called after context cancellation") + return false, nil + }) + require.ErrorIs(t, err, context.Canceled) +} + +func TestProbeAvailableVersions_RejectsNilArguments(t *testing.T) { + _, err := ProbeAvailableVersions(context.Background(), nil, func(_ context.Context, _ *semver.Version) (bool, error) { + return false, nil + }) + require.Error(t, err) + + c, err := NewSemanticVersionConstraint("^1.0.0") + require.NoError(t, err) + _, err = ProbeAvailableVersions(context.Background(), c, nil) + require.Error(t, err) +} diff --git a/internal/mirror/platform/platform.go b/internal/mirror/platform/platform.go index d3d25e19..b9ab1dcc 100644 --- a/internal/mirror/platform/platform.go +++ b/internal/mirror/platform/platform.go @@ -79,6 +79,17 @@ type Options struct { Timeout time.Duration // DryRun prints the pull plan without downloading any image blobs DryRun bool + // ProxyRegistry replaces catalog-based release discovery (ListTags + // of the deckhouse release-channel repo) with a sequential probe of + // individual version tags derived from IncludeConstraint. It exists + // for proxy/caching registries that do not implement the registry + // catalog API but DO serve manifests for tags they cache. + // + // The CLI requires IncludeConstraint when ProxyRegistry is set so + // the probe has a defined starting point — without a lower bound + // from the user the probe would have to start from 0.0.0 and would + // never find anything. + ProxyRegistry bool } type Service struct { @@ -516,14 +527,11 @@ func (svc *Service) expandVersionRange(ctx context.Context, channelVersions chan // upper bounds, so we ignore rock-solid/alpha endpoints entirely. We still // honour --since-version (when above the constraint's lower bound) to // preserve the existing knob without surprising the user. - svc.userLogger.Debugf("listing deckhouse releases for --include-platform") - - allTags, err := svc.deckhouseService.ReleaseChannels().ListTags(ctx) + matched, err := svc.discoverConstrainedPlatformVersions(ctx, semverConstraint) if err != nil { - return nil, fmt.Errorf("get tags from Deckhouse registry: %w", err) + return nil, err } - matched := parseTagsMatchingConstraint(allTags, semverConstraint) if since := svc.options.SinceVersion; since != nil { nb := make([]*semver.Version, 0, len(matched)) for _, v := range matched { @@ -541,6 +549,57 @@ func (svc *Service) expandVersionRange(ctx context.Context, channelVersions chan return append(baseVersions, selected...), nil } +// discoverConstrainedPlatformVersions returns every registry-served +// platform version that satisfies the user's --include-platform +// constraint. By default it pulls the full release-channel tag list and +// filters it locally, which is fast but requires the source registry to +// implement the catalog API. +// +// When --proxy-registry is set, the registry is treated as a +// proxy/cache that does NOT implement the catalog API: we synthesise +// the same set of versions by walking semver tags one HEAD-request at +// a time via modules.ProbeAvailableVersions, starting from the +// constraint's lower bound. The behaviour at this boundary is +// equivalent to listing for any registry that fully populates its +// catalog API; for proxies that don't, the probe is the only thing +// that can find anything at all. +func (svc *Service) discoverConstrainedPlatformVersions(ctx context.Context, semverConstraint *modules.SemanticVersionConstraint) ([]*semver.Version, error) { + if svc.options.ProxyRegistry { + svc.userLogger.Debugf("probing deckhouse releases for --include-platform via --proxy-registry") + matched, err := modules.ProbeAvailableVersions(ctx, semverConstraint, svc.releaseTagExists) + if err != nil { + return nil, fmt.Errorf("probe deckhouse releases: %w", err) + } + return matched, nil + } + + svc.userLogger.Debugf("listing deckhouse releases for --include-platform") + + allTags, err := svc.deckhouseService.ReleaseChannels().ListTags(ctx) + if err != nil { + return nil, fmt.Errorf("get tags from Deckhouse registry: %w", err) + } + + return parseTagsMatchingConstraint(allTags, semverConstraint), nil +} + +// releaseTagExists adapts the release-channel CheckImageExists call to +// the modules.ProbeChecker signature. It maps the registry client's +// "not found" sentinel to (false, nil) so the probe treats it as +// "patch series ended" instead of aborting; every other error +// (network, auth, unexpected status) propagates out to fail the pull. +func (svc *Service) releaseTagExists(ctx context.Context, v *semver.Version) (bool, error) { + tag := "v" + v.String() + err := svc.deckhouseService.ReleaseChannels().CheckImageExists(ctx, tag) + if err == nil { + return true, nil + } + if errors.Is(err, client.ErrImageNotFound) { + return false, nil + } + return false, fmt.Errorf("check release tag %q: %w", tag, err) +} + // parseTagsMatchingConstraint walks the registry's tag list, drops anything // that is not a valid semver, and keeps only versions that satisfy the // constraint. Returning *semver.Version (rather than tag strings) lets the diff --git a/internal/mirror/pull.go b/internal/mirror/pull.go index 663d46f8..08cec912 100644 --- a/internal/mirror/pull.go +++ b/internal/mirror/pull.go @@ -64,6 +64,12 @@ type PullServiceOptions struct { Timeout time.Duration // DryRun prints the pull plan without downloading any image blobs DryRun bool + // ProxyRegistry switches platform/module discovery from a single + // catalog ListTags call (which proxy registries typically return + // empty for) to a sequential probe of explicit version tags. The + // CLI guarantees that --include-platform and/or --include-module + // are supplied so the probe has a defined starting point. + ProxyRegistry bool } type PullService struct { @@ -112,6 +118,7 @@ func NewPullService( SkipVexImages: options.SkipVexImages, Timeout: options.Timeout, DryRun: options.DryRun, + ProxyRegistry: options.ProxyRegistry, }, logger, userLogger, @@ -139,6 +146,7 @@ func NewPullService( BundleChunkSize: options.BundleChunkSize, Timeout: options.Timeout, DryRun: options.DryRun, + ProxyRegistry: options.ProxyRegistry, }, logger, userLogger,