Skip to content

Add drift check for stale OpenStack source refs - #2642

Draft
ideaship wants to merge 3 commits into
mainfrom
feature/drift-source-ref-phase
Draft

Add drift check for stale OpenStack source refs#2642
ideaship wants to merge 3 commits into
mainfrom
feature/drift-source-ref-phase

Conversation

@ideaship

@ideaship ideaship commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Detects the silent failure that left OSISM's 2024.1 images on October 2025
sources for nine months.

Upstream renames stable/<release> to unmaintained/<release> when a release
enters the unmaintained phase, and deletes the branch at EOL.
tarballs.opendev.org keeps serving the artifact for a retired branch name but
stops regenerating it, so a build that still asks for
<project>-stable-<release>.tar.gz keeps succeeding while its sources sit
frozen at the moment of the rename. Nothing fails, so nothing prompts a fix —
which is exactly what happened to 2024.1, and only surfaced when a fix for
CVE-2026-55707 failed to appear in a freshly rebuilt image.

That instance is fixed (#2640, osism/container-images-kolla#762).
This adds the check that would have caught it on day one, and will catch 2025.1
when it transitions.

The three commits

Please rebase-merge rather than squash: the first commit is an independent
fix to pre-existing code and the third a general driver capability, and both
read better in main's history on their own.

  1. http: scope rate-limit hints to the host — a pre-existing bug, fixable
    and reviewable on its own. _rate_limit_hint() describes GitHub's rate
    limits but never checked that the response came from GitHub, so any
    markerless 429 was reported as raw.githubusercontent.com CDN throttling
    with advice built around that host. Reachable today through a non-GitHub
    github_raw/github_api override — the same case _auth_headers() already
    guards a token against. Behaviour for GitHub hosts is unchanged.
  2. Add drift check for stale OpenStack source refs — the plugin, plus the
    transport and parsing it needs.
  3. Refuse a local-only run that needs the network--base-dir without
    --remote-fallback is a local-only run, and source._resolve() enforces it
    by erroring rather than fetching. This plugin asks what upstream publishes
    right now, which no checkout records, so it would reach the network
    regardless of the mode requested. A plugin may now declare EXTERNAL_HOSTS
    and the driver refuses such a run up front, listing every offending plugin at
    once. Optional and read with getattr, so no other plugin changes.

Design notes worth a look

Artifact-driven, not branch-driven. The question is which tarball the build
consumes, so probing the published set answers it directly. It also avoids a
blocker: release_to_ref would need an upstream owner configured per project,
and sources maps only kolla and kolla_ansible (default owner osism), so
a branch probe would 404 on all 21 projects.

Only two phases are probed, because only two are published. Upstream marks
both transitions with a tag — <release>-eom when the stable branch becomes
unmaintained, <release>-eol when the unmaintained branch is deleted — and
those tags invite the assumption that they are further phases to move to. They
are not: opendev builds an artifact per branch, so no tarball exists for
either tag. Verified 404 for every <release>-eol and <release>-eom URL
checked across nova, aodh, keystone, cinder, glance and requirements, over
2023.1, 2023.2, 2024.1, 2024.2, victoria, wallaby and zed — while a release-tag
tarball such as nova-30.0.0.tar.gz does resolve. The tags also run the
opposite way round from what the names suggest: nova carries 2024.1-eom and
2023.1-eom while unmaintained/2024.1 and unmaintained/2023.1 are still
live branches, and 2023.2-eol/2024.2-eol sit on series whose branch is
already deleted. A full run is 68 probes, and a release already moved to
unmaintained costs none.

One HEAD per candidate, not the index listing. My first version read the
per-project index — fewer requests, one per project covers all releases — but
those listings are large (nova's is ~1.4 MB with 10k entries) and timed out
intermittently in live runs, first on keystone at 30s then nova at 90s. A HEAD
is a few hundred bytes.

An unanswered probe is unknown, not clean. Reading an outage or a throttled
response as "no later phase published" would report no drift and silently
reinstate the blind spot this check exists to close, so anything but a 200 or a
404 raises ProbeInconclusive. That guard sits on the finding, not on the
run: the driver wraps its whole plugin loop in one handler, so raising would
discard every other plugin's findings over one upstream hiccup. scan() catches
it per project and reports the ref as unverified — phase unknown rather than
confirmed current, in its own report block, actionable so the run still exits
non-zero. head() additionally retries a transport failure or a transient
status once with a backoff; measured first, 110 sequential HEADs against opendev
took 47s with no 429 and no 5xx and no rate-limit headers in the responses
(plain Apache, no CDN), so a blip is the failure mode there rather than
throttling.

Silence is not always a clean bill of health. For a release with no later
phase published — 2024.2, which is EOL — the frozen stable tarball is the only
option and nothing is flagged. That is correct behaviour, but it means silence
there says "no better ref exists", not "sources are current".

Verification

Replayed against the pre-fix manifest with live probes: 21 of 22 refs flagged
in 9s
, gnocchi correctly skipped because stable/4.6 does not follow the
release lifecycle. Against the current tree the plugin reports nothing, and
2024.2 and 2025.x stay silent. A local-only run exits 2 before any probe; a full
live run takes 30s.

Each commit was checked out into a clean worktree and verified there — 341,
392 and 398 tests passing
respectively, flake8 and black clean at every
step.

Not covered

The requirements ref in container-images-kolla scripts/002-generate.sh is
hardcoded rather than read from the manifest, so this plugin cannot see it.
Detecting that would mean parsing shell in another repo. The retirement guide's
checklist is the control there:

Two related refs are also outside what this checks, both currently still on
stable-: openstackclient/Containerfile in container-images (a no-op today —
that matrix builds only 2025.1/2025.2/2026.1) and the IPA artifacts in
container-image-kolla-ansible, where ipa-centos9-unmaintained-2024.1.initramfs
exists and is fresh while the stable-2024.1 one froze on 2025-10-09.

@ideaship
ideaship force-pushed the feature/drift-source-ref-phase branch from 70a0639 to d7397c3 Compare July 31, 2026 20:43
@berendt berendt moved this from New to In progress in Human Board Aug 3, 2026
_rate_limit_hint() explains GitHub's rate limits, but nothing checked that
the response came from GitHub. Its markerless-429 branch reads any 429
without X-RateLimit-* headers as raw.githubusercontent.com CDN throttling
and answers with advice built around that host: "Its rate limit is per-IP
[...] Prefer local checkouts (--base-dir) to avoid remote fetches."

That was sound while every caller read a GitHub host, and it is wrong the
moment one does not. A github_raw or github_api override pointing at a
mirror makes it reachable today -- the same case _auth_headers() already
guards a token against -- and the reader is then told to chase a service
that was never involved, with advice that need not apply to the host that
actually refused.

Hints are now host-scoped the way _auth_headers() is: a non-GitHub host
gets at most an echo of its own Retry-After, which is the one piece of
advice here that is not GitHub-specific. Behaviour for GitHub hosts is
unchanged, and the existing tests covering the API-quota message, the
secondary limit, the raw-CDN hint and the plain 403 still pass untouched.

_rate_limit_hint() takes the url to make that decision. It is module
internal, and its only caller is _http_error(), which already has one.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
Detects the silent failure that left OSISM's 2024.1 images on October 2025
sources for nine months.

Upstream renames stable/<release> to unmaintained/<release> when a release
enters the unmaintained phase, and deletes the branch at EOL.
tarballs.opendev.org keeps serving the artifact for a retired branch name
but stops regenerating it, so a build that still asks for
<project>-stable-<release>.tar.gz keeps succeeding while its sources sit
frozen at the moment of the rename. Nothing fails, so nothing prompts a
fix -- which is what happened to 2024.1, and only surfaced when a fix for
CVE-2026-55707 failed to appear in a freshly rebuilt image. That instance
is fixed (#2640, osism/container-images-kolla#762); this adds
the check that would have caught it on day one, and will catch 2025.1 when
it transitions.

kolla_source_ref_phase compares each openstack_projects ref in
latest/openstack-<release>.yml against the latest lifecycle phase opendev
actually publishes a tarball for, and reports drift when a later phase
exists.

Artifact-driven, not branch-driven. The question is which tarball the
build consumes, so probing the published set answers it directly. This
also avoids a blocker: release_to_ref would need an upstream owner
configured per project, and sources maps only kolla and kolla_ansible
(default owner osism), so a branch probe would 404 on all 21 projects.

Only two phases are probed, because only two are published. Upstream marks
both transitions with a tag -- <release>-eom when the stable branch becomes
unmaintained, <release>-eol when the unmaintained branch is deleted -- and
the tags are easy to mistake for further phases to move to. They are not:
opendev builds an artifact per branch, so no tarball exists for either
tag. Verified 404 for every <release>-eol and <release>-eom URL checked
across nova, aodh, keystone, cinder, glance and requirements, over 2023.1,
2023.2, 2024.1, 2024.2, victoria, wallaby and zed, while a release-tag
tarball such as nova-30.0.0.tar.gz does resolve. The tag order also runs
the other way round from what the phase names suggest: nova carries
2024.1-eom and 2023.1-eom while unmaintained/2024.1 and unmaintained/2023.1
are still live branches, and 2023.2-eol and 2024.2-eol sit on series whose
branch is already deleted. Probing them would spend a request per project
on a candidate that can never match and would offer the reader a target
they cannot download.

Only phases after the configured one are probed: an earlier phase's
tarball keeps being served after a rename, so its presence says nothing
about freshness. A full run over four releases is 68 probes, and a release
already moved to unmaintained costs none, because nothing later exists to
check.

The check self-limits at EOL. 2024.2 has no unmaintained tarball, so the
frozen stable one is the only option and nothing is flagged. Worth stating
plainly: for such a release, silence means "no better ref exists", not
"sources are current".

Existence is probed with one HEAD per candidate rather than by reading the
per-project index listing. Those listings are large -- nova's is ~1.4 MB
with 10k entries -- and timed out intermittently in live runs, first on
keystone at 30s then nova at 90s, which for a nightly check is noise. A
HEAD is a few hundred bytes.

A probe has three outcomes, not two: published, absent, and inconclusive.
Reading an outage or a throttled response as "no later phase published"
would report no drift and silently reinstate the very blind spot this
check exists to close, so anything but a 200 or a 404 raises
ProbeInconclusive. That guard sits on the finding rather than on the run:
the driver wraps its whole plugin loop in one handler, so raising would
discard every other plugin's findings over one upstream hiccup. scan()
catches it per project and reports the ref as unverified -- phase unknown
rather than confirmed current, in its own report block, actionable so the
run still exits non-zero. A project left unresolved never also yields a
stale finding: with one candidate's answer unknown, naming a target would
be a guess. ProbeInconclusive derives from SourceError so that one
escaping uncaught degrades to the ordinary abort instead of a traceback.

Supporting changes:

- enablement.parse_source_refs returns openstack_projects as
  {project: ref} with names verbatim, since they index tarball URLs where
  the real name (e.g. neutron-dynamic-routing) resolves. parse_build_set
  canonicalises and so cannot be reused.
- http.head probes existence under the same ok= whitelist contract as
  _get, retrying a transport failure or a transient status once with a
  backoff. A sweep makes dozens of small requests and one blip should not
  decide the outcome; a host that keeps refusing still raises. Measured
  before choosing this: 110 sequential HEADs against tarballs.opendev.org
  in 47s, no 429 and no 5xx, and no rate-limit headers in the responses
  (plain Apache, no CDN), so a blip is the failure mode there rather than
  throttling.
- SourceError carries the HTTP status, so ProbeInconclusive can classify a
  failure without parsing the message.
- The plugin declares EXTERNAL_HOSTS, naming the host no --base-dir can
  stand in for. The driver change that acts on it follows.
- The plugin sorts first in KOLLA_PLUGINS: source selection is the
  upstream-most input, so a finding here means every downstream result is
  about stale sources.

Verified by replaying the pre-fix manifest with live probes: 21 of 22 refs
flagged in 9s, gnocchi correctly skipped because stable/4.6 does not
follow the release lifecycle. Against the current tree the plugin reports
nothing, and 2024.2 and 2025.x stay silent. 392 tests pass.

Not covered: the requirements ref in container-images-kolla
scripts/002-generate.sh is hardcoded rather than read from the manifest,
so this plugin cannot see it. Detecting that would mean parsing shell in
another repo; the retirement guide's checklist is the control there
(osism/osism.github.io#1048).

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
--base-dir without --remote-fallback is a local-only run, and the promise
is enforced: source._resolve() errors on a repo it cannot find under a
base dir rather than fetching it. Every plugin honoured that, because
every plugin read only OSISM repos.

kolla_source_ref_phase does not. It asks what upstream publishes right
now, which no checkout records, so it reached tarballs.opendev.org
regardless of the mode requested -- and did so partway into the run, after
the resolution summary had already told the operator the sources were
local.

There is no local answer to substitute, so the honest behaviour is to
refuse rather than to degrade. A plugin may now declare EXTERNAL_HOSTS,
the hosts it reads that no --base-dir can serve. The driver checks it
before any comparison runs and, on a local-only run, exits 2 naming every
offending plugin and its hosts at once, the way describe_resolution()
reports every unresolvable repo in one pass. --remote-fallback allows the
read; --plugin deselects the plugin.

The attribute is optional and read with getattr, so plugins that only read
repos are unaffected and need no change. It is documented on the Plugin
protocol rather than declared there, since declaring it would imply every
plugin must define it.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

3 participants