Add drift check for stale OpenStack source refs - #2642
Draft
ideaship wants to merge 3 commits into
Draft
Conversation
ideaship
force-pushed
the
feature/drift-source-ref-phase
branch
from
July 31, 2026 20:43
70a0639 to
d7397c3
Compare
_rate_limit_hint() explains GitHub's rate limits, but nothing checked that the response came from GitHub. Its markerless-429 branch reads any 429 without X-RateLimit-* headers as raw.githubusercontent.com CDN throttling and answers with advice built around that host: "Its rate limit is per-IP [...] Prefer local checkouts (--base-dir) to avoid remote fetches." That was sound while every caller read a GitHub host, and it is wrong the moment one does not. A github_raw or github_api override pointing at a mirror makes it reachable today -- the same case _auth_headers() already guards a token against -- and the reader is then told to chase a service that was never involved, with advice that need not apply to the host that actually refused. Hints are now host-scoped the way _auth_headers() is: a non-GitHub host gets at most an echo of its own Retry-After, which is the one piece of advice here that is not GitHub-specific. Behaviour for GitHub hosts is unchanged, and the existing tests covering the API-quota message, the secondary limit, the raw-CDN hint and the plain 403 still pass untouched. _rate_limit_hint() takes the url to make that decision. It is module internal, and its only caller is _http_error(), which already has one. Assisted-by: Claude:claude-opus-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
Detects the silent failure that left OSISM's 2024.1 images on October 2025 sources for nine months. Upstream renames stable/<release> to unmaintained/<release> when a release enters the unmaintained phase, and deletes the branch at EOL. tarballs.opendev.org keeps serving the artifact for a retired branch name but stops regenerating it, so a build that still asks for <project>-stable-<release>.tar.gz keeps succeeding while its sources sit frozen at the moment of the rename. Nothing fails, so nothing prompts a fix -- which is what happened to 2024.1, and only surfaced when a fix for CVE-2026-55707 failed to appear in a freshly rebuilt image. That instance is fixed (#2640, osism/container-images-kolla#762); this adds the check that would have caught it on day one, and will catch 2025.1 when it transitions. kolla_source_ref_phase compares each openstack_projects ref in latest/openstack-<release>.yml against the latest lifecycle phase opendev actually publishes a tarball for, and reports drift when a later phase exists. Artifact-driven, not branch-driven. The question is which tarball the build consumes, so probing the published set answers it directly. This also avoids a blocker: release_to_ref would need an upstream owner configured per project, and sources maps only kolla and kolla_ansible (default owner osism), so a branch probe would 404 on all 21 projects. Only two phases are probed, because only two are published. Upstream marks both transitions with a tag -- <release>-eom when the stable branch becomes unmaintained, <release>-eol when the unmaintained branch is deleted -- and the tags are easy to mistake for further phases to move to. They are not: opendev builds an artifact per branch, so no tarball exists for either tag. Verified 404 for every <release>-eol and <release>-eom URL checked across nova, aodh, keystone, cinder, glance and requirements, over 2023.1, 2023.2, 2024.1, 2024.2, victoria, wallaby and zed, while a release-tag tarball such as nova-30.0.0.tar.gz does resolve. The tag order also runs the other way round from what the phase names suggest: nova carries 2024.1-eom and 2023.1-eom while unmaintained/2024.1 and unmaintained/2023.1 are still live branches, and 2023.2-eol and 2024.2-eol sit on series whose branch is already deleted. Probing them would spend a request per project on a candidate that can never match and would offer the reader a target they cannot download. Only phases after the configured one are probed: an earlier phase's tarball keeps being served after a rename, so its presence says nothing about freshness. A full run over four releases is 68 probes, and a release already moved to unmaintained costs none, because nothing later exists to check. The check self-limits at EOL. 2024.2 has no unmaintained tarball, so the frozen stable one is the only option and nothing is flagged. Worth stating plainly: for such a release, silence means "no better ref exists", not "sources are current". Existence is probed with one HEAD per candidate rather than by reading the per-project index listing. Those listings are large -- nova's is ~1.4 MB with 10k entries -- and timed out intermittently in live runs, first on keystone at 30s then nova at 90s, which for a nightly check is noise. A HEAD is a few hundred bytes. A probe has three outcomes, not two: published, absent, and inconclusive. Reading an outage or a throttled response as "no later phase published" would report no drift and silently reinstate the very blind spot this check exists to close, so anything but a 200 or a 404 raises ProbeInconclusive. That guard sits on the finding rather than on the run: the driver wraps its whole plugin loop in one handler, so raising would discard every other plugin's findings over one upstream hiccup. scan() catches it per project and reports the ref as unverified -- phase unknown rather than confirmed current, in its own report block, actionable so the run still exits non-zero. A project left unresolved never also yields a stale finding: with one candidate's answer unknown, naming a target would be a guess. ProbeInconclusive derives from SourceError so that one escaping uncaught degrades to the ordinary abort instead of a traceback. Supporting changes: - enablement.parse_source_refs returns openstack_projects as {project: ref} with names verbatim, since they index tarball URLs where the real name (e.g. neutron-dynamic-routing) resolves. parse_build_set canonicalises and so cannot be reused. - http.head probes existence under the same ok= whitelist contract as _get, retrying a transport failure or a transient status once with a backoff. A sweep makes dozens of small requests and one blip should not decide the outcome; a host that keeps refusing still raises. Measured before choosing this: 110 sequential HEADs against tarballs.opendev.org in 47s, no 429 and no 5xx, and no rate-limit headers in the responses (plain Apache, no CDN), so a blip is the failure mode there rather than throttling. - SourceError carries the HTTP status, so ProbeInconclusive can classify a failure without parsing the message. - The plugin declares EXTERNAL_HOSTS, naming the host no --base-dir can stand in for. The driver change that acts on it follows. - The plugin sorts first in KOLLA_PLUGINS: source selection is the upstream-most input, so a finding here means every downstream result is about stale sources. Verified by replaying the pre-fix manifest with live probes: 21 of 22 refs flagged in 9s, gnocchi correctly skipped because stable/4.6 does not follow the release lifecycle. Against the current tree the plugin reports nothing, and 2024.2 and 2025.x stay silent. 392 tests pass. Not covered: the requirements ref in container-images-kolla scripts/002-generate.sh is hardcoded rather than read from the manifest, so this plugin cannot see it. Detecting that would mean parsing shell in another repo; the retirement guide's checklist is the control there (osism/osism.github.io#1048). Assisted-by: Claude:claude-opus-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
--base-dir without --remote-fallback is a local-only run, and the promise is enforced: source._resolve() errors on a repo it cannot find under a base dir rather than fetching it. Every plugin honoured that, because every plugin read only OSISM repos. kolla_source_ref_phase does not. It asks what upstream publishes right now, which no checkout records, so it reached tarballs.opendev.org regardless of the mode requested -- and did so partway into the run, after the resolution summary had already told the operator the sources were local. There is no local answer to substitute, so the honest behaviour is to refuse rather than to degrade. A plugin may now declare EXTERNAL_HOSTS, the hosts it reads that no --base-dir can serve. The driver checks it before any comparison runs and, on a local-only run, exits 2 naming every offending plugin and its hosts at once, the way describe_resolution() reports every unresolvable repo in one pass. --remote-fallback allows the read; --plugin deselects the plugin. The attribute is optional and read with getattr, so plugins that only read repos are unaffected and need no change. It is documented on the Plugin protocol rather than declared there, since declaring it would imply every plugin must define it. Assisted-by: Claude:claude-opus-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
force-pushed
the
feature/drift-source-ref-phase
branch
from
August 4, 2026 11:21
d7397c3 to
f7e3bbe
Compare
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detects the silent failure that left OSISM's 2024.1 images on October 2025
sources for nine months.
Upstream renames
stable/<release>tounmaintained/<release>when a releaseenters the unmaintained phase, and deletes the branch at EOL.
tarballs.opendev.orgkeeps serving the artifact for a retired branch name butstops regenerating it, so a build that still asks for
<project>-stable-<release>.tar.gzkeeps succeeding while its sources sitfrozen at the moment of the rename. Nothing fails, so nothing prompts a fix —
which is exactly what happened to 2024.1, and only surfaced when a fix for
CVE-2026-55707 failed to appear in a freshly rebuilt image.
That instance is fixed (#2640, osism/container-images-kolla#762).
This adds the check that would have caught it on day one, and will catch 2025.1
when it transitions.
The three commits
Please rebase-merge rather than squash: the first commit is an independent
fix to pre-existing code and the third a general driver capability, and both
read better in
main's history on their own.http: scope rate-limit hints to the host— a pre-existing bug, fixableand reviewable on its own.
_rate_limit_hint()describes GitHub's ratelimits but never checked that the response came from GitHub, so any
markerless 429 was reported as
raw.githubusercontent.comCDN throttlingwith advice built around that host. Reachable today through a non-GitHub
github_raw/github_apioverride — the same case_auth_headers()alreadyguards a token against. Behaviour for GitHub hosts is unchanged.
Add drift check for stale OpenStack source refs— the plugin, plus thetransport and parsing it needs.
Refuse a local-only run that needs the network—--base-dirwithout--remote-fallbackis a local-only run, andsource._resolve()enforces itby erroring rather than fetching. This plugin asks what upstream publishes
right now, which no checkout records, so it would reach the network
regardless of the mode requested. A plugin may now declare
EXTERNAL_HOSTSand the driver refuses such a run up front, listing every offending plugin at
once. Optional and read with
getattr, so no other plugin changes.Design notes worth a look
Artifact-driven, not branch-driven. The question is which tarball the build
consumes, so probing the published set answers it directly. It also avoids a
blocker:
release_to_refwould need an upstream owner configured per project,and
sourcesmaps onlykollaandkolla_ansible(default ownerosism), soa branch probe would 404 on all 21 projects.
Only two phases are probed, because only two are published. Upstream marks
both transitions with a tag —
<release>-eomwhen the stable branch becomesunmaintained,
<release>-eolwhen the unmaintained branch is deleted — andthose tags invite the assumption that they are further phases to move to. They
are not: opendev builds an artifact per branch, so no tarball exists for
either tag. Verified 404 for every
<release>-eoland<release>-eomURLchecked across nova, aodh, keystone, cinder, glance and requirements, over
2023.1, 2023.2, 2024.1, 2024.2, victoria, wallaby and zed — while a release-tag
tarball such as
nova-30.0.0.tar.gzdoes resolve. The tags also run theopposite way round from what the names suggest: nova carries
2024.1-eomand2023.1-eomwhileunmaintained/2024.1andunmaintained/2023.1are stilllive branches, and
2023.2-eol/2024.2-eolsit on series whose branch isalready deleted. A full run is 68 probes, and a release already moved to
unmaintainedcosts none.One HEAD per candidate, not the index listing. My first version read the
per-project index — fewer requests, one per project covers all releases — but
those listings are large (nova's is ~1.4 MB with 10k entries) and timed out
intermittently in live runs, first on keystone at 30s then nova at 90s. A HEAD
is a few hundred bytes.
An unanswered probe is unknown, not clean. Reading an outage or a throttled
response as "no later phase published" would report no drift and silently
reinstate the blind spot this check exists to close, so anything but a 200 or a
404 raises
ProbeInconclusive. That guard sits on the finding, not on therun: the driver wraps its whole plugin loop in one handler, so raising would
discard every other plugin's findings over one upstream hiccup.
scan()catchesit per project and reports the ref as unverified — phase unknown rather than
confirmed current, in its own report block, actionable so the run still exits
non-zero.
head()additionally retries a transport failure or a transientstatus once with a backoff; measured first, 110 sequential HEADs against opendev
took 47s with no 429 and no 5xx and no rate-limit headers in the responses
(plain Apache, no CDN), so a blip is the failure mode there rather than
throttling.
Silence is not always a clean bill of health. For a release with no later
phase published — 2024.2, which is EOL — the frozen
stabletarball is the onlyoption and nothing is flagged. That is correct behaviour, but it means silence
there says "no better ref exists", not "sources are current".
Verification
Replayed against the pre-fix manifest with live probes: 21 of 22 refs flagged
in 9s,
gnocchicorrectly skipped becausestable/4.6does not follow therelease lifecycle. Against the current tree the plugin reports nothing, and
2024.2 and 2025.x stay silent. A local-only run exits 2 before any probe; a full
live run takes 30s.
Each commit was checked out into a clean worktree and verified there — 341,
392 and 398 tests passing respectively,
flake8andblackclean at everystep.
Not covered
The requirements ref in
container-images-kollascripts/002-generate.shishardcoded rather than read from the manifest, so this plugin cannot see it.
Detecting that would mean parsing shell in another repo. The retirement guide's
checklist is the control there:
Two related refs are also outside what this checks, both currently still on
stable-:openstackclient/Containerfileincontainer-images(a no-op today —that matrix builds only 2025.1/2025.2/2026.1) and the IPA artifacts in
container-image-kolla-ansible, whereipa-centos9-unmaintained-2024.1.initramfsexists and is fresh while the
stable-2024.1one froze on 2025-10-09.