Harden PyPI install verification against stale cache

[lelia]

Summary

The release workflow's "Verify package is installable" step retries pip install socketsecurity==$VERSION for 10 minutes (30 × 20s). That budget assumed PyPI's simple index reflects a successful upload within seconds, which held until 2026-08-05, when two releases in one day exceeded it:

• socketsecurity 2.5.9 — uploaded 16:40:20 UTC; all 30 verify attempts (16:40:35–16:50:46) got a version list ending at 2.5.8. The release run failed and the Docker publish was skipped, so socketdev/cli:2.5.9 was never pushed.
• socketdev 3.4.2 — uploaded 23:29:39 UTC; same failure 23:30–23:39 in that repo's identical verify step. Meanwhile other CI runners installed 3.4.2 from PyPI successfully during the same minutes, so the staleness was specific to individual CDN cache nodes.

In both cases the JSON API (/pypi/<pkg>/<version>/json) showed the release immediately; only the CDN-cached simple index (/simple/<pkg>/, which pip actually uses) was stale. PyPI's status page reported no incident. This failure class has recurred on PyPI's side for years (see references below).

Changes

• Extend the retry window from 10 to 30 minutes (60 × 30s), with no sleep after the final attempt.
• Add --no-cache-dir so each attempt fully refetches the index. The simple index is served with Cache-Control: max-age=600, so without this, pip revalidates its locally cached copy by ETag and a stale CDN node can keep answering 304 for the whole loop.
• Install via python -m pip with an explicit --index-url https://pypi.org/simple/, and quote workflow output redirects.
• On failed attempts, check the JSON API and log when the version exists there — making "index propagation delay" distinguishable from "publish actually failed" in the logs.

The verify step is now line-for-line consistent with the same hardening in SocketDev/socket-sdk-python#102.

Notes

• Rerunning the failed v2.5.9 release run after this propagation delay cleared will publish the missing Docker images; the workflow's existing pypi_exists check makes the rerun idempotent.

References

Prior reports of the same PyPI simple-index staleness class:

• PyPI caching packages too long (updated release still not showing up after 12 hours after upload) pypi/warehouse#17179 — PyPI caching packages too long (release not in index hours after upload)
• New packages not appearing in pypi.org/simple index pypi/warehouse#8416 — New packages not appearing in pypi.org/simple index
• New packages not appearing in pypi.org/simple index pypi/warehouse#5017 — New packages not appearing in pypi.org/simple index

[David Larsen (dc-larsen)]

Approving — reproduced the root cause locally and confirmed the fix.

Ran pip 26.2.1 against a local PEP 503 index serving Cache-Control: max-age=600 (what pypi.org/simple actually returns). With a stale index cached, pip served the stale version list from its own HTTP cache and never re-requested the index on retry; adding --no-cache-dir picked up the new version immediately. So the old 30 × 20s budget (attempts at t=0..580s) sat entirely inside the 600s cache window — once attempt 1 missed, no amount of waiting could recover. --no-cache-dir is the change that matters here; the longer window covers genuine CDN-side staleness on top of it.

Also verified both loop paths (success writes success=true/exit 0; failure writes success=false/exit 1 with no trailing sleep) and the verify step is shellcheck- and actionlint-clean.