Release validation

[cmaloney]

What sort of testing / what things should be verified before each binary release?

[brettcannon]

If we end up doing this and it's all doable from the CPython repo itself it would probably be part of general CI.

[cmaloney]

I worry that current CI doesn't validate well "here's a fresh install of <platform> does downloading and doing the steps on the webpage result in a working python".

Two of the issues I've run into around that on similar projects:

• Build succeeded but the artifact got corrupted on upload (or one of N variants was missing).
• Build picked up a new dependency from host libraries and "isolated" build now doesn't run on a newly installed system.

👍 that these should be automated / in CI / a tool to run which does it for the most part.

[indygreg]

PBS has a small pile of Rust code to validate built distributions. Lots of spot tests for loading extensions and verifying certain functionality works. (Like optional SQLite features.) It also enforces an allow list of library dependencies, glibc symbol versions, and spot checks other symbol properties. Various checks were added after regressions were accidentally shipped. It's important to do this at CI time at the latest.

[emmatyping]

Yeah I think checks similar to what PBS does would be good. I also think we can have install buildbots like https://github.com/python/buildmaster-config/blob/a3bd9128393e49f48c9cf97dc18966adf12a9051/master/custom/factories.py#L191

[freakboy3742]

FWIW, I've got a similar set of "smoke tests" for Apple Standalone in BeeWare. I've had occasions where CPython compiles fine, but the specific mechanics of the redistributable redistributable artefact then fail - sometimes because optional features in the build weren't autodetected, and sometimes because of formatting/configuration issue.