Migrate scripts/ from bash to Python with pytest coverage

[fnando]

What

Migrates all 15 shell scripts under scripts/ to Python (snake_case .py entry-points) and introduces a pytest suite with 189 tests. While here, also lifts the inline jq/bash/awk blocks embedded in publish.yml and release.yml into their own Python scripts so workflows remain thin orchestrators — and drops the project's runtime dependency on jq entirely.

Why

Two patterns in the bash scripts were correctness-critical and review-hostile: dynamic jq expression composition (refresh-*-digests.sh, release-prepare.sh) and complex jq pipelines with inline functions and named-group regex (resolve-matrix.sh, release-body.sh, validate-json.sh). They had zero test coverage — every bug shipped to CI first. Python gives us plain control flow, importable modules, and a real test harness.

How

• One script per commit. Each commit writes the .py, adds tests, updates every caller (workflows, calling scripts, README/RELEASE examples), and deletes the old .sh. CI stays green at every step; git bisect works cleanly.
• Shared library under scripts/lib/: semver, rust_keys, builds, common, runner, plus three adapters (docker_inspect, git_remote, gh_cli) and a tag_names helper. Adapters wrap the non-deterministic surfaces (docker, git, gh, HTTP) so tests patch one symbol per script.
• Four new entry-points lifted from inline workflow logic: publish_manifests.py, publish_aliases.py, release_pr_body.py, and write_metadata.py. Each has a --dry-run mode where applicable.
• No jq, no awk anywhere in the repo. grep -rn '\bjq\b' returns zero matches across scripts, workflows, and docs. The write_metadata.py step replaces the last two jq -n blocks in publish.yml and folds the awk '/^Digest:/' parsing of skipped-pair digests into lib/docker_inspect.index_digest.
• Invocation: every script opens with #!/usr/bin/env -S uv run python, so call sites stay ./scripts/foo.py — no uv run prefix in workflows, docs, or developer shells.
• Deps: uv + pyproject.toml + committed uv.lock. Runtime: jsonschema, semver. Dev: pytest, ruff. Python 3.14.5 pinned via .python-version.
• CI: new tests job (uv + pytest) and python job (ruff check + format). Dropped shellcheck and validate-shell jobs. The complete gate needs: list updated. Every job that runs a Python script gets astral-sh/setup-uv@08807647 (SHA-pinned per repo policy).

Contracts preserved

• resolve_matrix.py emits compact single-line JSON with sorted keys for fromJson() matrix consumption in publish.yml.
• newest_pair.py, tag_names.py, release_prepare.py print exactly one stdout line; all logs go to stderr.
• builds.py.dump() writes byte-for-byte identical output to jq --sort-keys . builds.json (verified by test_dump_matches_existing_builds_json_byte_for_byte).
• CLI flag vocabulary unchanged (--stellar-cli-version, --rust-version, --dry-run, --all/--force).
• Refresh-fills-blanks semantics retained.

Fork-test

End-to-end publish.yml runs against a personal fork (registry pointed at a personal repo): https://github.com/fnando/stellar-cli-docker/actions/runs/26620171750 — fully green across all 9 jobs (resolve matrix → 4 per-arch builds → manifest list assembly → alias publishing → release body composition → SBOM + provenance attestations). Validated locally against the published images:

• :latest and :<cli> both resolve to the multi-arch index of the default rust pair (1.96-slim-trixie).
• Per-rust list and per-arch tags all exist with the expected digests.
• smoke_test_image.py against a real published per-arch image: version, offline contract build --help, all four OCI labels (including base.digest cross-checked against builds.json) — all pass.
• verify_image.py against the per-arch digest: SLSA provenance + SPDX SBOM both verify.
• GitHub Release page has all 8 expected assets (2 rust × 2 arch × (sbom + prov)) and the rendered body matches the spec (sections, sorted rust newest-first, per-arch digests, verify-command blocks).
• Confirmed the new write_metadata.py produces correctly-formed meta-*.json files (sorted keys, trailing newline) across both the fresh-build path (--digest passed in) and the skipped-pair lookup path (--digest resolved from --image via lib/docker_inspect).
• Confirmed the cosign block in the rendered release body now uses single-backslash shell line continuation, copy-paste-runnable into a terminal.

Notes for review

• builds.json is untouched; this PR is pure tooling.
• Workflow logic that previously read derive_default_rust_for_cli from common.sh now goes through lib/builds.derive_default_rust() via the lifted scripts — same behavior, no shell sourcing left.
• 189 unit + integration tests; integration tests mock at the adapter boundary so no real docker/git/gh calls.
• Includes a small Dockerfile fix — the in-image version-assertion's stellar version | head -n1 pipeline panicked on Rust 1.96+ due to EPIPE-on-stdio. Surfaced by an early fork-test run when 1.96.0-slim-trixie was added to the fork's builds.json. Production builds.json stops at 1.95 so this never bit on main, but the fix lands now so 1.96 is safe whenever it's pulled in. The fix captures stellar version output once into a variable and parses in memory.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pytest@​8.3.5
	jsonschema@​4.26.0
	ruff@​0.15.15
	semver@​3.0.4

View full report

[socket-security]

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report

[Copilot]

Pull request overview

This PR migrates the repo’s operational tooling from bash to Python, adds a shared scripts/lib/ Python library, and introduces a substantial pytest suite to make the release/publish pipeline testable and easier to review/maintain.

Changes:

• Replaces scripts/*.sh entrypoints with scripts/*.py equivalents and adds reusable helpers under scripts/lib/.
• Adds pytest + ruff tooling (pyproject.toml, uv.lock) and a comprehensive unit/integration test suite with fixtures.
• Updates GitHub workflows to run the new Python scripts via uv, and includes a Dockerfile tweak to avoid EPIPE failures during version parsing.

Reviewed changes

Copilot reviewed 85 out of 91 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/test_write_metadata.py	Unit coverage for write_metadata.py output/digest resolution and formatting.
tests/unit/test_verify_image.py	Unit coverage for verify_image.py success/failure paths and digest-only enforcement.
tests/unit/test_validate_json.py	Unit coverage for JSON key ordering, schema, and cross-field validation helpers.
tests/unit/test_tag_names.py	Unit coverage for tag composition and CLI argument requirements.
tests/unit/test_smoke_test_image.py	Unit coverage for smoke-test checks (version/help/labels).
tests/unit/test_semver.py	Unit coverage for numeric semver parsing/sorting behavior.
tests/unit/test_rust_keys.py	Unit coverage for parsing composite rust base keys.
tests/unit/test_runner.py	Unit coverage for subprocess wrapper and HTTP JSON fetch helper.
tests/unit/test_resolve_matrix.py	Unit coverage for matrix construction and JSON output modes.
tests/unit/test_repro_test.py	Unit coverage for reproducibility test helpers and error messaging.
tests/unit/test_release_push_branch.py	Unit coverage for release branch push logic (force/orphan/PR-exists).
tests/unit/test_release_pr_body.py	Unit coverage for release PR title/body composition.
tests/unit/test_release_body.py	Unit coverage for release body composition and ordering/formatting.
tests/unit/test_publish_manifests.py	Unit coverage for manifest list creation/skip/dry-run behavior.
tests/unit/test_publish_aliases.py	Unit coverage for alias publishing and latest/newest selection.
tests/unit/test_newest_pair.py	Unit coverage for newest-cli selection and CLI output modes.
tests/unit/test_meta.py	Ensures test runtime meets minimum Python version.
tests/unit/test_git_remote.py	Unit coverage for git ls-remote parsing/peel preference logic.
tests/unit/test_gh_cli.py	Unit coverage for gh adapter functions and argument composition.
tests/unit/test_docker_inspect.py	Unit coverage for digest extraction + manifest creation wrapper.
tests/unit/test_common.py	Unit coverage for stderr logging/die behavior and preflight checks.
tests/unit/test_builds.py	Unit coverage for builds.json IO/query helpers and atomic dump semantics.
tests/unit/test_build_image.py	Unit coverage for docker build command composition and validation.
tests/unit/init.py	Declares unit test package.
tests/integration/test_release_prepare.py	Integration coverage for release-prepare flow (staged builds.json, stubbing adapters).
tests/integration/test_refresh_stellar_cli_digests.py	Integration coverage for stellar-cli ref refresh behavior.
tests/integration/test_refresh_rust_digests.py	Integration coverage for rust digest refresh behavior.
tests/integration/init.py	Declares integration test package.
tests/fixtures/rust_hub_tags.json	Fixture payload for Docker Hub tag picker tests.
tests/fixtures/meta_arm64.json	Fixture metadata row for release body tests.
tests/fixtures/meta_amd64.json	Fixture metadata row for release body tests.
tests/fixtures/builds_unsorted_keys.json	Fixture for unsorted-key validation behavior.
tests/fixtures/builds_unpinned.json	Fixture for rust digest refresh behavior.
tests/fixtures/builds_unpinned_refs.json	Fixture for stellar-cli ref refresh behavior.
tests/fixtures/builds_orphan_rust.json	Fixture for cross-field constraint violation tests.
tests/fixtures/builds_multi_cli.json	Fixture for multi-cli/newest selection tests.
tests/fixtures/builds_minimal.json	Fixture for minimal builds.json behaviors.
tests/conftest.py	Shared fixtures for builds.json payloads and fixture directory paths.
tests/init.py	Declares tests package.
scripts/write_metadata.py	New Python entrypoint to write meta-*.json files (with optional digest lookup).
scripts/verify-image.sh	Deleted; replaced by scripts/verify_image.py.
scripts/verify_image.py	New Python verifier for provenance+SBOM attestation chains via gh.
scripts/validate-shell.sh	Deleted; shell validation job removed in favor of Python tooling checks.
scripts/validate-json.sh	Deleted; replaced by scripts/validate_json.py.
scripts/validate_json.py	New Python JSON validator (sorted keys, schema, cross-field constraints).
scripts/tag-names.sh	Deleted; replaced by scripts/tag_names.py.
scripts/tag_names.py	New Python tag composer (CLI/rust/ref/platform).
scripts/smoke-test-image.sh	Deleted; replaced by scripts/smoke_test_image.py.
scripts/smoke_test_image.py	New Python smoke test runner for built images and OCI labels.
scripts/resolve-matrix.sh	Deleted; replaced by scripts/resolve_matrix.py.
scripts/resolve_matrix.py	New Python matrix generator for GitHub Actions fromJson().
scripts/repro-test.sh	Deleted; replaced by scripts/repro_test.py.
scripts/repro_test.py	New Python reproducibility tester (clone + two cold builds per contract).
scripts/release-push-branch.sh	Deleted; replaced by scripts/release_push_branch.py.
scripts/release_push_branch.py	New Python release branch commit/push logic with PR safety checks.
scripts/release-prepare.sh	Deleted; replaced by scripts/release_prepare.py.
scripts/release_prepare.py	New Python release preparer (pick rust keys, refresh digests/refs, validate, emit tag).
scripts/release-body.sh	Deleted; replaced by scripts/release_body.py.
scripts/release_pr_body.py	New Python helper to generate PR title/body text for release staging PRs.
scripts/release_body.py	New Python release body composer from meta-*.json artifacts.
scripts/refresh-stellar-cli-digests.sh	Deleted; replaced by scripts/refresh_stellar_cli_digests.py.
scripts/refresh-rust-digests.sh	Deleted; replaced by scripts/refresh_rust_digests.py.
scripts/refresh_stellar_cli_digests.py	New Python stellar-cli git tag -> commit SHA refresher.
scripts/refresh_rust_digests.py	New Python rust base digest refresher via buildx imagetools inspect.
scripts/publish_manifests.py	New Python workflow helper to create manifest lists per rust key.
scripts/publish_aliases.py	New Python workflow helper to repoint :<cli> and :latest aliases.
scripts/newest-pair.sh	Deleted; replaced by scripts/newest_pair.py.
scripts/newest_pair.py	New Python newest CLI / default rust selector.
scripts/lib/semver.py	Shared numeric semver parsing/sorting wrapper.
scripts/lib/rust_keys.py	Shared composite rust key parsing helpers.
scripts/lib/runner.py	Shared subprocess + HTTP JSON wrapper for testable boundaries.
scripts/lib/git_remote.py	Shared adapter for git ls-remote tag resolution.
scripts/lib/gh_cli.py	Shared adapter for gh (release list, PR list, attestation verify).
scripts/lib/docker_inspect.py	Shared adapter for buildx imagetools (digest lookup, exists, create).
scripts/lib/common.sh	Deleted; replaced by scripts/lib/common.py.
scripts/lib/common.py	Shared stderr logging, preflight checks, repo_root, sha256 helper.
scripts/lib/builds.py	Shared builds.json load/dump/query helpers with atomic writes.
scripts/lib/init.py	Declares shared lib package.
scripts/build-image.sh	Deleted; replaced by scripts/build_image.py.
scripts/build_image.py	New Python local builder for a declared (cli, rust) pair.
RELEASE.md	Updates docs/examples and workflow descriptions to reference Python scripts.
README.md	Updates tooling references and local dev requirements for the Python/uv migration.
pyproject.toml	Adds Python project metadata, dependencies, ruff, and pytest configuration.
Dockerfile	Fixes version assertion to avoid EPIPE by avoiding head piping.
.python-version	Pins Python version for uv/tooling.
.gitignore	Ignores common Python/uv/pytest/ruff artifacts.
.github/workflows/release.yml	Switches release workflow from bash + pipx to uv + Python scripts.
.github/workflows/publish.yml	Switches publish workflow from jq/awk inline logic to Python script entrypoints.
.github/workflows/lint.yml	Replaces shellcheck/shell validation with ruff + pytest jobs; updates JSON validation step.
.github/workflows/build.yml	Switches CI build/smoke/repro steps to Python scripts and installs uv.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.