Skip to content

fix(ci): push tutorial images from buildx instead of re-pushing a stale local tag - #478

Merged
deepthi-rao-scale merged 2 commits into
nextfrom
fix/tutorial-agent-push-directly
Jul 29, 2026
Merged

fix(ci): push tutorial images from buildx instead of re-pushing a stale local tag#478
deepthi-rao-scale merged 2 commits into
nextfrom
fix/tutorial-agent-push-directly

Conversation

@deepthi-rao-scale

@deepthi-rao-scale deepthi-rao-scale commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Problem

020_state_machine:latest (and other tutorial-agent images) have been frozen at a December 2025 build and re-published unchanged on every run — which is why the merged mcp<2 pin never reached the image and scale-agentex's 020-state-machine integration test keeps failing on the mcp 2.0.0 McpErrorMCPError crash.

Root cause — the publish path re-pushes a stale image, not the one it just built:

# build step: multi-platform, NO --push
agentex agents build ... --platforms linux/amd64,linux/arm64
# validate step:
docker run "$FULL_IMAGE" ...     # pulls the EXISTING :latest from ghcr
# push step:
docker push "$FULL_IMAGE"        # re-pushes that pulled (stale) image

A multi-platform buildx build cannot be loaded into the local Docker image store, so the build step never creates a local :latest. The validation step's docker run "$FULL_IMAGE" then pulls the existing (stale) :latest from the registry, and the separate docker push re-pushes that same stale image back. The freshly-built image is silently discarded — so :latest is self-perpetuating and stuck at its December digest.

This was verified end-to-end:

  • A local single-arch docker build from the pinned source produces a fresh image with the pin.
  • CI (multi-arch, build-then-docker push) re-pushed the identical December digest (1f90…, Created 2025-12-30, no pin) even with --no-cache and SDK 0.22.0.

Fix

  • On the publish path, pass --push to agentex agents build so buildx sends the freshly-built multi-arch image straight to the registry.
  • Drop the now-redundant separate docker push step. Validation still runs and now exercises the freshly-pushed image.
  • Revert the CLI build-cache default back to True (keeping the --cache/--no-cache flag). The staleness was never a layer-cache problem, so forcing --no-cache globally only slowed builds.

Effect

The next publish (push to main under examples/tutorials/**, or a rebuild_all dispatch) will push the freshly-built, pinned 020_state_machine:latest, turning the scale-agentex integration check green — and fixes the same silent-stale-publish for every tutorial agent.

Note on the earlier PR

This supersedes the root-cause theory behind the earlier no-cache change (#476): --no-cache was confirmed to run in CI yet the image stayed stale, which is what led to finding the real build/push bug here.

🤖 Generated with Claude Code

Greptile Summary

This PR fixes a long-standing silent publish bug where multi-platform buildx builds were being silently discarded and a stale :latest re-pushed in its place. The fix changes the publish path to push directly from buildx to an immutable SHA-tagged candidate, validate that exact image, then promote it to :latest via docker buildx imagetools create — keeping :latest clean until validation passes. The cache default is also reverted to True since caching was never the root cause of staleness.

  • Publish path overhauled: build step now passes --push so buildx sends the fresh multi-arch manifest straight to the SHA tag; a separate "promote" step then points :latest at that validated digest with docker buildx imagetools create, preserving the multi-arch manifest list without rebuilding.
  • Validation now exercises the live SHA-tagged image: because multi-platform images cannot be loaded into the local Docker store, validation pulls the freshly-pushed registry image — an inherent constraint of multi-arch buildx noted in the PR description.
  • Cache default reverted: cache: bool = True in both agents.py and agent_handlers.py; --no-cache remains available but is no longer the default since layer-cache staleness was not the root cause.

Confidence Score: 5/5

Safe to merge — the change correctly routes the publish path through buildx --push, validates the live SHA-tagged image, and only then promotes :latest.

The three changed files are tightly scoped: the workflow correctly pushes from buildx to an immutable SHA tag, validates that exact image, and promotes :latest via a registry-side manifest copy. The Python cache-default reversal is a one-line change with no side effects. The only finding is a dead env variable that is exported but never read, which has no functional impact.

Files Needing Attention: No files require special attention beyond a quick read of the SHOULD_PUSH vs PROMOTE_LATEST interplay in the workflow.

Important Files Changed

Filename Overview
.github/workflows/build-and-push-tutorial-agent.yml Core fix: publish path now pushes from buildx directly to a SHA tag, then promotes to :latest after validation — eliminating the stale-re-push bug. One dead env-var (SHOULD_PUSH) is exported but never referenced by any subsequent step condition.
src/agentex/lib/cli/commands/agents.py Reverts cache default from False to True; docstring updated to match. Straightforward and correct.
src/agentex/lib/cli/handlers/agent_handlers.py Reverts cache default from False to True; docstring updated accordingly. No logic changes.

Reviews (2): Last reviewed commit: "validate before promoting :latest (candi..." | Re-trigger Greptile

deepthi-rao-scale and others added 2 commits July 29, 2026 17:58
…le local tag

The publish path built a multi-platform image without --push, then a separate
'docker push' step shipped ':latest'. But a multi-arch buildx build can't be
loaded into the local Docker image store, so no fresh local ':latest' exists;
the preceding validation step's 'docker run' pulls the *existing* (stale)
':latest' from ghcr, and 'docker push' then re-pushes that same stale image.
Result: ':latest' has been frozen at a Dec-2025 build, re-pushed unchanged on
every run, which is why the 020_state_machine agent's mcp<2 pin never reached
the published image and the scale-agentex integration test kept failing.

Fix: on the publish path, pass --push to 'agentex agents build' so buildx sends
the freshly-built multi-arch image straight to the registry, and drop the
now-redundant separate 'docker push' step. Validation still runs and now
exercises the freshly-pushed image.

Also revert the CLI build cache default back to True (keep the --cache/--no-cache
flag): the staleness was never a layer-cache problem, so forcing --no-cache
globally only slowed builds without fixing anything.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address the validate-after-publish gap: build+push to an immutable candidate
tag (the commit SHA), validate that exact pushed image, then promote :latest
onto it via 'docker buildx imagetools create' (a registry-side manifest copy,
no rebuild). If validation fails, :latest is left on the last known-good build
and only the SHA tag is dirty -- no unvalidated image ever lands on :latest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@deepthi-rao-scale
deepthi-rao-scale merged commit a7c8f5a into next Jul 29, 2026
81 of 83 checks passed
@deepthi-rao-scale
deepthi-rao-scale deleted the fix/tutorial-agent-push-directly branch July 29, 2026 23:08
@stainless-app stainless-app Bot mentioned this pull request Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants