Skip to content

fix(deploy): tag tarballs with run_id to prevent parallel-deploy clobber#175

Merged
JohnRDOrazio merged 1 commit into
mainfrom
fix/deploy-tarball-race
Jun 5, 2026
Merged

fix(deploy): tag tarballs with run_id to prevent parallel-deploy clobber#175
JohnRDOrazio merged 1 commit into
mainfrom
fix/deploy-tarball-race

Conversation

@JohnRDOrazio
Copy link
Copy Markdown
Member

Problem

The deploy workflow's concurrency group is keyed per-environment:

concurrency:
  group: deploy-${{ github.event_name == 'release' && 'production' || inputs.environment }}

So a production dispatch and a staging dispatch can — and do — run in parallel. Both used the same /tmp/<name>.tar.gz paths on the shared VPS, racing each other:

  1. Runner A and Runner B both scp to /tmp/nextjs-bundle.tar.gz
  2. The second upload wins (overwrites the first)
  3. The first SSH-extract sees the OTHER deploy's bundle
  4. After extract, that first deploy rms the tarball
  5. The second extract finds the file gone → exit 1

Today's deploy hit this (staging run 27030067375, prod run 27030068492):

  • Production upload landed first at 17:32:23Z
  • Staging upload landed second at 17:32:25Z (overwriting)
  • Staging extracted its bundle (correct by upload ordering luck) and rmd it at 17:32:25–27Z
  • Production's extract found nothing at 17:32:27Z → failed

Staging deployed the right code by luck of upload ordering. Had the order flipped, staging would have silently deployed main's tarball to staging's APP_DIR — a worse failure mode (silent wrong-branch deploy) than the visible exit-1 we got.

Fix

Tag every /tmp tarball with $BUNDLE_TAG = github.run_id (system-assigned integer per workflow run — no injection risk):

/tmp/nextjs-bundle-<run_id>.tar.gz
/tmp/wp-theme-<run_id>.tar.gz
/tmp/wp-plugin-redis-translations-<run_id>.tar.gz
/tmp/wp-plugin-cdcf-mcp-<run_id>.tar.gz

BUNDLE_TAG is added to the job env: so every step inherits it; the SSH commands interpolate it runner-side (same pattern as the existing $APP_DIR / $WP_THEME_DIR) before sending the literal text to the VPS shell.

Concurrency group is unchanged — it still prevents a second push to the SAME environment from racing. The change only enables parallel prod + staging to coexist on disk.

Diff stats

  • 1 file changed, 18 insertions(+), 13 deletions(-)
  • 1 new env var, 13 tarball-path substitutions across create / upload / extract / rm
  • YAML syntax: python -c "import yaml; yaml.safe_load(...)" clean

Test plan

  • Manual: dispatch staging deploy of this branch (fix/deploy-tarball-race) and confirm green
  • Manual: dispatch staging + production at the same time after merge, confirm BOTH succeed (the failure mode this fixes)
  • If the second test fails, the prior race wasn't the only issue — re-investigate before more parallel deploys

…oy clobber

The deploy workflow's concurrency block keys per-environment, so prod
and staging dispatches can run in parallel. Both used the same
/tmp/<name>.tar.gz paths on the shared VPS, racing each other:

  1. Runner A and Runner B both scp to /tmp/nextjs-bundle.tar.gz
  2. The second upload wins (overwrites)
  3. The first SSH-extract sees the OTHER deploy's bundle
  4. After extract, that first deploy rm's the tarball
  5. The second extract finds the file gone → exit 1

Today's deploy hit this: staging dispatched 1s after production,
production's upload landed first (17:32:23Z), staging's overwrote
(17:32:25Z), staging extracted ITS bundle (correct branch by upload
ordering luck) and rm'd it, production's extract found nothing.
Production failed; staging happened to deploy the right code, but the
race could equally have flipped — staging would have silently
deployed main's tarball to staging's APP_DIR.

Fix: tag every /tmp tarball with $BUNDLE_TAG = github.run_id (a
monotonic system-assigned integer per workflow run — no injection
risk). Each deploy now owns a unique filename:

  /tmp/nextjs-bundle-<run_id>.tar.gz
  /tmp/wp-theme-<run_id>.tar.gz
  /tmp/wp-plugin-redis-translations-<run_id>.tar.gz
  /tmp/wp-plugin-cdcf-mcp-<run_id>.tar.gz

BUNDLE_TAG is added to the job env so every step inherits it; the SSH
commands interpolate it runner-side (same pattern as the existing
$APP_DIR / $WP_THEME_DIR) before sending the literal text to the VPS
shell.

Concurrency group is unchanged — it still prevents a second push to
the SAME environment from racing. The change only enables parallel
prod + staging to coexist on disk.

Verification: `python -c "import yaml; yaml.safe_load(...)"` clean;
14 BUNDLE_TAG references confirmed across create/upload/extract/rm.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Jun 5, 2026

Warning

Review limit reached

@JohnRDOrazio, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 26 minutes and 35 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 617b6f40-45ae-4b0f-b9c6-1832078e4db2

📥 Commits

Reviewing files that changed from the base of the PR and between a6a12e4 and 0943bc6.

📒 Files selected for processing (1)
  • .github/workflows/deploy.yml
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/deploy-tarball-race

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codacy-production
Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@JohnRDOrazio
Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Jun 5, 2026

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@JohnRDOrazio JohnRDOrazio merged commit 7c56d56 into main Jun 5, 2026
11 checks passed
@JohnRDOrazio JohnRDOrazio deleted the fix/deploy-tarball-race branch June 5, 2026 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant