Skip to content

fix(ci): add retry logic to sccache --start-server in setup-sccache action#60577

Merged
gantoine merged 3 commits into
masterfrom
mendral/sccache-retry-startup
May 29, 2026
Merged

fix(ci): add retry logic to sccache --start-server in setup-sccache action#60577
gantoine merged 3 commits into
masterfrom
mendral/sccache-retry-startup

Conversation

@mendral-app
Copy link
Copy Markdown
Contributor

@mendral-app mendral-app Bot commented May 28, 2026

Summary

  • Wrap sccache --start-server in a 3-attempt retry loop (5s delay between attempts) to handle transient WebDAV 500 errors that have been causing ~1-2 CI failures per week for 4+ months
  • On persistent failure, gracefully degrade by unsetting RUSTC_WRAPPER so builds complete (slower, without cache) rather than failing the entire job

Context

Related insight: sccache WebDAV backend 500 errors causing recurring Rust CI failures

The WebDAV cache backend intermittently returns HTTP 500 on the startup health check (GET .sccache_check). Since the setup-sccache action had no retry logic, every transient error immediately failed the step and the entire job. 22+ occurrences observed across 14+ distinct backend IPs over 4 months.


Note

Created by Mendral. Tag @mendral-app with feedback or questions.

…ction

Wrap sccache --start-server in a 3-attempt retry loop with 5-second delays
to handle transient WebDAV 500 errors. On persistent failure, gracefully
degrade by unsetting RUSTC_WRAPPER so builds complete without cache rather
than failing the entire job.

This addresses recurring CI failures (~1-2/week over 4+ months) caused by
the WebDAV cache backend returning HTTP 500 on health check.
@mendral-app mendral-app Bot marked this pull request as ready for review May 28, 2026 23:12
@assign-reviewers-posthog assign-reviewers-posthog Bot requested a review from a team May 28, 2026 23:12
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 28, 2026

Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
.github/actions/setup-sccache/action.yml:71-87
Setting `RUSTC_WRAPPER=sccache` before the retry loop and then attempting to clear it with `RUSTC_WRAPPER=` (empty string) is risky. GitHub Actions sets the env var to an empty string rather than unsetting it, and cargo's handling of `RUSTC_WRAPPER=""` varies by version — older cargo releases may attempt to exec an empty string as the wrapper instead of treating it as "no wrapper", causing builds to fail anyway. The intent of graceful degradation is better served by only writing `RUSTC_WRAPPER=sccache` on success, eliminating the need for the override entirely.

```suggestion
              echo "$RUNNER_TEMP/sccache" >> "$GITHUB_PATH"
              for i in 1 2 3; do
                if "$RUNNER_TEMP/sccache/sccache" --start-server; then
                  echo "sccache server started successfully"
                  echo "RUSTC_WRAPPER=sccache" >> "$GITHUB_ENV"
                  break
                fi
                if [ "$i" -eq 3 ]; then
                  echo "::warning::sccache failed to start after 3 attempts, continuing without cache"
                  exit 0
                fi
                echo "sccache startup attempt $i failed, retrying in 5 seconds..."
                "$RUNNER_TEMP/sccache/sccache" --stop-server 2>/dev/null || true
                sleep 5
              done
              "$RUNNER_TEMP/sccache/sccache" --version
```

Reviews (1): Last reviewed commit: "fix(ci): add retry logic to sccache --st..." | Re-trigger Greptile

Comment thread .github/actions/setup-sccache/action.yml
Move RUSTC_WRAPPER=sccache into the success branch of the retry loop
instead of setting it before and clearing on failure. This avoids
issues with cargo interpreting an empty RUSTC_WRAPPER as a command
to execute.
@mendral-app mendral-app Bot changed the title Add retry logic to sccache --start-server in setup-sccache action fix(ci): add retry logic to sccache --start-server in setup-sccache action May 28, 2026
@gantoine gantoine enabled auto-merge (squash) May 29, 2026 13:01
@gantoine gantoine self-assigned this May 29, 2026
@gantoine gantoine merged commit f9028b3 into master May 29, 2026
147 checks passed
@gantoine gantoine deleted the mendral/sccache-retry-startup branch May 29, 2026 13:13
@deployment-status-posthog
Copy link
Copy Markdown

deployment-status-posthog Bot commented May 29, 2026

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-05-29 13:41 UTC Run
prod-us ✅ Deployed 2026-05-29 14:09 UTC Run
prod-eu ✅ Deployed 2026-05-29 14:12 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant