Skip to content

ci: resilient protoc install to fix flaky release-CDN 504s - #557

Merged
moonming merged 1 commit into
mainfrom
ci/protoc-resilient-install
Jun 8, 2026
Merged

ci: resilient protoc install to fix flaky release-CDN 504s#557
moonming merged 1 commit into
mainfrom
ci/protoc-resilient-install

Conversation

@moonming

@moonming moonming commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Problem

arduino/setup-protoc@v3.0.0 flakes intermittently with

Error: Failed to download version v23.4: Unexpected HTTP response: 504

on protoc-23.4-linux-x86_64.zip from the GitHub release CDN (objects.githubusercontent.com). It hits whichever of the four protoc-needing jobs (lint, schema drift, rust unit + coverage, build aisix (instrumented)) loses the lottery, failing unrelated PRs. When build aisix (instrumented) is the loser, the dependent e2e + coverage >= 90% jobs are skipped, so a green-code PR shows red CI (this blocked #550 across three reruns).

repo-token is already set on all four steps — but it only authenticates the GitHub API release lookup, not the asset download, so it does not prevent the 504.

Fix

A local composite action .github/actions/setup-protoc that downloads the same pinned protoc 23.4 with hard retries (curl --retry 5 --retry-all-errors inside an outer 5× loop, 15 s backoff) and installs it under $HOME (no sudo), exposed to later steps via $GITHUB_PATH. Drop-in replacement at all four call sites (−12/+4 in ci.yml).

  • Keeps protoc 23.4 — no risk of a distro protoc (e.g. the apt package on ubuntu-22.04 lacks proto3 optional) silently changing build behavior.
  • DRY — one action / one version / one retry policy for all four jobs (previously four copies of the action block).
  • No sudo — installs under $HOME/.local, pins PROTOC for prost/tonic build scripts.

Validation

This PR modifies the workflow, so its own CI runs every job through the new install path — if they go green (incl. build aisix (instrumented)e2e), the fix is proven against the real flake.

Why not apt / a retry-action

  • apt protobuf-compiler is the simplest "never touch the CDN" fix but changes the protoc version (version-dependent build risk) — rejected to stay zero-risk.
  • A third-party retry-wrapper action adds a supply-chain dependency for what a ~14-line shell step does.

Integrity note: the asset is fetched over HTTPS from GitHub's official release CDN (TLS-authenticated), matching the integrity guarantee of the action it replaces (neither verifies a SHA-256). A pinned checksum can be added later if desired.

@coderabbitai

coderabbitai Bot commented Jun 8, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@moonming, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 2 minutes and 39 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Free

Run ID: b99b0ef9-4b17-4217-95d5-c4c66a172dd6

📥 Commits

Reviewing files that changed from the base of the PR and between fb1d79c and aefb179.

📒 Files selected for processing (2)
  • .github/actions/setup-protoc/action.yml
  • .github/workflows/ci.yml

Note

🎁 Summarized by CodeRabbit Free

Your organization has reached its limit of developer seats under the Pro Plan. For new users, CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please add seats to your subscription by visiting https://app.coderabbit.ai/login.If you believe this is a mistake and have available seats, please assign one to the pull request author through the subscription management page using the link above.

Comment @coderabbitai help to get the list of available commands and usage tips.

arduino/setup-protoc@v3.0.0 downloads protoc-23.4 from the GitHub release CDN
(objects.githubusercontent.com), which intermittently returns HTTP 504. The
asset download is plain CDN traffic — repo-token (already set on all four steps)
only authenticates the GitHub API release lookup, not the download — so the
existing token does not prevent the flake. It hits whichever protoc-needing job
loses the lottery (lint / schema-drift / rust-unit / build-bin), failing
unrelated PRs; when build-bin is the loser, the dependent e2e + coverage jobs
are skipped, so a green-code PR shows red CI.

Replace it with a local composite action .github/actions/setup-protoc that
downloads the SAME pinned protoc 23.4 with hard retries (curl --retry plus an
outer 5x loop, 15s backoff) and installs it under $HOME (no sudo), exposing it
via $GITHUB_PATH and pinning $PROTOC for prost/tonic build scripts. Keeps the
exact version (no build risk from a distro protoc that may lack proto3
'optional') and DRYs the four call sites behind one action.

Refs #550 (whose instrumented build kept losing the 504 lottery).
@moonming
moonming force-pushed the ci/protoc-resilient-install branch from b38cdf6 to aefb179 Compare June 8, 2026 07:19
@moonming
moonming merged commit 963068f into main Jun 8, 2026
7 checks passed
@moonming
moonming deleted the ci/protoc-resilient-install branch June 8, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant