Skip to content

infra: proxy.golang.org INTERNAL_ERROR aborts run-tests before test execution #1115

Description

Recurring infrastructure failure in CI

Pattern: `proxy.golang.org INTERNAL_ERROR` during `go mod download` / `go test`
Job type: `run-tests`

Evidence

Metric Value
Occurrences in last 7 days 2 (confirmed; likely more in older runs)
Example runs https://github.com/agent-substrate/substrate/actions/runs/32418052116, https://github.com/agent-substrate/substrate/actions/runs/32419279874

Impact

Both occurrences caused the entire `run-tests` job to abort before any test ran, producing a false `failure` conclusion on the workflow run. This:

  • Inflates the apparent failure rate of unit/integration tests
  • Masks real flakiness signal (flaky tests appear to have fewer total runs)
  • Causes spurious PR-blocking red checks

Log excerpts

Run 32418052116:
```
cloud.google.com/go/auth@v0.19.0: read "https://proxy.golang.org/cloud.google.com/go/auth/@v/v0.19.0.zip":
stream error: stream ID 241; INTERNAL_ERROR; received from peer
```

Run 32419279874:
```
k8s.io/metrics@v0.36.1: read "https://proxy.golang.org/k8s.io/metrics/@v/v0.36.1.zip":
stream error: stream ID 137; INTERNAL_ERROR; received from peer
```

Triple-check (infra vs test failure)

  1. Pattern match ✓ — both logs contain `INTERNAL_ERROR` from `proxy.golang.org` during module download
  2. Timeline ✓ — error appears before the first `=== RUN` line; zero test results in log
  3. Cross-run consistency ✓ — two separate runs, same failure pattern, different modules (cloud.google.com/go/auth and k8s.io/metrics), indicating the proxy itself is intermittently failing

Suggested fixes

  • Add `GONOSUMCHECK=*` + `GOFLAGS=-mod=mod` and retry on proxy error
  • Or add `GOPROXY=off` after a successful first download (use module cache) and fail-fast only on genuine source changes
  • Consider switching to `GOPROXY=direct` as fallback in CI

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions