flake: macOS Go setup failure - illegal byte sequence during tar extraction

[blink-so]

CI Failure Details

CI Run: https://github.com/coder/coder/actions/runs/18569038887
Failed Job: test-go-pg (macos-latest) - https://github.com/coder/coder/actions/runs/18569038887/job/52937928968
Commit: a53a5682d51d35e3d2f62a7fa8c93e22db2f3103
Commit Author: Steven Masley (@Emyrk)
Date: 2025-10-16 17:07:21 UTC (failed at 17:19:12 UTC)

Root Cause Analysis

Error Classification: D. Infrastructure

This is a macOS runner infrastructure issue, NOT a code or test issue. The failure occurs during the "Setup Go" step when attempting to extract the Go 1.24.6 distribution.

Failure Details

The job failed during Go installation with:

./test/weak.go: Write failed: Illegal byte sequence
./test/floatcmp.go: Write failed: Illegal byte sequence
./test/mergemul.go: Write failed: Illegal byte sequence
...
tar: Error exit delayed from previous errors.
The process '/usr/bin/tar' failed with exit code 1

After the first attempt failed, the setup action tried downloading Go from the official distribution:

Falling back to download directly from Go
Acquiring go1.24.6 from https://storage.googleapis.com/golang/go1.24.6.darwin-arm64.tar.gz

However, this also failed when trying to copy the extracted files to the tool cache:

##[error]Failed to download version 1.24.6: Error: EILSEQ: illegal byte sequence, copyfile 
'/Users/runner/work/_temp/18008491-fb31-43cf-be4f-1c9e7fbac9c8/go/src/encoding/json/indent.go' -> 
'/Users/runner/work/_tool/go/1.24.6/arm64/src/encoding/json/indent.go'

Root Cause

The error EILSEQ: illegal byte sequence indicates a filesystem encoding issue on the macOS runner. This is typically caused by:

1. Locale/encoding mismatch: The runner's filesystem or locale settings are incompatible with certain file contents
2. Corrupted filesystem state: The ephemeral runner's filesystem may be in a corrupted state
3. macOS runner infrastructure issue: The depot macOS runner (depot-bgbxhchqtp) may have environment issues

Key Evidence:

• The commit being tested is unrelated (updates to coder/guts dependency)
• No code changes that would affect Go installation
• Failure is in the CI infrastructure setup, not in test execution
• The error affects multiple files in the Go distribution during extraction
• Both the cached and fresh download attempts failed with encoding errors

No Duplicate Issues Found

Searched for:

• repo:coder/internal illegal byte sequence macOS
• repo:coder/internal Setup Go macOS
• repo:coder/internal copyfile EILSEQ
• repo:coder/internal tar Write failed macOS

No existing issues found for this specific failure mode.

Assignment

Unassigned - This is a CI infrastructure/runner issue, not a code issue.

Recommended Actions

1. Immediate: Retry the CI run - this may be a transient runner issue
2. Short-term: Monitor for recurrence to determine if this is a systemic macOS runner problem
3. Investigation: If recurring, check:
   • Depot macOS runner environment configuration
   • Locale and encoding settings on macOS runners
   • macOS runner filesystem health
4. Workaround: Consider adding retry logic to the Go setup step

Impact

• Blocks CI on main branch
• No actual test failures - pure infrastructure issue
• Affects only macOS runners (Ubuntu and Windows jobs succeeded)

Related Context

• Runner: depot-bgbxhchqtp (macOS-latest, Apple Silicon ARM64)
• Go version being installed: 1.24.6
• Cache restore had earlier warning: "Failed to restore: Premature close"

---

Investigated by Blink at 2025-10-16 17:19 UTC

[dmitshur]

Note that the official Go distributions can be downloaded (following redirects) using URLs at https://go.dev/dl/, which follow the pattern https://go.dev/dl/{file name}. Attempting to download those files from the storage.googleapis.com/golang URL is not supported and won't keep working.

[Emyrk]

We rely on this github action for Go setup: https://github.com/actions/setup-go

We are on version v5.0.2 (Jun 27, 2024). The latest is v6.0.0 on Sep 3, 2025.

We should probably update to the latest