[P3] Compress temp files in os.TempDir() not cleaned on atelet crash — /tmp fills up

[aditya-shantanu]

Severity: P3 (slow disk leak in /tmp; accelerates ENOSPC via ISSUE-007)
Component: Node Layer — cmd/atelet/internal/ategcs/objects.go
Audit ID: NL-9

---

Summary

sendBufferedZstd (used for S3/non-streaming GCS uploads) creates a temp file via
os.CreateTemp("", "substrate-upload-compress-") in os.TempDir() (typically /tmp).
The deferred os.Remove cleanup only runs if the process exits normally. A SIGKILL
(OOM kill, node restart, crash) leaves potentially multi-GiB temp files in /tmp.
There is no startup sweep to clean these up. Repeated crashes during large checkpoint
uploads can exhaust /tmp.

---

Root Cause

File: cmd/atelet/internal/ategcs/objects.go lines 186–212

func sendBufferedZstd(ctx context.Context, ...) error {
    tmpFile, err := os.CreateTemp("", "substrate-upload-compress-")
    if err != nil { return err }
    defer os.Remove(tmpFile.Name())  // ← NOT called on SIGKILL

    // Write compressed data to tmpFile (may be hundreds of MB)
    // Upload tmpFile to S3/GCS
    return nil
}

No code in atelet/main.go's startup path sweeps for leftover substrate-upload-compress-*
files.

---

Steps to Reproduce

# Start a large checkpoint upload (use a big actor snapshot)
kubectl ate suspend actor large-actor -a demo &

# Kill atelet mid-upload
kubectl exec -n ate-system <atelet-pod> -- kill -9 1

# Check /tmp
kubectl exec -n ate-system <atelet-pod> -- ls -lh /tmp/substrate-upload-compress-*
# Files remain, potentially several GB each

---

Suggested Fix

On atelet startup, add a sweep:

func sweepCompressTempFiles() {
    pattern := filepath.Join(os.TempDir(), "substrate-upload-compress-*")
    files, _ := filepath.Glob(pattern)
    for _, f := range files {
        _ = os.Remove(f)
    }
}

Call sweepCompressTempFiles() in main.go before the gRPC server starts.

Alternatively, create temp files inside BasePath (the atelet's dedicated volume)
rather than os.TempDir(), so they are isolated from system /tmp and can be swept
by the existing per-actor cleanup paths.

[lorenzo85]

Hi, I'd like to work on this.

My plan is to introduce a generic internal/startupsweep package that lets any package register cleanup tasks to be executed on startup.

Both ategcs and imagecache (which has a similar problem with .tmp-* and .rm-* dirs left behind on crash) would register their entries, and main.go would run a single sweeper.Sweep(ctx) call before the gRPC server starts.

I've put together a fix — draft PR here: #685