feat(otel): add cold start metrics

[Krisztian F (krisztianfekete)]

This PR implements the last two metrics from #433.

Right now, we can see that a resume was slow but not where. ate.actor.lifecycle.operation.duration covers the whole ateapi operation, and atenet.router.route.duration covers the edge, but everything between ateapi-atelet-actors is one block that contains fetching the manifest, downloading the snapshot, unpack the OCI image, call to ateom.

We have rpc.server.call.duration that gives us the atelet restore total time, but template, kind, and scope labels are missing, so today we cannot really pinpoint why/where we have a regressions in latency.

In this PR I am adding per-phase histograms, here's an example of what we can know after these changes:

ateom_restore   522 ms   ###############################
download        8.9 ms   #
manifest_fetch  3.2 ms
oci_unpack      2.6 ms
---------------------------------------------------------
total           535 ms

It also fixes a gap #683 opened where a data_on_golden resume was labeled identically to a plain one on the lifecycle histogram.

Things folks might want to argue with:

• total as a phase value. Partly duplicates rpc.server.call.duration, but that one has no domain labels and gRPC-specifc. We can drop it, but it's an inferior operational UX, so I'd rather have it here.
• Phases overlap, they are not a partition of total, because download runs concurrently with the asset fetch and unpack. I called this out in the metric description. Do not sum across phases.
• New ate.snapshot.scope key rather than a new ate.snapshot.kind value for data_on_golden. A new value would collapse local and external into one bucket, which is the biggest latency difference there is. This does add a label to the already shipped lifecycle histogram.

Verified on kind, and all e2e suites pass, and the emitted series cover every kind (golden, latest, local) on both metrics with no unknown values.

• Tests pass
• Appropriate changes to documentation are included in the PR

[Krisztian F (krisztianfekete)]

Adding the last two metrics from #433, cc. Jeff Luo (@JeffLuoo), Da Huang (@git286).

[Jeff Luo (JeffLuoo)]

failedPhase and the error can come from different goroutines, so the reason gets pinned to the wrong phase.

errgroup.WithContext cancels gctx as soon as either goroutine fails. If the asset/OCI-prep goroutine fails first, the in-flight download aborts with a context error and its defer sets downloadFailedPhase — even though it was only collateral. Here download wins unconditionally, while g.Wait() returned the prep error. The datapoint reads:

ate_snapshot_phase="download", ate_failure_reason="INVALID_CONTAINER_CONFIG"

The error is correct; the phase isn't. On a cold node the download is the long-running one, so it's still in flight almost every time prep fails.

Wait returns the first error verbatim, so identity tells you which goroutine owns it:

var downloadErr, prepErr error

g.Go(func() (err error) {
    t := time.Now()
    defer func() {
        dDownload = time.Since(t)
        downloadErr = err
        if err != nil {
            downloadFailedPhase = ateattr.SnapshotPhaseDownload
        }
    }()
    ...
})

g.Go(func() (err error) {
    defer func() { prepErr = err }()
    ...
})

// The cancelled goroutine's error is only gctx fallout; it must not claim the
// phase.
if err := g.Wait(); err != nil {
    switch err {
    case downloadErr:
        op.failedPhase = downloadFailedPhase
    case prepErr:
        op.failedPhase = prepFailedPhase
    }
    return nil, err
}

errors.Is(err, context.Canceled) won't work as a filter here — CrashIfReason and the GCS wrapping don't reliably preserve it.

[Krisztian F (krisztianfekete)]

Thanks!
Fixed this and while I testing I found the same bug in the durations, which is also fixed now. Also swapped ate.snapshot.file for the semconv's file.name on atelet.snapshot.size.

[Jeff Luo (JeffLuoo)]

Please rebase, thanks