fix: backfill rollout status fields from logs when polling completes

[SunnySoldier357]

Summary

Fireworks Tracing Tests is red on main and on every PR — see run 25409077579. The failure is a regression introduced by #446:

ERROR:root:❌ Rollout failed (non-retryable error encountered): InternalError()
    assert row.rollout_status.message == "test error"
E   AssertionError: assert '' == 'test error'

The lightweight /status endpoint on the tracing gateway is a point-read on the Status Spanner table, which only stores RolloutId, AccountId, StatusCode. Message, Details, and Extras still live exclusively on the Logs table. After #446 dropped the /logs backfill (commit 20b0f23), the SDK was constructing Status(code=..., message="", details=[]) on every completed rollout and EvalProtocolError(message="") on every failure — which is what the propagate-status integration test catches.

This PR restores the two-phase polling shape from the original design of #446:

1. Poll /status for the status code (cheap point-read, runs every poll_interval).
2. On a terminal (non-RUNNING) code, do exactly one async_search_logs call to backfill message / details / extras from the matching log row.

That's still ~1000× cheaper on the Logs table than the pre-#446 polling loop, because the search runs once per rollout completion instead of every poll interval.

The cleanest long-term fix is on the gateway side: extend either the Status Spanner schema (and spanner_reader.get_status) to also persist these fields, or have the /status handler do an internal Logs read on terminal codes and inline them into the response. Either change would let the SDK go back to a single read per completion. Filing a follow-up for that.

Test plan

• Fireworks Tracing Tests workflow goes green on this branch.
• test_remote_rollout_and_fetch_fireworks (happy path) still passes.
• test_remote_rollout_and_fetch_fireworks_propagate_status sees Code.INTERNAL with message == "test error".

Made with Cursor

---

Note

Medium Risk
Changes terminal-status handling in RemoteRolloutProcessor by adding a one-time logs lookup and merging returned extras, which affects error propagation and could introduce edge cases if log entries are missing or mismatched.

Overview
Restores two-phase rollout polling in RemoteRolloutProcessor: continue polling /status for the status code, then on a terminal code perform a single async_search_logs query to backfill Status.message, Status.details, and execution extras.

When backfilling, it selects the log entry whose embedded status code matches the terminal /status code (avoiding intermediate RUNNING checkpoints) and filters out noisy extras keys before merging into row.execution_metadata.extra.

^{Reviewed by Cursor Bugbot for commit 3298857. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a4b13417d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".