fix(tracing): catch only the context reset on generator close

[adityasingh2400]

Follow-up to #4221, requested by @seratch there.

#4221 moved the foreign-token tolerance into _finish_on_generator_exit, but that helper wraps the whole finish() call. TraceImpl.finish runs the processor before resetting the scope:

def finish(self, reset_current: bool = False):
    if not self._started:
        return
    self._processor.on_trace_end(self)          # can raise
    if reset_current and self._prev_context_token is not None:
        Scope.reset_current_trace(self._prev_context_token)
        self._prev_context_token = None

So a processor whose on_trace_end raises ValueError is mistaken for a foreign context token. The error is swallowed, the saved token is dropped, and the reset never runs, which leaves the finished trace current for everything that runs after the close. Closing from the same task returns normally while the scope is quietly wrong.

The change

Run finish(reset_current=False) so processor failures propagate normally, then reset the scope separately and tolerate only that failure, which is the one that genuinely cannot succeed when another task finalizes the generator:

trace.finish(reset_current=False)

token = trace._prev_context_token
if token is None:
    return
trace._prev_context_token = None
try:
    Scope.reset_current_trace(token)
except ValueError:
    logger.debug("Skipping trace context reset, token belongs to another context")

Verified

• test_generator_close_surfaces_processor_failure fails on current main with DID NOT RAISE <class 'ValueError'> and passes here. It also asserts the token is left in place rather than silently dropped, since the reset never ran.
• tests/tracing is 49 passed, with ruff check, ruff format --check, and mypy clean on both changed files.
• The generator-close behavior added in fix(tracing): release the trace scope when a generator is closed #4221 is unchanged: same-task closes still reset, and a cross-task close still tolerates the foreign token.

[seratch]

Please run the reset cleanup in a finally block so that a finish() failure does not leave the finished trace current in the same task. Also clear _prev_context_token only after the reset succeeds or its expected foreign-context ValueError has been handled; clearing it before the reset loses the ownership handle if any unexpected reset error occurs. Update the regression to assert that the original failure propagates while Scope.get_current_trace() is restored and the token is cleared.

This keeps the cross-task behavior from #4221 while preserving cleanup ownership on failure. After that focused change and green CI, this should be ready for another review.

[adityasingh2400]

All three points done in b15119f. You are right that my version traded one leak for another: a failing finish skipped the reset entirely, so a processor error left the finished trace current in the same task, which is the thing the PR was supposed to stop.

try:
    trace.finish(reset_current=False)
finally:
    token = trace._prev_context_token
    if token is not None:
        try:
            Scope.reset_current_trace(token)
        except ValueError:
            logger.debug("Skipping trace context reset, token belongs to another context")
        trace._prev_context_token = None

The finally releases the scope either way. Clearing the token after the inner try rather than before it means an unexpected reset failure propagates with _prev_context_token still set, so the ownership handle survives, while a success or the expected foreign-context ValueError clears it.

The regression now asserts what you asked for. It still expects the processor's ValueError to be the exception that propagates, and additionally that Scope.get_current_trace() is None and _prev_context_token is None afterwards. On the previous commit it fails with:

assert <agents.tracing.traces.TraceImpl object at ...> is None
 +  where <...> = get_current_trace()

which is exactly the finished trace being left current.

Verification: tests/tracing is 49 passed, with ruff, ruff format, and mypy clean. I ran mypy over the test file as well as the source this time, since I had a typecheck failure earlier today from checking only the source.

Cross-task behavior from #4221 is unchanged, and I confirmed this does not overlap #4233, which touches spans.py and test_spans_impl.py only.