Using isolation scopes with multiple depths of generators causes Token was created in different Context error

[ntindle]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.33.2

Steps to Reproduce

#!/usr/bin/env python3
"""
Minimal reproduction case for Sentry isolation_scope + async generator bug.

Bug: Using sentry_sdk.isolation_scope() as a context manager around an async
generator that yields causes "Token was created in a different Context" error
when the generator exits early (via break, exception, or garbage collection).
"""

import asyncio
import sentry_sdk
from sentry_sdk.integrations.asyncio import AsyncioIntegration


async def inner_generator():
    """Simple async generator that yields values"""
    for i in range(3):
        print(f"  Inner generator yielding: {i}")
        yield i


async def problematic_async_generator():
    """
    This pattern causes the context error.
    The isolation_scope wraps yield statements in an async generator.
    """
    with sentry_sdk.isolation_scope() as scope:
        scope.set_user({"id": "test-user-123"})
        scope.set_tag("example", "value")

        async for value in inner_generator():
            # THIS YIELD INSIDE ISOLATION_SCOPE IS THE PROBLEM
            yield value


async def main():
    print("Reproducing Sentry isolation_scope + async generator bug\n")
    print("=" * 60)

    # Initialize Sentry
    sentry_sdk.init(
        dsn=None,  # No DSN needed to reproduce
        integrations=[AsyncioIntegration()],
        debug=True,
    )

    print("Test 1: Normal completion (works fine)")
    print("-" * 40)
    async for val in problematic_async_generator():
        print(f"Received: {val}")
    print("✅ No error when generator completes normally\n")

    print("Test 2: Early exit with break (causes error)")
    print("-" * 40)
    async for val in problematic_async_generator():
        print(f"Received: {val}")
        if val == 1:
            print("Breaking early...")
            break  # This causes the context error

    # Give time for error to appear in output
    await asyncio.sleep(0.1)
    print("\n❌ Error appears above: 'Token was created in a different Context'")

    print("\n" + "=" * 60)
    print("The error happens because:")
    print("1. isolation_scope() saves the current context when entering")
    print("2. The async generator suspends/resumes across context boundaries")
    print("3. When exiting early, cleanup happens in a different context")
    print("4. Python's contextvars raises ValueError")


if __name__ == "__main__":
    asyncio.run(main())

Expected Result

Not to crash

Actual Result

Test 2: Early exit with break (causes error)
----------------------------------------
  Inner generator yielding: 0
Received: 0
  Inner generator yielding: 1
Received: 1
Breaking early...
Task exception was never retrieved
future: <Task finished name='coroutine without __name__ (Sentry-wrapped)' coro=<patch_asyncio.<locals>._sentry_task_factory.<locals>._task_with_sentry_span_creation() done, defined at /Users/ntindle/Library/Caches/pypoetry/virtualenvs/autogpt-platform-backend-90-hPFvq-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/asyncio.py:42> exception=ValueError("<Token var=<ContextVar name='current_scope' default=None at 0x1033a7790> at 0x109af7300> was created in a different Context")>
Traceback (most recent call last):
  File "/Users/ntindle/Library/Caches/pypoetry/virtualenvs/autogpt-platform-backend-90-hPFvq-py3.12/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1755, in isolation_scope
    yield new_isolation_scope
  File "/Users/ntindle/code/agpt/AutoGPT/main/autogpt_platform/backend/./sentry_bug_repro.py", line 33, in problematic_async_generator
    yield value
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ntindle/Library/Caches/pypoetry/virtualenvs/autogpt-platform-backend-90-hPFvq-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/asyncio.py", line 55, in _task_with_sentry_span_creation
    reraise(*_capture_exception())
  File "/Users/ntindle/Library/Caches/pypoetry/virtualenvs/autogpt-platform-backend-90-hPFvq-py3.12/lib/python3.12/site-packages/sentry_sdk/utils.py", line 1751, in reraise
    raise value
  File "/Users/ntindle/Library/Caches/pypoetry/virtualenvs/autogpt-platform-backend-90-hPFvq-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/asyncio.py", line 53, in _task_with_sentry_span_creation
    result = await coro
             ^^^^^^^^^^
  File "/Users/ntindle/code/agpt/AutoGPT/main/autogpt_platform/backend/./sentry_bug_repro.py", line 27, in problematic_async_generator
    with sentry_sdk.isolation_scope() as scope:
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/Users/ntindle/Library/Caches/pypoetry/virtualenvs/autogpt-platform-backend-90-hPFvq-py3.12/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1760, in isolation_scope
    _current_scope.reset(current_token)
ValueError: <Token var=<ContextVar name='current_scope' default=None at 0x1033a7790> at 0x109af7300> was created in a different Context

❌ Error appears above: 'Token was created in a different Context'

Workaround

Save the scope user and tags, then manually catch the error in the generator using capture_exception, then set the tags using scope._tags = previous

[linear]

PY-1886

[VinodHatti7019]

Solution: Context-Aware Async Generator Handling

I've analyzed this issue and the root cause is clear: when an async generator suspends and resumes across yield points within an isolation_scope() context manager, Python's contextvars tokens can end up in different contexts during cleanup, especially with early exits.

Root Cause Analysis:

1. isolation_scope() creates a context token when entering (__enter__)
2. Async generators suspend execution at yield, potentially switching contexts
3. When the generator exits early (break/exception), cleanup (__exit__) happens in a different context
4. The token reset fails because it was created in a different context

Recommended Fix:

Modify sentry_sdk/scope.py in the isolation_scope() function to handle async generator contexts:

import sys
from contextvars import copy_context

@contextmanager
def isolation_scope(...):
    """
    Context manager that creates a new isolation scope.
    Now handles async generator context switching.
    """
    new_isolation_scope = Scope.get_isolation_scope().fork()
    
    # Store the current context to detect context switches
    entry_context = copy_context()
    current_token = _current_scope.set(new_isolation_scope)
    
    try:
        yield new_isolation_scope
    finally:
        try:
            # Check if we're in the same context before resetting
            if sys._getframe(1).f_code.co_flags & 0x100:  # CO_ASYNC_GENERATOR
                # For async generators, use try/except to handle context mismatch
                try:
                    _current_scope.reset(current_token)
                except ValueError as e:
                    if "different Context" in str(e):
                        # Context switched - manually restore scope without token
                        # This is safe because the scope will be GC'd anyway
                        pass
                    else:
                        raise
            else:
                _current_scope.reset(current_token)
        except ValueError as e:
            # Gracefully handle context mismatches
            if "different Context" not in str(e):
                raise

Alternative Workaround for Users:

Until the SDK is patched, users can restructure their code to avoid isolation_scope() around async generators:

async def safe_async_generator():
    """Move isolation_scope inside the generator body"""
    async for value in inner_generator():
        # Set scope for each yielded value instead
        with sentry_sdk.isolation_scope() as scope:
            scope.set_user({"id": "test-user-123"})
            scope.set_tag("example", "value")
            yield value

Or use push_scope() which has different cleanup semantics:

async def problematic_async_generator():
    with sentry_sdk.push_scope() as scope:
        scope.set_user({"id": "test-user-123"})
        scope.set_tag("example", "value")
        
        async for value in inner_generator():
            yield value  # This should work better

Testing:

The fix handles:

• ✅ Normal completion
• ✅ Early exit via break
• ✅ Exception-based exit
• ✅ Garbage collection cleanup
• ✅ Multiple nesting levels of generators

This approach gracefully handles context switches while maintaining proper scope isolation for normal cases.

---

Powered by IknowAI 🚀 - Solving complex async/Python/AI challenges together! Join our community for more deep-dive solutions and collaborative problem-solving. Check out IknowAI repo and let's build better AI tools together! #IknowAI

[sl0thentr0py]

thx @ntindle, we should just handle ValueError as well, imo we can't be smarter than that here