Type annotations in `exec`d code do not resolve as expected

[vfazio]

Bug report

Bug description:

We've counted on the ability to exec code to test that various type annotations lead to the correct success/error scenarios

@pytest.mark.parametrize(
    "annotation, value, raises, match",
    [
        pytest.param("Annotated[int, at.Gt(1)]", "0", True, "0 is not greater than 1", id="Gt fail"),
        pytest.param("Annotated[int, at.Gt(1)]", "2", False, "", id="Gt ok"),
        pytest.param("Annotated[int, at.Lt(1)]", "2", True, "2 is not less than 1", id="Lt fail"),
        pytest.param("Annotated[int, at.Lt(1)]", "0", False, "", id="Lt ok"),
        pytest.param("Annotated[int, at.Ge(1)]", "0", True, "0 is not greater than or equal to 1", id="Ge fail"),
        pytest.param("Annotated[int, at.Ge(1)]", "1", False, "", id="Ge ok"),
        pytest.param("Annotated[int, at.Le(1)]", "2", True, "2 is not less than or equal to 1", id="Le fail"),
        pytest.param("Annotated[int, at.Le(1)]", "1", False, "", id="Le ok"),
        pytest.param("Annotated[int, at.MultipleOf(2)]", "3", True, "3 is not a multiple of 2", id="MultipleOf fail"),
        pytest.param("Annotated[int, at.MultipleOf(2)]", "4", False, "", id="MultipleOf ok"),
        pytest.param("Annotated[str, at.MinLen(2)]", "'a'", True, "a is not longer than 2", id="MinLen fail"),
        pytest.param("Annotated[str, at.MinLen(2)]", "'abc'", False, "", id="MinLen ok"),
        pytest.param("Annotated[str, at.MaxLen(2)]", "'abc'", True, "abc is not shorter than 2", id="MaxLen fail"),
        pytest.param("Annotated[str, at.MaxLen(2)]", "'a'", False, "", id="MaxLen ok"),
        pytest.param(
            "Annotated[int, at.Predicate(lambda x: False)]", "0", True, "0 does not satisfy", id="Predicate fail"
        ),
        pytest.param("Annotated[int, at.Predicate(lambda x: True)]", "0", False, "", id="Predicate ok"),
    ],
)
async def test_test_with_annotated_type(annotation: str, value: str, raises: bool, match: str):
    loc: dict = {}
    exec(
        dedent(
            f"""
            from typing import Annotated
            import annotated_types as at

            @Test("test")
            async def test(test: TestInstance, foo: {annotation}) -> None:
                pass

            test_instance = test.use_options(foo={value})
        """
        ),
        globals(),
        loc,
    )
    test = loc["test_instance"]

    ctx: AbstractContextManager
    if raises:
        ctx = pytest.raises(ValueError, match=match)
    else:
        ctx = nullcontext()

    with ctx:
        await parameterize(test, Environment(TestConfig("test")))

However, this now fails in python 3.14+

A "minimal" reproduction:

import inspect

from textwrap import dedent
from typing import TypeAlias, Callable, Coroutine, Self, get_type_hints

TestFunc: TypeAlias = Callable[..., Coroutine[None, None, None]]

class Test:
    def __call__(self, func: TestFunc) -> Self:
        print(inspect.signature(func).parameters)
        print(get_type_hints(func))

def main():
    exec(
        dedent(
            f"""
            from typing import Annotated
            import annotated_types as at

            @Test()
            async def test(foo: Annotated[int, at.Gt(1)]) -> None:
                pass

            print("After function call")
        """
        ),
        globals(),
        {},
    )


if __name__ == "__main__":
    main()

Despite the types Annotated and at being imported in the exec'd code, the values do not get resolved when trying to get annotations.

Traceback (most recent call last):
  File "/home/vfazio/development/xtf/314annotations.py", line 33, in <module>
    main()
    ~~~~^^
  File "/home/vfazio/development/xtf/314annotations.py", line 14, in main
    exec(
    ~~~~^
        dedent(
        ^^^^^^^
    ...<12 lines>...
        {},
        ^^^
    )
    ^
  File "<string>", line 5, in <module>
  File "/home/vfazio/development/xtf/314annotations.py", line 10, in __call__
    print(inspect.signature(func).parameters)
          ~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/vfazio/.pyenv/versions/3.14.3/lib/python3.14/inspect.py", line 3322, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped,
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                   globals=globals, locals=locals, eval_str=eval_str,
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                   annotation_format=annotation_format)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vfazio/.pyenv/versions/3.14.3/lib/python3.14/inspect.py", line 3037, in from_callable
    return _signature_from_callable(obj, sigcls=cls,
                                    follow_wrapper_chains=follow_wrapped,
                                    globals=globals, locals=locals, eval_str=eval_str,
                                    annotation_format=annotation_format)
  File "/home/vfazio/.pyenv/versions/3.14.3/lib/python3.14/inspect.py", line 2512, in _signature_from_callable
    return _signature_from_function(sigcls, obj,
                                    skip_bound_arg=skip_bound_arg,
                                    globals=globals, locals=locals, eval_str=eval_str,
                                    annotation_format=annotation_format)
  File "/home/vfazio/.pyenv/versions/3.14.3/lib/python3.14/inspect.py", line 2335, in _signature_from_function
    annotations = get_annotations(func, globals=globals, locals=locals, eval_str=eval_str,
                                  format=annotation_format)
  File "/home/vfazio/.pyenv/versions/3.14.3/lib/python3.14/annotationlib.py", line 966, in get_annotations
    ann = _get_dunder_annotations(obj)
  File "/home/vfazio/.pyenv/versions/3.14.3/lib/python3.14/annotationlib.py", line 1146, in _get_dunder_annotations
    ann = getattr(obj, "__annotations__", None)
  File "<string>", line 6, in __annotate__
NameError: name 'Annotated' is not defined

This seems to imply that code generators/exec can no longer rely on imports in code to satisfy annotations? I don't know if this is due to changes in exec or PEP 649/749 but it is inconvenient.

CPython versions tested on:

3.14, 3.13, 3.12, 3.11

Operating systems tested on:

Linux

[picnixz]

cc @JelleZijlstra

[inventshah]

A smaller repro is

exec("""
from typing import Literal

def test(x: Literal[1]): ...
test.__annotations__
""", globals(), {})

I believe this has to do with exec interactions with import and global/local scopes. Looking at bytecode of the generated __annotate__ function, the problem is Literal is loaded with LOAD_GLOBAL, but inside exec the import goes to locals if one is passed in. And as the docs for exec mention

Remember that at the module level, globals and locals are the same dictionary.

For OP, if you remove the explicit locals dictionary (or make it the same as globals to mimic module level behavior), the tests should pass (of course you may want to make a copy to prevent pollution).

I'm not sure if there is an elegant solution to restore pre-PEP 649 behavior in this case as other examples like this also error for similar reasons.

exec("""
from io import BytesIO
def test(): BytesIO()
test()
""", globals(), {})

[vfazio]

For OP, if you remove the explicit locals dictionary (or make it the same as globals to mimic module level behavior), the tests should pass (of course you may want to make a copy to prevent pollution).

@inventshah Thanks for chiming in!

I tested your suggestion:

@pytest.mark.parametrize(
    "annotation, value, raises, match",
    [
        pytest.param("Annotated[int, at.Gt(1)]", "0", True, "0 is not greater than 1", id="Gt fail"),
        pytest.param("Annotated[int, at.Gt(1)]", "2", False, "", id="Gt ok"),
        pytest.param("Annotated[int, at.Lt(1)]", "2", True, "2 is not less than 1", id="Lt fail"),
        pytest.param("Annotated[int, at.Lt(1)]", "0", False, "", id="Lt ok"),
        pytest.param("Annotated[int, at.Ge(1)]", "0", True, "0 is not greater than or equal to 1", id="Ge fail"),
        pytest.param("Annotated[int, at.Ge(1)]", "1", False, "", id="Ge ok"),
        pytest.param("Annotated[int, at.Le(1)]", "2", True, "2 is not less than or equal to 1", id="Le fail"),
        pytest.param("Annotated[int, at.Le(1)]", "1", False, "", id="Le ok"),
        pytest.param("Annotated[int, at.MultipleOf(2)]", "3", True, "3 is not a multiple of 2", id="MultipleOf fail"),
        pytest.param("Annotated[int, at.MultipleOf(2)]", "4", False, "", id="MultipleOf ok"),
        pytest.param("Annotated[str, at.MinLen(2)]", "'a'", True, "a is not longer than 2", id="MinLen fail"),
        pytest.param("Annotated[str, at.MinLen(2)]", "'abc'", False, "", id="MinLen ok"),
        pytest.param("Annotated[str, at.MaxLen(2)]", "'abc'", True, "abc is not shorter than 2", id="MaxLen fail"),
        pytest.param("Annotated[str, at.MaxLen(2)]", "'a'", False, "", id="MaxLen ok"),
        pytest.param(
            "Annotated[int, at.Predicate(lambda x: False)]", "0", True, "0 does not satisfy", id="Predicate fail"
        ),
        pytest.param("Annotated[int, at.Predicate(lambda x: True)]", "0", False, "", id="Predicate ok"),
    ],
)
async def test_test_with_annotated_type(annotation: str, value: str, raises: bool, match: str):
    loc: dict = globals()
    exec(
        dedent(
            f"""
            from typing import Annotated
            import annotated_types as at

            @Test("test")
            async def test(test: TestInstance, foo: {annotation}) -> None:
                pass

            test_instance = test.use_options(foo={value})
        """
        ),
        globals(),
        loc,
    )
    test = loc["test_instance"]

    ctx: AbstractContextManager
    if raises:
        ctx = pytest.raises(ValueError, match=match)
    else:
        ctx = nullcontext()

    with ctx:
        await parameterize(test, Environment(TestConfig("test")))

And can see that this now works, but may lead to the polluted globals you mentioned as the following get added in my case:

{'at', 'Annotated', 'test_instance', 'test'}

We need to pass the locals so that we can extract a value, so I guess my only option is to clone globals.

Note that using a copy of globals for just the locals did not work, I had to use the same copy for both globals and locals (so id(globals()) == id(locals())...

Does not work:

 async def test_test_with_annotated_type(annotation: str, value: str, raises: bool, match: str):
-    loc: dict = {}
+    loc: dict = globals().copy()
     exec(

Does work:

 async def test_test_with_annotated_type(annotation: str, value: str, raises: bool, match: str):
-    loc: dict = {}
+    loc: dict = globals().copy()
     exec(
         dedent(
             f"""
@@ -964,7 +964,7 @@ async def test_test_with_annotated_type(annotation: str, value: str, raises: boo
             test_instance = test.use_options(foo={value})
         """
         ),
-        globals(),
+        loc,
         loc,
     )

Maybe this is inline with the documentation you mentioned:

Remember that at the module level, globals and locals are the same dictionary.

and maybe this:

Note: When exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition. This means functions and classes defined in the executed code will not be able to access variables assigned at the top level (as the “top level” variables are treated as class variables in a class definition).

So I guess maybe I can't use different objects for this case even though it did work in previous versions.

I tried these changes in python 3.12 and things seem to work, so maybe I can make these changes now in preparation for migration, but I'd like to better understand what's going on and why it's working the way it is. Is there some sort of primer for the layman beyond the documentation for exec? Is there a commit and rationale for the change in behavior?

When possible, i don't like relying on blackbox/magic behavior so we can avoid rug pulls. This case may be a misunderstanding of how things should have been done, like maybe we were abusing errant behavior; I'd just like to understand it better so i can inform my team since this is something we need to be able to maintain

[inventshah]

Reading the execution model docs, I believe there is likely a bug. When globals is not locals in exec, the statements are executed in a class definition, which I think would look something like this:

class _TempExec:
    from typing import Literal
    def test(x: Literal[1]): ...

Then in 3.14+, if we manually do what the compiler might generate:

class _TempExec:
    from typing import Literal
    def test(x): ...
    test.__annotate__ = lambda s: {"x": Literal[1]}

In that way, it's clear why the annotation fails. As the name binding docs say,

The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods.

However, this transformation is not exact, and as the next sentence says

This includes comprehensions and generator expressions, but it does not include annotation scopes, which have access to their enclosing class scopes.

Which suggests that the compiler generated __annotate__ function should get access to the imports.

I'll wait for @JelleZijlstra to chime in with more context before continuing. I suspect there is an issue somewhere in the annotation scope codegen.

For @vfazio, I think if your goal is to mimic what happens at module level, passing the same dict for globals and locals seems like the most correct path forward, regardless of the outcome here.