gh-135700: Fix instructions in annotate have incorrect code positions #135814

AndPuQing · 2025-06-22T10:45:31Z

Issue: some instructions in __annotate__ have incorrect code positions #135700

from dis import Bytecode, dis

src = """\
22
333
__dataclass_fields__: ClassVar
"""

code = compile(src, "<string>", "exec")
print(code.co_consts)
print(dis(src))
annotate_code = code.co_consts[1]

for bc in Bytecode(annotate_code):
    print(bc.positions, bc.opname)

output

(0, <code object __annotate__ at 0x7b19f4095f20, file "<string>", line 1>, None)
  --           MAKE_CELL                0 (__conditional_annotations__)

   0           RESUME                   0

   1           LOAD_CONST               1 (<code object __annotate__ at 0x7b19f4096c30, file "<dis>", line 1>)
               MAKE_FUNCTION
               STORE_NAME               1 (__annotate__)
               BUILD_SET                0
               STORE_NAME               0 (__conditional_annotations__)

   2           NOP

   3           LOAD_NAME                0 (__conditional_annotations__)
               LOAD_SMALL_INT           0
               SET_ADD                  1
               POP_TOP
               LOAD_CONST               2 (None)
               RETURN_VALUE

Disassembly of <code object __annotate__ at 0x7b19f4096c30, file "<dis>", line 1>:
  1           RESUME                   0
              LOAD_FAST_BORROW         0 (format)
              LOAD_SMALL_INT           2
              COMPARE_OP             132 (>)
              POP_JUMP_IF_FALSE        3 (to L1)
              NOT_TAKEN
              LOAD_COMMON_CONSTANT     1 (NotImplementedError)
              RAISE_VARARGS            1
      L1:     BUILD_MAP                0

  3           LOAD_SMALL_INT           0
              LOAD_GLOBAL              0 (__conditional_annotations__)
              CONTAINS_OP              0 (in)
              POP_JUMP_IF_FALSE       10 (to L2)
              NOT_TAKEN
              LOAD_GLOBAL              2 (ClassVar)
              COPY                     2
              LOAD_CONST               1 ('__dataclass_fields__')

  1           STORE_SUBSCR
      L2:     RETURN_VALUE
None
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RESUME
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_FAST_BORROW
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_SMALL_INT
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) COMPARE_OP
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) POP_JUMP_IF_FALSE
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) NOT_TAKEN
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_COMMON_CONSTANT
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RAISE_VARARGS
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) BUILD_MAP
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) LOAD_SMALL_INT
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) LOAD_GLOBAL
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) CONTAINS_OP
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) POP_JUMP_IF_FALSE
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) NOT_TAKEN
Positions(lineno=3, end_lineno=3, col_offset=22, end_col_offset=30) LOAD_GLOBAL
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) COPY
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) LOAD_CONST
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) STORE_SUBSCR
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) RETURN_VALUE

Misc/NEWS.d/next/Core_and_Builtins/2025-06-22-10-47-27.gh-issue-135700.0qdtCl.rst

…e-135700.0qdtCl.rst Co-authored-by: sobolevn <mail@sobolevn.me>

JelleZijlstra · 2025-06-22T13:38:57Z

Good catch, thanks! Would you be able to add a unit test? Also test a class with annotations so we can see how it works in a class-level __annotate__ function.

Lib/test/test_dis.py

iritkatriel · 2025-06-23T19:04:50Z

Lib/test/test_dis.py

+
+        for case_name, annotate_code in test_cases:
+            with self.subTest(case=case_name):
+                instructions = list(dis.Bytecode(annotate_code))


You can use dis.findlinestarts(annotate_code) instead.

Lib/test/test_dis.py

Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

iritkatriel · 2025-06-24T11:42:46Z

Lib/test/test_dis.py

+            ),
+        ]
+
+        for case_name, annotate_code in test_cases:


assert here that annotate_code is a code object and that annotate_code.co_name is '__annotate__'.

iritkatriel · 2025-06-24T11:44:46Z

Lib/test/test_dis.py

+                    for instr in dis.get_instructions(annotate_code)
+                    if setup_scope_begin <= instr.offset < setup_scope_end
+                    and instr.positions
+                }


Why does this need to be so complicated? Don't we just need to check that all the instructions in this code object have the same line number (excluding Nones)?

Because the important thing is to make sure the instructions in setup_annotations_scope are consistent with the end_col_offset of RESUME, not just the lineno

no fix:

Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RESUME Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) LOAD_FAST_BORROW # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) LOAD_SMALL_INT # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) COMPARE_OP # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) POP_JUMP_IF_FALSE # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) NOT_TAKEN # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) LOAD_COMMON_CONSTANT # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) RAISE_VARARGS # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) BUILD_MAP # incorrect

fix:

Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RESUME Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_FAST_BORROW Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_SMALL_INT Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) COMPARE_OP Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) POP_JUMP_IF_FALSE Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) NOT_TAKEN Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_COMMON_CONSTANT Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RAISE_VARARGS Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) BUILD_MAP

And not all annotate lines are the same, because some opcodes need to indicate the "code" where the annotation occurs.

Ok, so we need to check they all have the same end_col_offset as well.

And not all annotate lines are the same, because some opcodes need to indicate the "code" where the annotation occurs.

The test doesn't need to work for all inputs. Only the two inputs it runs on.

And not all annotate lines are the same, because some opcodes need to indicate the "code" where the annotation occurs.

The test doesn't need to work for all inputs. Only the two inputs it runs on.

Just to be clear, instructions within code object can correspond to different line numbers. So I made an offset restriction here, limiting the check to between RESUME and BUILD_MAP (setup_annotations_scope)

Maybe @JelleZijlstra has an idea how to write a meaningful test for this.

I don't have much of an intuition on how line numbers should work for synthetic code.

@15r10nk are you able to suggest a meaningful unit test for this? Alternatively, can you confirm that the PR resolved the issue?

I can verify that this fixes the issue which I have reported 👍 . I also found out that the reported bug affected debugging with pdb.

example.py

import example_mod print(example_mod.__annotations__)

example_mod.py

22 class ClassVar(): pass __dataclass_fields__: ClassVar

./python -m pdb example.py > /home/orbit/projects/cpython/example.py(1)<module>() -> import example_mod (Pdb) b example_mod.py:2 Breakpoint 1 at /home/orbit/projects/cpython/example_mod.py:2 (Pdb) c > /home/orbit/projects/cpython/example_mod.py(2)<module>() # first hit which is expected -> 22 (Pdb) c > /home/orbit/projects/cpython/example_mod.py(2)__annotate__() # pdb should not break here -> 22 (Pdb) l 1 2 B-> 22 3 4 class ClassVar(): 5 pass 6 7 __dataclass_fields__: ClassVar 8 [EOF] (Pdb) bt /home/orbit/projects/cpython/Lib/bdb.py(899)run() -> exec(cmd, globals, locals) <string>(1)<module>() /home/orbit/projects/cpython/example.py(4)<module>() -> print(example_mod.__annotations__) > /home/orbit/projects/cpython/example_mod.py(2)__annotate__() -> 22 (Pdb)

The breakpoint is hit a second time when the synthetic code is executed which uses the incorrect source positions.

I remember that I found multiple different problems with similar synthetic codes like this in the past (super(),except related and maybe more) but I was always able to work around this issues and I did not know that they can also lead to problems when you debug with pdb. I will write a tool to find these problems in the next days.

Great, thank you!

The pdb example can probably be used for a unit test (in test_pdb). I'd remove the test that is currently in this PR.

AndPuQing requested review from markshannon and iritkatriel as code owners June 22, 2025 10:45

bedevere-app bot added the awaiting review label Jun 22, 2025

bedevere-app bot mentioned this pull request Jun 22, 2025

some instructions in __annotate__ have incorrect code positions #135700

Open

AndPuQing and others added 2 commits June 22, 2025 18:48

fix instructions in __annotate__ have incorrect code positions

983ae79

📜🤖 Added by blurb_it.

dd3bea2

sobolevn reviewed Jun 22, 2025

View reviewed changes

Misc/NEWS.d/next/Core_and_Builtins/2025-06-22-10-47-27.gh-issue-135700.0qdtCl.rst Outdated Show resolved Hide resolved

Update Misc/NEWS.d/next/Core_and_Builtins/2025-06-22-10-47-27.gh-issu…

c2f0a4d

…e-135700.0qdtCl.rst Co-authored-by: sobolevn <mail@sobolevn.me>

JelleZijlstra added the needs backport to 3.14 bugs and security fixes label Jun 22, 2025

add unit tests

b79060e

sobolevn reviewed Jun 22, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

fix test

2201864

iritkatriel reviewed Jun 23, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

AndPuQing force-pushed the fix-issue-135700 branch from 9045ebe to aed06b7 Compare June 23, 2025 10:56

fix test

aed06b7

iritkatriel reviewed Jun 23, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

iritkatriel reviewed Jun 23, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

iritkatriel reviewed Jun 23, 2025

View reviewed changes

AndPuQing and others added 3 commits June 24, 2025 14:34

Update Lib/test/test_dis.py

d85247e

Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

Apply suggestions from code review

eca9248

Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

fix test

e14cb29

iritkatriel reviewed Jun 24, 2025

View reviewed changes

Uh oh!

gh-135700: Fix instructions in __annotate__ have incorrect code positions #135814

Are you sure you want to change the base?

gh-135700: Fix instructions in __annotate__ have incorrect code positions #135814

Conversation

AndPuQing commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented Jun 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndPuQing Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gh-135700: Fix instructions in annotate have incorrect code positions #135814

gh-135700: Fix instructions in annotate have incorrect code positions #135814

AndPuQing commented Jun 22, 2025 •

edited

Loading

AndPuQing Jun 24, 2025 •

edited

Loading