gh-135700: Fix instructions in annotate have incorrect code positions #135814

AndPuQing · 2025-06-22T10:45:31Z

Issue: some instructions in __annotate__ have incorrect code positions #135700

from dis import Bytecode, dis

src = """\
22
333
__dataclass_fields__: ClassVar
"""

code = compile(src, "<string>", "exec")
print(code.co_consts)
print(dis(src))
annotate_code = code.co_consts[1]

for bc in Bytecode(annotate_code):
    print(bc.positions, bc.opname)

output

(0, <code object __annotate__ at 0x7b19f4095f20, file "<string>", line 1>, None)
  --           MAKE_CELL                0 (__conditional_annotations__)

   0           RESUME                   0

   1           LOAD_CONST               1 (<code object __annotate__ at 0x7b19f4096c30, file "<dis>", line 1>)
               MAKE_FUNCTION
               STORE_NAME               1 (__annotate__)
               BUILD_SET                0
               STORE_NAME               0 (__conditional_annotations__)

   2           NOP

   3           LOAD_NAME                0 (__conditional_annotations__)
               LOAD_SMALL_INT           0
               SET_ADD                  1
               POP_TOP
               LOAD_CONST               2 (None)
               RETURN_VALUE

Disassembly of <code object __annotate__ at 0x7b19f4096c30, file "<dis>", line 1>:
  1           RESUME                   0
              LOAD_FAST_BORROW         0 (format)
              LOAD_SMALL_INT           2
              COMPARE_OP             132 (>)
              POP_JUMP_IF_FALSE        3 (to L1)
              NOT_TAKEN
              LOAD_COMMON_CONSTANT     1 (NotImplementedError)
              RAISE_VARARGS            1
      L1:     BUILD_MAP                0

  3           LOAD_SMALL_INT           0
              LOAD_GLOBAL              0 (__conditional_annotations__)
              CONTAINS_OP              0 (in)
              POP_JUMP_IF_FALSE       10 (to L2)
              NOT_TAKEN
              LOAD_GLOBAL              2 (ClassVar)
              COPY                     2
              LOAD_CONST               1 ('__dataclass_fields__')

  1           STORE_SUBSCR
      L2:     RETURN_VALUE
None
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RESUME
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_FAST_BORROW
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_SMALL_INT
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) COMPARE_OP
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) POP_JUMP_IF_FALSE
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) NOT_TAKEN
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_COMMON_CONSTANT
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RAISE_VARARGS
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) BUILD_MAP
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) LOAD_SMALL_INT
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) LOAD_GLOBAL
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) CONTAINS_OP
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) POP_JUMP_IF_FALSE
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) NOT_TAKEN
Positions(lineno=3, end_lineno=3, col_offset=22, end_col_offset=30) LOAD_GLOBAL
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) COPY
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) LOAD_CONST
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) STORE_SUBSCR
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) RETURN_VALUE

Misc/NEWS.d/next/Core_and_Builtins/2025-06-22-10-47-27.gh-issue-135700.0qdtCl.rst

…e-135700.0qdtCl.rst Co-authored-by: sobolevn <mail@sobolevn.me>

JelleZijlstra · 2025-06-22T13:38:57Z

Good catch, thanks! Would you be able to add a unit test? Also test a class with annotations so we can see how it works in a class-level __annotate__ function.

Lib/test/test_dis.py

iritkatriel · 2025-06-23T19:04:50Z

Lib/test/test_dis.py

+
+        for case_name, annotate_code in test_cases:
+            with self.subTest(case=case_name):
+                instructions = list(dis.Bytecode(annotate_code))


You can use dis.findlinestarts(annotate_code) instead.

Lib/test/test_dis.py

Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

iritkatriel · 2025-06-24T11:42:46Z

Lib/test/test_dis.py

+            ),
+        ]
+
+        for case_name, annotate_code in test_cases:


assert here that annotate_code is a code object and that annotate_code.co_name is '__annotate__'.

iritkatriel · 2025-06-24T11:44:46Z

Lib/test/test_dis.py

+                    for instr in dis.get_instructions(annotate_code)
+                    if setup_scope_begin <= instr.offset < setup_scope_end
+                    and instr.positions
+                }


Why does this need to be so complicated? Don't we just need to check that all the instructions in this code object have the same line number (excluding Nones)?

Because the important thing is to make sure the instructions in setup_annotations_scope are consistent with the end_col_offset of RESUME, not just the lineno

no fix:

Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RESUME Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) LOAD_FAST_BORROW # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) LOAD_SMALL_INT # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) COMPARE_OP # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) POP_JUMP_IF_FALSE # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) NOT_TAKEN # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) LOAD_COMMON_CONSTANT # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) RAISE_VARARGS # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) BUILD_MAP # incorrect

fix:

Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RESUME Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_FAST_BORROW Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_SMALL_INT Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) COMPARE_OP Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) POP_JUMP_IF_FALSE Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) NOT_TAKEN Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_COMMON_CONSTANT Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RAISE_VARARGS Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) BUILD_MAP

And not all annotate lines are the same, because some opcodes need to indicate the "code" where the annotation occurs.

Ok, so we need to check they all have the same end_col_offset as well.

And not all annotate lines are the same, because some opcodes need to indicate the "code" where the annotation occurs.

The test doesn't need to work for all inputs. Only the two inputs it runs on.

And not all annotate lines are the same, because some opcodes need to indicate the "code" where the annotation occurs.

The test doesn't need to work for all inputs. Only the two inputs it runs on.

Just to be clear, instructions within code object can correspond to different line numbers. So I made an offset restriction here, limiting the check to between RESUME and BUILD_MAP (setup_annotations_scope)

I don't have much of an intuition on how line numbers should work for synthetic code.

@15r10nk are you able to suggest a meaningful unit test for this? Alternatively, can you confirm that the PR resolved the issue?

I can verify that this fixes the issue which I have reported 👍 . I also found out that the reported bug affected debugging with pdb.

example.py

import example_mod print(example_mod.__annotations__)

example_mod.py

22 class ClassVar(): pass __dataclass_fields__: ClassVar

./python -m pdb example.py > /home/orbit/projects/cpython/example.py(1)<module>() -> import example_mod (Pdb) b example_mod.py:2 Breakpoint 1 at /home/orbit/projects/cpython/example_mod.py:2 (Pdb) c > /home/orbit/projects/cpython/example_mod.py(2)<module>() # first hit which is expected -> 22 (Pdb) c > /home/orbit/projects/cpython/example_mod.py(2)__annotate__() # pdb should not break here -> 22 (Pdb) l 1 2 B-> 22 3 4 class ClassVar(): 5 pass 6 7 __dataclass_fields__: ClassVar 8 [EOF] (Pdb) bt /home/orbit/projects/cpython/Lib/bdb.py(899)run() -> exec(cmd, globals, locals) <string>(1)<module>() /home/orbit/projects/cpython/example.py(4)<module>() -> print(example_mod.__annotations__) > /home/orbit/projects/cpython/example_mod.py(2)__annotate__() -> 22 (Pdb)

The breakpoint is hit a second time when the synthetic code is executed which uses the incorrect source positions.

I remember that I found multiple different problems with similar synthetic codes like this in the past (super(),except related and maybe more) but I was always able to work around this issues and I did not know that they can also lead to problems when you debug with pdb. I will write a tool to find these problems in the next days.

Great, thank you!

The pdb example can probably be used for a unit test (in test_pdb). I'd remove the test that is currently in this PR.

hi @iritkatriel @AndPuQing , I wrote a test with test_pdb.py maybe that can be used for this pr.

class PdbTestCase(unittest.TestCase): ... def test_issue135700(self): """ https://github.com/python/cpython/issues/135700 """ module_code = """\ 22 class ClassVar: pass __dataclass_fields__: ClassVar """ with open('testmod.py', 'w') as f: f.write(module_code) self.addCleanup(os_helper.unlink, "testmod.py") script = """ import testmod print(testmod.__annotations__) """ commands = """ b testmod.py:1 c c """ result = self.run_pdb_script(script, commands) self.assertNotIn("(1)__annotate__()", result[0])

Closes python#53243

python#134665)

… asyncio REPL (python#136448)

…H-136270)

…btrfs subvolumes (pythonGH-136058)

…136477)

…ry (pythonGH-136146) --------- Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Éric <merwok@netwok.org>

…ython#136430) Adds a test skip on platforms where hard links are not available (which includes Android).

…ython#136431)

…ote_debugging (python#136483)

…136253) Adds zstd to the Android build process. --------- Co-authored-by: Malcolm Smith <smith@chaquo.com>

… levels (python#136474)

…ythonGH-136210) Co-authored-by: Victor Stinner <vstinner@python.org>

…nts (python#135585) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org>

…ssion` (python#136441)

Closes python#82663

…iter` functions (python#136498) Closes python#52876

…#135998) Implement a statistical sampling profiler that can profile external Python processes by PID. Uses the _remote_debugging module and converts the results to pstats-compatible format for analysis. Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

…ode was set (python#136518)

…le (python#136519) Omit perf_jit_trampoline from "JIT" codeowners

…honGH-132273) * Fix flag mask inversion when unnamed flags exist. For example: class Flag(enum.Flag): A = 0x01 B = 0x02 MASK = 0xff ~Flag.MASK is Flag(0) * EJECT and KEEP flags (IntEnum is KEEP) use direct value. * correct Flag inversion to only flip flag bits IntFlag will flip all bits -- this only makes a difference in flag sets with missing values. * correct negative assigned values in flags negative values are no longer used as-is, but become inverted; i.e. class Y(self.enum_type): A = auto() B = auto() C = ~A # aka ~1 aka 0b1 110 (from enum.bin()) aka 6 D = auto() assert Y.C. is Y.B|Y.D

…ython#136475)

…6531) Closes python#76637

AndPuQing requested review from markshannon and iritkatriel as code owners June 22, 2025 10:45

bedevere-app bot added the awaiting review label Jun 22, 2025

bedevere-app bot mentioned this pull request Jun 22, 2025

some instructions in __annotate__ have incorrect code positions #135700

Open

AndPuQing and others added 2 commits June 22, 2025 18:48

fix instructions in __annotate__ have incorrect code positions

983ae79

📜🤖 Added by blurb_it.

dd3bea2

sobolevn reviewed Jun 22, 2025

View reviewed changes

Misc/NEWS.d/next/Core_and_Builtins/2025-06-22-10-47-27.gh-issue-135700.0qdtCl.rst Outdated Show resolved Hide resolved

Update Misc/NEWS.d/next/Core_and_Builtins/2025-06-22-10-47-27.gh-issu…

c2f0a4d

…e-135700.0qdtCl.rst Co-authored-by: sobolevn <mail@sobolevn.me>

JelleZijlstra added the needs backport to 3.14 label Jun 22, 2025

add unit tests

b79060e

sobolevn reviewed Jun 22, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

fix test

2201864

iritkatriel reviewed Jun 23, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

AndPuQing force-pushed the fix-issue-135700 branch from 9045ebe to aed06b7 Compare June 23, 2025 10:56

fix test

aed06b7

iritkatriel reviewed Jun 23, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

iritkatriel reviewed Jun 23, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

iritkatriel reviewed Jun 23, 2025

View reviewed changes

AndPuQing and others added 3 commits June 24, 2025 14:34

Update Lib/test/test_dis.py

d85247e

Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

Apply suggestions from code review

eca9248

Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

fix test

e14cb29

iritkatriel reviewed Jun 24, 2025

View reviewed changes

AndPuQing requested review from pablogsal, ambv, 1st1, lysnikolaou, kumaraditya303, willingc and AA-Turner as code owners July 11, 2025 11:44

StanFromIreland and others added 29 commits July 11, 2025 19:48

pythongh-53243: Document codecs.readbuffer_encode() (python#136284)

fc448ff

Closes python#53243

pythongh-134657: Remove newly added private names from asyncio.__all__ (

43275ad

python#134665)

pythongh-136447: Use self.loop instead of global loop variable in…

795dd84

… asyncio REPL (python#136448)

Docs: unittest.enterModuleContext is not a classmethod (python#136464)

fb49280

pythongh-131825: Fix sqlite3 timezone-naive adapter recipe (pythonG…

978b416

…H-136270)

pythongh-81520: Document unexpected os.path.ismount behaviour with …

6be101d

…btrfs subvolumes (pythonGH-136058)

Minor edit: Move comments closer to the code they describe (pythongh-…

a317ea0

…136477)

pythonGH-115802: Optimize JIT stencils for size (pythonGH-136393)

21b1d77

Add Diego Russo as code owner of the JIT (python#136460)

4e75f78

pythongh-136145: Define 'standard library' and 'stdlib' in the glossa…

84ac4ca

…ry (pythonGH-136146) --------- Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Éric <merwok@netwok.org>

pythongh-136156: Skip test_tempfile.test_link_tmpfile() on Android (p…

18eb5ff

…ython#136430) Adds a test skip on platforms where hard links are not available (which includes Android).

pythongh-102740: Clarify time.monotonic() "system-wide" in the doc (p…

391d651

…ython#136431)

pythongh-136476: Show the full stack in get_async_stack_trace in _rem…

892634d

…ote_debugging (python#136483)

pythongh-135846: Add zstd dependency to Android build script (python#…

53fbe1b

…136253) Adds zstd to the Android build process. --------- Co-authored-by: Malcolm Smith <smith@chaquo.com>

pythongh-136476: Remove creation of unused list (pythonGH-136494)

fecd270

pythongh-136438: Make sure test_builtins pass with all optimization…

74291d2

… levels (python#136474)

pythongh-136209: Add .. c:var:: declarations for C exception types (p…

559b6a2

…ythonGH-136210) Co-authored-by: Victor Stinner <vstinner@python.org>

pythongh-82088: Improve performance of PyLong_As*() for multi-digit i…

77512cd

…nts (python#135585) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org>

pythongh-136394: Fix race condition in test_zstd (pythonGH-136432)

b0793d2

pythongh-132661: Add default value (of "") for `Interpolation.expre…

0f2138d

…ssion` (python#136441)

pythongh-82663: Clarify codecs.iterdecode/encode docs (python#136497)

d684224

Closes python#82663

pythongh-52876: Implement missing parameter in `codecs.StreamReaderWr…

27b12f0

…iter` functions (python#136498) Closes python#52876

pythongh-136517: Print uncollectable objects if DEBUG_UNCOLLECTABLE m…

4af89d2

…ode was set (python#136518)

Omit Python/perf_jit_trampoline.c from the **/*jit* CODEOWNERS ru…

40da9d3

…le (python#136519) Omit perf_jit_trampoline from "JIT" codeowners

pythongh-101100: Fix sphinx warnings in library/email.parser.rst (p…

52ff16e

…ython#136475)

pythongh-76637: Note that undefined Codec is for testing (python#13…

960a247

…6531) Closes python#76637

add test_pdb case

3dbba40

15r10nk mentioned this pull request Jul 14, 2025

gh-135700: Fix instructions in __annotate__ have incorrect code positions #136543

Open

Uh oh!

gh-135700: Fix instructions in __annotate__ have incorrect code positions #135814

gh-135700: Fix instructions in __annotate__ have incorrect code positions #135814

Conversation

AndPuQing commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented Jun 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndPuQing Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gh-135700: Fix instructions in annotate have incorrect code positions #135814

gh-135700: Fix instructions in annotate have incorrect code positions #135814

AndPuQing commented Jun 22, 2025 •

edited

Loading

AndPuQing Jun 24, 2025 •

edited

Loading