dis does not work with the new optimized ops

[martmists-gh]

BPO	45362
Nosy	@markshannon, @corona10, @Fidget-Spinner, @Martmists-GH

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-05.13:31:06.963>
created_at = <Date 2021-10-04.09:56:52.463>
labels = ['invalid', 'type-bug', 'library', '3.11']
title = 'dis does not work with the new optimized ops'
updated_at = <Date 2021-10-05.13:31:06.962>
user = 'https://github.com/martmists-gh'

bugs.python.org fields:

activity = <Date 2021-10-05.13:31:06.962>
actor = 'kj'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-05.13:31:06.963>
closer = 'kj'
components = ['Library (Lib)']
creation = <Date 2021-10-04.09:56:52.463>
creator = 'martmists'
dependencies = []
files = []
hgrepos = []
issue_num = 45362
keywords = []
message_count = 7.0
messages = ['403126', '403127', '403136', '403137', '403139', '403140', '403241']
nosy_count = 4.0
nosy_names = ['Mark.Shannon', 'corona10', 'kj', 'martmists']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45362'
versions = ['Python 3.11']

[martmists-gh]

Python 3.11.0a0 (heads/main:e6d1aa1, Oct  4 2021, 10:33:36) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> c = bytes([127, 1, 0, 2])  # LOAD_FAST__LOAD_FAST (1, 2)
>>> dis.dis(c)
          0 <127>                    1
          2 <0>
>>> c = bytes([48, 0, 161, 0])  # LOAD_GLOBAL_BUILTIN (0) CALL_FUNCTION (0)
>>> dis.dis(c)
          0 <48>
          2 CALL_METHOD              0

---

The reason for this seems to be the lack of definition in opcode.py aside from being mentioned in _specialized_instructions. Additionally, it seems dis._unpack_opargs is not yet prepared to support opcodes like LOAD_FAST__LOAD_FAST which take two arguments.

While I don't expect this to be a first priority in this big change, it's important to not forget it.

[martmists-gh]

Another thing I found to be quite odd is JUMP_ABSOLUTE_QUICK, which is set as 40, but as it's derived from JUMP_ABSOLUTE it still requires an argument. This makes HAVE_ARGUMENT useless as there's an op that doesn't follow the convention.

[Fidget-Spinner]

Thanks Martmists for the bug report, I'll try to address some of your concerns:

The reason for this seems to be the lack of definition in opcode.py aside from being mentioned in _specialized_instructions.

If I understood Mark's intentions in PEP-659 correctly, when you call dis.dis on any normally generated code object, specialized instructions will never show up. Specialized instructions are a hidden implementation detail, and instead the normal generic instruction will appear. So there isn't really a need to support it (any Python hand-crafted specialized instruction will likely segfault anyways).

Additionally, it seems dis._unpack_opargs is not yet prepared to support opcodes like LOAD_FAST__LOAD_FAST which take two arguments

Sorry, I don't quite understand what you mean here. The superinstructions do not use different bytecode format: ie they take only one oparg. LOAD_FAST__LOAD_FAST works by reading the *next* instruction's oparg. So:

LOAD_FAST 0
LOAD_FAST 1
LOAD_CONST 0

Becomes:

LOAD_FAST__LOAD_FAST 0
LOAD_FAST 1
LOAD_CONST 0

LOAD_FAST__LOAD_FAST will increment program counter inline [1] to read the oparg of the instruction after it without going through eval loop, then on dispatch it will effectively "skip" the next instruction and go straight to LOAD_CONST. The speedup comes from skipping eval-loop and dispatch overhead of one instruction. The bytecode format is unchanged.

[1]

cpython/Python/ceval.c

Line 1755 in 07cf10b

NEXTOPARG();

I hope this helps you.

[martmists-gh]

Specialized instructions are a hidden implementation detail, and instead the normal generic instruction will appear.

Will this change be reflected in co_code? If yes, then I'm aware of cases where this may be used by bytecode modification libraries. If not, how would it deal with co_code being modified?

Sorry, I don't quite understand what you mean here. The superinstructions do not use different bytecode format: ie they take only one oparg.
I see, so it'll just replace the first of the two ops to streamline the process rather than replacing the second one with a dummy instruction

[Fidget-Spinner]

Will this change be reflected in co_code?
No. As per PEP-659:

"Once any instruction in a code object has executed a few times, that code object will be "quickened" by allocating a new array for the bytecode that can be modified at runtime, and is not constrained as the code.co_code object is."

Example:

x = lambda a: len(a)
print(x.__code__.co_code)

#quickened
for _ in range(10):
 x('')

print(x.__code__.co_code)

There should be no change.

If not, how would it deal with co_code being modified?
Disclaimer: I am not a contributor to such libraries, so my knowledge in this area is limited.

co_code is readonly and the bytes object it points is also immutable. So I'd imagine that they'd have to create a brand new code object, in which case it shouldn't matter. But if there is some way to modify co_code in-place using ctypes: I don't know. Sorry! The best way to find out is to test those libraries with 3.11 and see if such functionality breaks, then file a bug report here please. Thank you.

[markshannon]

127 and 48 aren't instructions.
>>> opcode.stack_effect(48)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid opcode or oparg

So the output from dis seems correct.

If not, how would it deal with co_code being modified?

The co_code attribute is immutable and read only. It cannot be modified.

Specialized instructions and super instructions are private to the interpreter.
If they are visible to Python code (not counting ctypes hacks), then that is a bug.

[Fidget-Spinner]

Closing this issue as the behavior in dis module is as intended. For any potential issues to 3rd party packages discovered in the future, consider opening an issue in that package's issue tracker, or submitting a brand new issue to bugs.python.org.