Python 3.12 support

[greenozon]

Description

Does python-xdis support python 3.12?

Background

Right now it shows an error when trying to run it on python 3.12, eg:

c:\Prg\>python
Python 3.12.0 (main, Oct 10 2023, 10:37:17) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import xdis.std as dis
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python312\Lib\site-packages\xdis\std.py", line 220, in <module>
    _std_api = make_std_api()
               ^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\xdis\std.py", line 218, in make_std_api
    return _StdApi(python_version, variant)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\xdis\std.py", line 73, in __init__
    self.opc = opc = get_opcode_module(python_version, variant)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\xdis\op_imports.py", line 174, in get_opcode_module
    return op_imports[canonic_python_version[vers_str]]
                      ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
KeyError: '3.12.0'
>>>

[rocky]

xdis doesn't have 3.12 opcodes yet. Please feel free to add it, though!

[greenozon]

Thanks for supersonic reply
I've explored branches and there is a python-3.12 one

does it support 3.12 version or that's the work in progress you are referring to?

[greenozon]

I"ve tried to use it, but looks like there is some issue in it..

pip install .
Processing c:\dev\python\python-xdis
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: six>=1.10.0 in c:\python312\lib\site-packages (from xdis==6.1.0.dev0) (1.16.0)
Requirement already satisfied: click in c:\python312\lib\site-packages (from xdis==6.1.0.dev0) (8.1.7)
Requirement already satisfied: colorama in c:\python312\lib\site-packages (from click->xdis==6.1.0.dev0) (0.4.6)
Building wheels for collected packages: xdis
  Building wheel for xdis (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for xdis (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [111 lines of output]
      running bdist_wheel
      running build
      running build_py
      copying test_unit\test_codetype.py -> build\lib\test_unit
...


      error: [WinError 87] The parameter is incorrect
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for xdis
Failed to build xdis
ERROR: Could not build wheels for xdis, which is required to install pyproject.toml-based projects

[rocky]

Installing via the older method python setup.py develop still works, at least for a little while longer.

One of the annoying parts of Python, especially for a programming language with attitude that there is only one way to do things, namely the Python way, is that it keeps changing its mind about what the one "right" way is. It has too often kept me performing a bit of work which has little to no benefit for me as the developer that I can tell. If I had just one project to update it wouldn't be as bad.

So yes, I think the packaging needs to be "updated" as well. I became aware of this from this: Mathics3/mathics-core#932 . As I am getting older and slowing down I am hoping that someone in the community will update on projects rather than me having to spend to time to understand follow figure out what I have to do to keep things working the way that had been working fine before.

$ pyenv local 3.12.1
$ pip install setuptools
Collecting setuptools
  Downloading setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB)
Downloading setuptools-69.0.3-py3-none-any.whl (819 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.5/819.5 kB 14.5 MB/s eta 0:00:00

Installing collected packages: setuptools
Successfully installed setuptools-69.0.3
$ python ./setup.py develop
python ./setup.py develop
running develop
/home/rocky/.pyenv/versions/3.12.1/lib/python3.12/site-packages/setuptools/command/develop.py:39: EasyInstallDeprecationWarning: easy_install command is deprecated.


        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  easy_install.initialize_options(self)
/home/rocky/.pyenv/versions/3.12.1/lib/python3.12/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
running egg_info
writing xdis.egg-info/PKG-INFO
writing dependency_links to xdis.egg-info/dependency_links.txt
writing entry points to xdis.egg-info/entry_points.txt
writing requirements to xdis.egg-info/requires.txt
writing top-level names to xdis.egg-info/top_level.txt
reading manifest file 'xdis.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found under directory 'test'
adding license file 'COPYING'
writing manifest file 'xdis.egg-info/SOURCES.txt'
running build_ext
Creating /home/rocky/.pyenv/versions/3.12.1/lib/python3.12/site-packages/xdis.egg-link (link to .)
Adding xdis 6.1.0.dev0 to easy-install.pth file
Installing pydisasm script to /home/rocky/.pyenv/versions/3.12.1/bin

Installed /src/external-vcs/github/rocky/python-xdis
Processing dependencies for xdis==6.1.0.dev0
Searching for click==8.1.7
Best match: click 8.1.7
Adding click 8.1.7 to easy-install.pth file

Using /home/rocky/.pyenv/versions/3.12.1/lib/python3.12/site-packages
Searching for six==1.16.0
Best match: six 1.16.0
Adding six 1.16.0 to easy-install.pth file

Using /home/rocky/.pyenv/versions/3.12.1/lib/python3.12/site-packages
Finished processing dependencies for xdis==6.1.0.dev0

[greenozon]

MAGIC!
it did really installs using the way you've just proposed! kudos!

PS I was under hard impression that I"ve got some env (windows) related issue on my PC...

[greenozon]

but the story continues! :)

any ideas what is that then?...

C:\Dev\python\pycdc-master\Debug>pydisasm  --version
pydisasm, version 6.1.0.dev0

C:\Dev\python\pycdc-master\Debug>python -V
Python 3.12.0

C:\Dev\python\pycdc-master\Debug>pydisasm --show-source -F extended bytecode test6.cpython-312.pyc
Traceback (most recent call last):
  File "\\?\C:\Python312\Scripts\pydisasm-script.py", line 33, in <module>
    sys.exit(load_entry_point('xdis', 'console_scripts', 'pydisasm')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\dev\python\python-xdis\xdis\bin\pydisasm.py", line 62, in main
    sys.stderr.write(mess % PYTHON_VERSION_STR)
                     ~~~~~^~~~~~~~~~~~~~~~~~~~
TypeError: not all arguments converted during string formatting

[greenozon]

shouldn't it be the

if not ((2, 7) <= PYTHON_VERSION_TRIPLE <= (3, 12)):

instead of existing

if not ((2, 7) <= PYTHON_VERSION_TRIPLE < (3, 12)):

[rocky]

In the master branch this is correct. In the 3.12 branch this has been corrected. BTW I think we need to compare using < rather than <= because of versions like 3.12.0.

I know you'll run into other problems along the way - that is why this hasn't been merged into master. However, when these arise please investigate an put in a PR. Thanks.

[rocky]

Actually, I am not seeing this code in master either:

https://github.com/rocky/python-xdis/blob/master/xdis/bin/pydisasm.py#L54

[greenozon]

Yeah, sorry for confusion
I was inside python-3.12 branch
master is fine

Happy New Year 2024!

[greenozon]

xdis doesn't have 3.12 opcodes yet. Please feel free to add it, though!

BTW, could you elaborate on this statement please

[rocky]

BTW, could you elaborate on this statement please

$ make check-pytest
python -m pytest pytest
=============================================== test session starts ================================================
platform linux -- Python 3.12.1, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/python-xdis
collected 27 items                                                                                                 

pytest/test_bytecode.py ..                                                                                   [  7%]
pytest/test_codetype.py .                                                                                    [ 11%]
pytest/test_cross_dis.py .                                                                                   [ 14%]
pytest/test_disasm.py ....                                                                                   [ 29%]
pytest/test_instructions.py ..                                                                               [ 37%]
pytest/test_load_file.py .                                                                                   [ 40%]
pytest/test_opcode.py F                                                                                      [ 44%]
pytest/test_stack_effect.py .F                                                                               [ 51%]
pytest/test_std.py ............s                                                                             [100%]

===================================================== FAILURES =====================================================
___________________________________________________ test_opcode ____________________________________________________

    def test_opcode():
        opc = get_opcode(PYTHON_VERSION_TRIPLE, IS_PYPY)
        opmap = dict([(k.replace("+", "_"), v) for (k, v) in dis.opmap.items()])
        opname = dis.opname
        weird_pypy = IS_PYPY and PYTHON_VERSION_TRIPLE[:2] in (
            (2, 7),
            (3, 6),
            (3, 7),
            (3, 8),
            (3, 9),
        )
    
        if not weird_pypy:
            print("Extra opmap items in dis:", set(opmap.items()) - set(opc.opmap.items()))
            print("Extra opmap items in xdis:", set(opc.opmap.items()) - set(opmap.items()))
    
            for item in opmap.items():
                assert item in opc.opmap.items(), item
    
            assert all(item in opc.opmap.items() for item in opmap.items())
    
>           assert opname == opc.opname
E           AssertionError: assert ['CACHE', 'PO...ND_SEND', ...] == ['CACHE', 'PO...ND_SEND', ...]
E             At index 86 diff: '<86>' != 'YIELD_VALUE'
E             Use -v to get more diff

pytest/test_opcode.py:27: AssertionError
----------------------------------------------- Captured stdout call -----------------------------------------------
Extra opmap items in dis: set()
Extra opmap items in xdis: set()
_____________________________________________ test_stack_effect_vs_dis _____________________________________________

    @pytest.mark.skipif(
        xdis.PYTHON_VERSION_TRIPLE < (3, 4) or xdis.IS_PYPY,
        reason="Python version is before 3.4. Can't test",
    )
    def test_stack_effect_vs_dis():
        import dis
    
        def test_one(xdis_args, dis_args, has_arg):
            effect = xstack_effect(*xdis_args)
            check_effect = dis.stack_effect(*dis_args[:2])
            assert (
                effect != -100
            ), "%d (%s) needs adjusting; should be: should have effect %d" % (
                opcode,
                opname,
                check_effect,
            )
            if has_arg:
                op_val = "with operand %d" % dis_args[1]
            else:
                op_val = ""
    
            assert check_effect == effect, "%d (%s) %s not okay; effect %d vs %d" % (
                opcode,
                opname,
                op_val,
                effect,
                check_effect,
            )
            print("%d (%s) is good: effect %d" % (opcode, opname, effect))
    
        if xdis.IS_PYPY:
            variant = "pypy"
        else:
            variant = ""
        opc = get_opcode_module(None, variant)
        for (
            opname,
            opcode,
        ) in opc.opmap.items():
            if opname in ("EXTENDED_ARG", "NOP"):
                continue
            xdis_args = [opcode, opc]
            dis_args = [opcode]
    
            # TODO: if opcode takes an argument, we should vary the arg and try
            # values in addition to 0 as done below.
            if op_has_argument(opcode, opc):
                xdis_args.append(0)
                dis_args.append(0)
                has_arg = True
            else:
                has_arg = False
    
            if (
                xdis.PYTHON_VERSION_TRIPLE >= (3, 7)
                and opcode in opc.CONDITION_OPS
                and opname
                not in (
                    "JUMP_IF_FALSE_OR_POP",
                    "JUMP_IF_TRUE_OR_POP",
                    "POP_JUMP_IF_FALSE",
                    "POP_JUMP_IF_TRUE",
                    "SETUP_FINALLY",
                )
            ):
                xdis_args.append(0)
                dis_args.append(0)
    
            if has_arg:
                for i in range(0, 3):
                    dis_args[1] = xdis_args[2] = i
>                   test_one(xdis_args, dis_args, has_arg)

pytest/test_stack_effect.py:160: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

xdis_args = [150, <module 'xdis.opcodes.opcode_312' from '/tmp/python-xdis/xdis/opcodes/opcode_312.py'>, 0]
dis_args = [150, 0], has_arg = True

    def test_one(xdis_args, dis_args, has_arg):
        effect = xstack_effect(*xdis_args)
        check_effect = dis.stack_effect(*dis_args[:2])
        assert (
            effect != -100
        ), "%d (%s) needs adjusting; should be: should have effect %d" % (
            opcode,
            opname,
            check_effect,
        )
        if has_arg:
            op_val = "with operand %d" % dis_args[1]
        else:
            op_val = ""
    
>       assert check_effect == effect, "%d (%s) %s not okay; effect %d vs %d" % (
            opcode,
            opname,
            op_val,
            effect,
            check_effect,
        )
E       AssertionError: 150 (YIELD_VALUE) with operand 0 not okay; effect 1 vs 0
E       assert 0 == 1

pytest/test_stack_effect.py:110: AssertionError
----------------------------------------------- Captured stdout call -----------------------------------------------
1 (POP_TOP) is good: effect -1
11 (UNARY_NEGATIVE) is good: effect 0
12 (UNARY_NOT) is good: effect 0
15 (UNARY_INVERT) is good: effect 0
25 (BINARY_SUBSCR) is good: effect -1
60 (STORE_SUBSCR) is good: effect -3
61 (DELETE_SUBSCR) is good: effect -2
68 (GET_ITER) is good: effect 0
71 (LOAD_BUILD_CLASS) is good: effect 1
83 (RETURN_VALUE) is good: effect -1
================================================= warnings summary =================================================
pytest/test_bytecode.py::test_find_linestarts
pytest/test_bytecode.py::test_find_linestarts
  /tmp/python-xdis/xdis/cross_dis.py:120: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    lineno_table = code.co_lnotab

pytest/test_bytecode.py::test_find_linestarts
pytest/test_bytecode.py::test_find_linestarts
  /tmp/python-xdis/xdis/cross_dis.py:134: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    byte_increments = list(code.co_lnotab[0::2])

pytest/test_bytecode.py::test_find_linestarts
pytest/test_bytecode.py::test_find_linestarts
  /tmp/python-xdis/xdis/cross_dis.py:135: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_deltas = list(code.co_lnotab[1::2])

pytest/test_codetype.py::test_codeType2Portable
  /tmp/python-xdis/xdis/codetype/__init__.py:99: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    co_linetable = code.co_lnotab,

pytest/test_load_file.py::test_load_file
  /tmp/python-xdis/pytest/test_load_file.py:42: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    if hasattr(co_file, field):

pytest/test_load_file.py::test_load_file
  /tmp/python-xdis/pytest/test_load_file.py:45: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    load_file_field = getattr(co_file, field)

pytest/test_load_file.py::test_load_file
  /tmp/python-xdis/pytest/test_load_file.py:46: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    load_module_field = getattr(co_module, field)

pytest/test_std.py::test_bytecode_from_traceback
pytest/test_std.py::test_bytecode_codeobj
pytest/test_std.py::test_bytecode_first_line
pytest/test_std.py::test_bytecode_info
pytest/test_std.py::test_bytecode__iter__
pytest/test_std.py::test_distb
pytest/test_std.py::test_get_instructions
pytest/test_std.py::test_get_instructions
pytest/test_std.py::test_findlinestarts
  /tmp/python-xdis/xdis/opcodes/opcode_311.py:395: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    locations_table_bytes = getattr(code, "co_linetable", code.co_lnotab)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================= short test summary info ==============================================
FAILED pytest/test_opcode.py::test_opcode - AssertionError: assert ['CACHE', 'PO...ND_SEND', ...] == ['CACHE', 'P...
FAILED pytest/test_stack_effect.py::test_stack_effect_vs_dis - AssertionError: 150 (YIELD_VALUE) with operand 0 n...
=============================== 2 failed, 24 passed, 1 skipped, 19 warnings in 0.20s ===============================
make: *** [Makefile:54: check-pytest] Error 1

[rocky]

704d8c8 finally addresses this.