Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rebuild for python 3.11 #12

Closed
bazaah opened this issue Apr 8, 2023 · 4 comments
Closed

Rebuild for python 3.11 #12

bazaah opened this issue Apr 8, 2023 · 4 comments
Assignees
@bazaah
Labels
next-pkgrel upstream/v17

Comments

@bazaah
Copy link
Owner

bazaah commented Apr 8, 2023

Seems Archlinux finally managed to navigate the 3.11 mess and is testing 3.11 rebuilds now.

This package has to be rebuilt once that hits stable.
@bazaah bazaah self-assigned this Apr 8, 2023
@bazaah
Copy link
Owner Author

bazaah commented May 7, 2023 •
edited
Loading

First error found is in node-gyp during the mgr dashboard build:

Details

Traceback (most recent call last):
File "/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.
py", line 50, in
sys.exit(gyp.script_main())
^^^^^^^^^^^^^^^^^
File "/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp
/init.py", line 554, in script_main
return main(sys.argv[1:])
^^^^^^^^^^^^^^^^^^
File "/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp
/init.py", line 547, in main
return gyp_main(args)
^^^^^^^^^^^^^^
File "/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp
/init.py", line 520, in gyp_main
[generator, flat_list, targets, data] = Load(
^^^^^
File "/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp
/init.py", line 136, in Load
result = gyp.input.Load(build_files, default_variables, includes[:],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp
/input.py", line 2782, in Load
LoadTargetBuildFile(build_file, data, aux_data,
File "/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp
/input.py", line 391, in LoadTargetBuildFile
build_file_data = LoadOneBuildFile(build_file_path, data, aux_data,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp
/input.py", line 234, in LoadOneBuildFile
build_file_contents = open(build_file_path, 'rU').read()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid mode: 'rU' while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/np
m/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack at ChildProcess.emit (events.js:315:20)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
gyp ERR! System Linux 6.2.13-arch1-1
gyp ERR! command "/build/ceph/src/ceph-17.2.6/build/src/pybind/mgr/dashboard/frontend/node-env/bin/node" "/build/ceph/src/ceph-17.2.6/build/s
rc/pybind/mgr/dashboard/frontend/node-env/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/frontend/node_modules/tree-sitter
gyp ERR! node -v v14.15.1
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok

Seems to be fixed in nodejs/node-gyp#2219, so have to figure out why this isn't pulled in (update npm version?)

@bazaah
Copy link
Owner Author

bazaah commented May 7, 2023 •
edited
Loading

Looks like they use a bundled version of node: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/dashboard/frontend/CMakeLists.txt#L66 and I vaguely remember running into all sorts of weird errors when I enabled WITH_SYSTEM_NPM

Ideally, I can bump this to 16 or 18 and get a npm with a late enough bundled node-gyp

EDIT: npm/cli#5113, npm/cli#4664, npm/cli#4660 seems new npm bothers to validate package-lock.json files, and the upstream has something hinky in theirs. See if we can work around this somehow

@bazaah
Copy link
Owner Author

bazaah commented May 8, 2023 •
edited
Loading

Also ran into inspect.getargspec() removal in 3.11: https://docs.python.org/3/whatsnew/3.11.html#removed, though seems easy to fix based on: pytorch/pytorch#15344 (use getfullargspec() instead)

EDIT:

Actually, Ceph is on the ball here, and fixed all of their usages 3 years ago, however they use a 6 year old version of cherrypy which is missing cherrypy/cherrypy@8245a74, so trying another build with cherrypy>=18.7 -- hopefully jumping 5 major versions doesn't break anything (har har)

@bazaah
Copy link
Owner Author

bazaah commented May 10, 2023 •
edited
Loading

During a pylint run

Details 
lint: commands[4] /build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard> pylint -rn --rcfile=.pylintrc --jobs=1 . api controllers plugins serv
ices tests ../../../../qa/tasks/mgr/dashboard
Traceback (most recent call last):
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
             ^^^^^^^^^^^^
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/lib/python3.11/site-packages/pylint/__init__.py", line 19, in run_pyli
nt
    from pylint.lint import Run as PylintRun
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/lib/python3.11/site-packages/pylint/lint/__init__.py", line 75, in <mo
dule>
    from pylint.lint.check_parallel import check_parallel
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/lib/python3.11/site-packages/pylint/lint/check_parallel.py", line 7, i
n <module>
    from pylint import reporters
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/lib/python3.11/site-packages/pylint/reporters/__init__.py", line 24, i
n <module>
    from pylint import utils
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/lib/python3.11/site-packages/pylint/utils/__init__.py", line 47, in <m
odule>
    from pylint.utils.ast_walker import ASTWalker
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/lib/python3.11/site-packages/pylint/utils/ast_walker.py", line 6, in <
module>
    from astroid import nodes
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/lib/python3.11/site-packages/astroid/__init__.py", line 45, in <module
>
    import wrapt
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/lib/python3.11/site-packages/wrapt/__init__.py", line 10, in <module>
    from .decorators import (adapter_factory, AdapterFactory, decorator,
  File "/build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard/.tox/lint/lib/python3.11/site-packages/wrapt/decorators.py", line 34, in <module
>
    from inspect import ismethod, isclass, formatargspec
ImportError: cannot import name 'formatargspec' from 'inspect' (/usr/lib/python3.11/inspect.py)
lint: exit 1 (0.05 seconds) /build/ceph/src/ceph-17.2.6/src/pybind/mgr/dashboard> pylint -rn --rcfile=.pylintrc --jobs=1 . api controllers pl
ugins services tests ../../../../qa/tasks/mgr/dashboard pid=71561
  lint: FAIL code 1 (138.49=setup[91.10]+cmd[0.90,0.37,45.77,0.30,0.05] seconds)
  evaluation failed :( (140.78 seconds)

See: GrahamDumpleton/wrapt#196

Not sure how relevant this one is as this is only linting, not part of the mgr code directly, but I'm worried other bugs made be hiding behind this one (like this one was hiding behind the cherrypy error)

bazaah added a commit that referenced this issue May 12, 2023
@bazaah
repo: add mgr-dashboard-node-version.patch
d931804 
up the bundled node version to 16, so as to fix node-gyp errors
originating from a too-old npm version in 14.

Basically, python 3.11 removed a deprecated file open option from
python, which consequently blows up `npm ci`, preventing the build.

Issue: #12
References: npm/cli#5113
References: npm/cli#4664
References: npm/cli#4660
bazaah added a commit that referenced this issue May 12, 2023
@bazaah
repo: mgr/dashboard deps pylint->2.17, cherrypy->18
6741cd4 
cherrypy is a runtime dependency, while pylint is a build/lint dep, both
of them encountered import errors due to API removals in py3.11, see the
links for more.

pylint's is indirect, a dependency of a dependency:

  pylint->astroid->wrapt

but fixed by moving to a more recent version

- ceph-17.2.6-mgr-dashboard-cherrypy-18.patch
- ceph-17.2.6-mgr-dashboard-pylint-217.patch

Issue: #12
References: cherrypy/cherrypy@8245a74
References: GrahamDumpleton/wrapt#196
@bazaah bazaah closed this as completed in cbc75ed May 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bazaah bazaah

Labels
next-pkgrel upstream/v17
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bazaah