Update python from 3.8 to 3.10 in v16.0 #553
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ap-wtioit
approved these changes
Jul 13, 2023
|
This approach is related with the fastapi integration in v16.0 that requires py3.10. Previous versions will not be supported rest-framework#359 |
|
Let's see the result of the CI. |
|
It looks like the python:3.X-slim-bullseye images are not installing the corresponding version of python3-dev but the one actually present in debian bullseye:
Somehow python3-dev (3.9.2-3) seems to work for python3.8 and docutils==0.14 and not for python3.10 and docutils==0.14 However even when installing python3-dev in plain python3.10-slim-bullseye image it does not break the docutils==0.14 pip install: This is using the docutils-0.14-py3-none-any.whl instead of the docutils-0.14.tar.gz that doodba (test) is using. |
Change Debian distro from bullseye to bookworm
Update 16.0.Dockerfile
ap-wtioit
suggested changes
Jul 18, 2023
There was a problem hiding this comment.
When running the install in Line 42 without the -qq parameter we get the following error message when installing wkhtmltox:
RUN apt-get install -y --no-install-recommends ./wkhtmltox.deb ...
0.279 Reading package lists...
0.500 Building dependency tree...
0.550 Reading state information...
0.588 Some packages could not be installed. This may mean that you have
0.588 requested an impossible situation or if you are using the unstable
0.588 distribution that some required packages have not yet been created
0.588 or been moved out of Incoming.
0.588 The following information may help to resolve the situation:
0.588
0.588 The following packages have unmet dependencies:
0.618 wkhtmltox : Depends: libssl1.1 but it is not installable
0.619 E: Unable to correct problems, you have held broken packages.
We need to get libssl1.1 from somewhere to also support wkhtmltopdf in bookworm.
There was a problem hiding this comment.
Currently running tests locally with getting libssl1.1 from bullseye:
...
RUN apt-get -qq update \
&& apt-get install -yqq --no-install-recommends \
curl \
&& curl -SLo wkhtmltox.deb https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/${WKHTMLTOPDF_VERSION}/wkhtmltox_${WKHTMLTOPDF_VERSION}-1.buster_amd64.deb \
&& echo "${WKHTMLTOPDF_CHECKSUM} wkhtmltox.deb" | sha256sum -c - \
&& curl -SLo libssl1.1.deb http://security.debian.org/debian-security/pool/updates/main/o/openssl/libssl1.1_1.1.1n-0+deb11u5_amd64.deb \
&& apt-get install -yqq --no-install-recommends \
./libssl1.1.deb \
./wkhtmltox.deb \
...Idea from zerotier/ZeroTierOne#1802 (comment)
There was a problem hiding this comment.
git-aggregator depends on PyYAML which is not compatible with pip installing as well:
....
31.35 Collecting PyYAML<6,>=3.13 (from kaptan->git-aggregator)
31.36 Downloading PyYAML-5.4.1.tar.gz (175 kB)
31.37 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 64.0 MB/s eta 0:00:00
31.43 Installing build dependencies: started
32.76 Installing build dependencies: finished with status 'done'
32.76 Getting requirements to build wheel: started
32.86 Getting requirements to build wheel: finished with status 'error'
32.87 error: subprocess-exited-with-error
32.87
32.87 × Getting requirements to build wheel did not run successfully.
32.87 │ exit code: 1
32.87 ╰─> [62 lines of output]
32.87 /tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
32.87 !!
32.87
32.87 ********************************************************************************
32.87 The license_file parameter is deprecated, use license_files instead.
32.87
32.87 By 2023-Oct-30, you need to update your project and remove deprecated calls
32.87 or your builds will no longer be supported.
32.87
32.87 See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
32.87 ********************************************************************************
32.87
32.87 !!
32.87 parsed = self.parsers.get(option_name, lambda x: x)(value)
32.87 running egg_info
32.87 writing lib3/PyYAML.egg-info/PKG-INFO
32.87 writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
32.87 writing top-level names to lib3/PyYAML.egg-info/top_level.txt
32.87 Traceback (most recent call last):
32.87 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
32.87 main()
32.87 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
32.87 json_out['return_val'] = hook(**hook_input['kwargs'])
32.87 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
32.87 return hook(config_settings)
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
32.87 return self._get_build_requires(config_settings, requirements=['wheel'])
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
32.87 self.run_setup()
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
32.87 exec(code, locals())
32.87 File "<string>", line 271, in <module>
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
32.87 return distutils.core.setup(**attrs)
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
32.87 return run_commands(dist)
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
32.87 dist.run_commands()
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
32.87 self.run_command(cmd)
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1234, in run_command
32.87 super().run_command(command)
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
32.87 cmd_obj.run()
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 314, in run
32.87 self.find_sources()
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
32.87 mm.run()
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 551, in run
32.87 self.add_defaults()
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
32.87 sdist.add_defaults(self)
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
32.87 super().add_defaults()
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
32.87 self._add_defaults_ext()
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
32.87 self.filelist.extend(build_ext.get_source_files())
32.87 File "<string>", line 201, in get_source_files
32.87 File "/tmp/pip-build-env-9ueu6gbn/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
32.87 raise AttributeError(attr)
32.87 AttributeError: cython_sources
32.87 [end of output]
32.87
32.87 note: This error originates from a subprocess, and is likely not a problem with pip.
32.87 error: subprocess-exited-with-error
32.87
32.87 × Getting requirements to build wheel did not run successfully.
32.87 │ exit code: 1
32.87 ╰─> See above for output.
32.87
32.87 note: This error originates from a subprocess, and is likely not a problem with pip.
Testing with PyYAML==6.0.1:
...
&& pip install \
-r https://raw.githubusercontent.com/$ODOO_SOURCE/$ODOO_VERSION/requirements.txt \
'websocket-client~=0.56' \
astor \
click-odoo-contrib \
debugpy \
pydevd-odoo \
flanker[validator] \
geoip2 \
"git-aggregator<3.0.0" \
# pin PyYAML to 6.0.1 for installing in bookworm python3.10
"PyYAML==6.0.1" \
There was a problem hiding this comment.
And the gitaggregator dependency kaptan seems not compatible with Python3.10 (tests.ScaffoldingCase.test_addons_env_double):
0.473 doodba INFO: Running gitaggregate with /opt/odoo/custom/src/repos.yaml
0.504 Traceback (most recent call last):
0.504 File "/usr/local/bin/gitaggregate", line 5, in <module>
0.504 from git_aggregator.main import main
0.504 File "/usr/local/lib/python3.10/site-packages/git_aggregator/main.py", line 22, in <module>
0.504 from .config import load_config
0.504 File "/usr/local/lib/python3.10/site-packages/git_aggregator/config.py", line 9, in <module>
0.504 import kaptan
0.504 File "/usr/local/lib/python3.10/site-packages/kaptan/__init__.py", line 15, in <module>
0.504 from collections import Mapping, Sequence
0.504 ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/lib/python3.10/collections/__init__.py)
0.507 Traceback (most recent call last):
0.507 File "/usr/local/bin/autoaggregate", line 170, in <module>
0.507 aggregate(REPOS_YAML)
0.507 File "/usr/local/bin/autoaggregate", line 72, in aggregate
0.507 check_call(
0.507 File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
0.508 raise CalledProcessError(retcode, cmd)
0.508 subprocess.CalledProcessError: Command '['gitaggregate', '--expand-env', '--config', '/opt/odoo/custom/src/repos.yaml', '--log-level', 'INFO', '--jobs', '16', 'aggregate']' returned non-zero exit status 1.
0.515 Traceback (most recent call last):
0.515 File "/opt/odoo/common/build", line 30, in <module>
0.515 subprocess.check_call(command)
0.515 File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
0.515 raise CalledProcessError(retcode, cmd)
0.515 subprocess.CalledProcessError: Command '/opt/odoo/common/build.d/100-repos-aggregate' returned non-zero exit status 1.
Which checks out:
docker run --rm python:3.8 python -c 'from collections import Mapping, Sequence'
...
<string>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
docker run --rm python:3.10 python -c 'from collections import Mapping, Sequence'
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/lib/python3.10/collections/__init__.py)
There was a problem hiding this comment.
using "git+https://github.com/rjocoleman/git-aggregator.git@feat/config-reader" instead of "git-aggregator<3.0.0" makes git-aggregator work with python3.10.
There was a problem hiding this comment.
But i still end up in the same distutils / python-devel error during the tests. It seems we would need to refactor the dotd and dependencies test dir to not mix the test cases for installing dependencies from source (pycrypto==2.6.1 --no-binary :all:) and wheels (docutils==0.14). Right now the --no-binary from pycrypto also affects wheels and I'm not sure that docutils 0.14 compilation is a valid test for Odoo 16.0 (as the line above states we test an upgrade of docutils when in fact it's a downgrade now, 0.16 -> 0.14). Both tests (the compilation of pycrypto and the install of docutils) work if they are not done in the same environment.
|
FYI docker build -f 16.0.Dockerfile .to locally run the test we use something like: python3 -m unittest -f tests |
|
Added the issue in git-aggregator as a heads up as i expect that to be blocking us in autumn when Odoo 17.0 is released. |
[IMP]Set git-aggregator to <=4.0.0 & add install libssl1
|
Tested with git-aggregator 4.0.0 |
|
How can I test poetry? |
probably by running the same commands as given here for the CI: https://github.com/Tecnativa/doodba/blob/master/.github/workflows/ci.yaml#L63 to me it would seem the a |
[FIX] Git aggregator verison >=4.0.0
|
I have a problem trying to update pyyaml because docker-compose depends on pyyaml >=3.10,<6. This throws an error caused because the version 3.0 of cython is not compatible with pyyaml 5.4.1 I can't upgrade the version of docker-compose because is no more a python module. Furthermore, docker-compose v1 is deprecated. -> https://www.docker.com/blog/announcing-compose-v2-general-availability/ Since docker compose is now built in docker, would be correct if we use docker instead of docker-compose in pyproject.toml? |
if we get rid of on our new servers we are only running docker compose and not docker-compose and it seems to work well. the only difference we discovered was the names of the containers now using |
Add pyyaml 6.0 & upgrade pre-commit to 3.3.3 Remove docker-compose dependency (incompatible with pyyaml >6) Rebuild poetry.lock
[FIX]Change pyproject.toml dependencies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Upgrade python to 3.10 for Doodba v16.0. Not sure how I can test it.