python3.10 compatibility

[kousu]

Motivation for the feature

Python3.10 is standard now: Ubuntu 22.04 came out last month and uses it as its default. But ivadomed is only marked compatible up to 3.9, meaning it cannot be installed on Ubuntu 22.04, or anyone else using an up-to-date system.

This follows #770.

Description of the feature

Change

ivadomed/setup.py

Line 35 in 1da019b

python_requires='>=3.6,<3.10',

to read

    python_requires='>=3.6,<3.11',

and fix anything this breaks.

Additional context

Due to https://github.com/neuropoly/computers/issues/338, ivadomed now has to hide inside of conda envs set up like conda create -n my_env python==3.8 when we are using it internally on romane.neuro.polymtl.ca or rosenberg.neuro.polymtl.ca to do research. And come to think of it, this would affect SCT too, which we usually run on joplin.neuro.polymtl.ca -- except SCT bundles its own conda.

It would be nice to have the flexibility to install ivadomed with the faster pip install --user or python -m venv again.

[kousu]

I've started a draft #1137 to see what obvious failures pop out of CI.

[kousu]

onnxruntime==1.7.0 is one blocker problem:

https://github.com/ivadomed/ivadomed/runs/6658529084?check_suite_focus=true#step:4:68

ERROR: Ignored the following versions that require a different python version: 2.5.2 Requires-Python !=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,<3.9dev,>=2.7
ERROR: Could not find a version that satisfies the requirement onnxruntime==1.7.0 (from versions: none)
ERROR: No matching distribution found for onnxruntime==1.7.0
Error: ERROR: Action failed during dependency installation attempt with error: The process '/opt/hostedtoolcache/Python/3.10.4/x64/bin/python' failed with exit code 1

This is being tracked in #1057, though it looks like it's going to be a bit of a dead end at the moment.

I've tweaked ivadomed to use the nightly build. I expect it to fail pretty hard but we'll see: https://github.com/ivadomed/ivadomed/actions/runs/2410383291

[kousu]

Another blocker: torch 1.8.x doesn't have 3.10 builds. Reviewing our research, torch needs to be torch>=1.11.0.

I've bumped it: b21e77c and results will be https://github.com/ivadomed/ivadomed/actions/runs/2410415666

EDIT: needed some work, attempt 2: https://github.com/ivadomed/ivadomed/actions/runs/2410465467

Hot diggity, python3.10 on Linux passed all the tests:

but macOS is completely broken: :P

EDIT: I see the problem 3b944f9: attempt 3: https://github.com/ivadomed/ivadomed/actions/runs/2410531735

[kanishk16]

onnxruntime 3.12 that supports python 3.10 is scheduled to be released by mid-July. Also, microsoft/onnxruntime#9782 suggests to downgrade to 3.9.x. I was wondering if we could wait till mid-July and in the meantime bump up torch & drop support for python 3.6.

[kousu]

I bumped torch to 1.11 for this, but @kanishk16 pointed out that that is a big jump and maybe not necessary.

In https://download.pytorch.org/whl/ there's these cp310 (=cpython 3.10) builds available:

• https://download.pytorch.org/whl/cpu/torch-1.10.2-cp310-cp310-manylinux2014_aarch64.whl
• https://download.pytorch.org/whl/cpu/torch-1.11.0%2Bcpu-cp310-cp310-linux_x86_64.whl
• https://download.pytorch.org/whl/cpu/torch-1.11.0%2Bcpu-cp310-cp310-win_amd64.whl
• https://download.pytorch.org/whl/cpu/torch-1.11.0-cp310-none-macosx_10_9_x86_64.whl
• https://download.pytorch.org/whl/cpu/torch-1.11.0-cp310-none-macosx_11_0_arm64.whl
• https://download.pytorch.org/whl/torch-1.11.0-cp310-cp310-manylinux2014_aarch64.whl

But if I try to use one of them:

p115628@joplin:~$ pip install torch==9.9.9 --extra-index-url https://download.pytorch.org/whl/
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/
ERROR: Could not find a version that satisfies the requirement torch==9.9.9 (from versions: 1.11.0, 1.11.0+cpu, 1.11.0+cu102, 1.11.0+cu113, 1.11.0+cu115, 1.11.0+rocm4.3.1, 1.11.0+rocm4.5.2)
ERROR: No matching distribution found for torch==9.9.9
WARNING: There was an error checking the latest version of pip.
p115628@joplin:~$ pip install torch==9.9.9 --extra-index-url https://download.pytorch.org/whl/cpu
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
ERROR: Could not find a version that satisfies the requirement torch==9.9.9 (from versions: 1.11.0, 1.11.0+cpu)
ERROR: No matching distribution found for torch==9.9.9
WARNING: There was an error checking the latest version of pip.

Notice that it says "from versions: 1.11.0 ....". There's no 1.10.x that it considered. For some reason it's ignoring https://download.pytorch.org/whl/cpu/torch-1.10.2-cp310-cp310-manylinux2014_aarch64.whl. Maybe this platform (which is Ubuntu 22.04) doesn't count as manylinux2014?

I'll try to pin it down to 1.10 but I don't think it'll work: 8b988d0 -> https://github.com/ivadomed/ivadomed/actions/runs/2410969434

EDIT: indeed, no bueno:

log:

ERROR: Ignored the following versions that require a different python version: 1.6.2 Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0 Requires-Python >=3.7,<3.10; 1.7.0rc1 Requires-Python >=3.7,<3.10; 1.7.0rc2 Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10; 2.5.2 Requires-Python !=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,<3.9dev,>=2.7
ERROR: Could not find a version that satisfies the requirement torch~=1.10.0 (from versions: 1.11.0, 1.11.0+cpu)
ERROR: No matching distribution found for torch~=1.10.0

[GeorgeS2019]

@kanishk16

onnxruntime 1.12 that supports python 3.10 is scheduled to be released by mid-July. Also, microsoft/onnxruntime#9782 suggests to downgrade to 3.9.x. I was wondering if we could wait till mid-July and in the meantime bump up torch & drop support for python 3.6.

Information where to download onnxruntime 1.12 is embedded within the notebook

The notebook [updated ~1 month ago] requirement for onnxruntime 1.12 is mandatory.

from packaging import version
from onnxruntime import __version__ as ort_version

if version.parse(ort_version) >= version.parse("1.12.0"):
    from onnxruntime.transformers.models.gpt2.gpt2_helper import Gpt2Helper, MyGPT2LMHeadModel
else:
    from onnxruntime.transformers.gpt2_helper import Gpt2Helper, MyGPT2LMHeadModel

    raise RuntimeError("Please install onnxruntime 1.12.0 or later to run this notebook")

Onnxruntime 1.12 needs to be provided as nightly (not just for python, but also for c#, c++ etc. )

[kanishk16]

@kousu looks like if we want to support python3.10, torch 1.11.0 is the only way to go pytorch/pytorch#67669 (comment)

[lifetheater57]

ONNXRuntime v1.12 is out! https://github.com/microsoft/onnxruntime/releases/tag/v1.12.0