COMP: Update python packages to latest

[jamesobutler]

Python packages were last updated on July 15th 2023 (7d58308). This PR updates python packages to their latest versions. This type of update is something I usually do on about a quarterly cadence (every 3 months).

Note that SimpleITK isn't being updated here as we build it from source rather than using the provided whl on pypi.

PS C:\Users\butlej30383\AppData\Local\slicer.org\Slicer 5.5.0-2023-10-10\bin> ./PythonSlicer.exe "C:\Slicer\Utilities\Scripts\python_package_updater.py"
Searching external projects in C:\Slicer\SuperBuild
Validate external projects

[notice] A new release of pip is available: 23.1.2 -> 23.3
[notice] To update, run: python-real.exe -m pip install --upgrade pip
Package            Version         Latest    Type
------------------ --------------- --------- -----
certifi            2023.5.7        2023.7.22 wheel
cffi               1.15.1          1.16.0    wheel
chardet            5.1.0           5.2.0     wheel
charset-normalizer 3.2.0           3.3.0     wheel
cryptography       41.0.1          41.0.4    wheel
GitPython          3.1.31          3.1.37    wheel
numpy              1.25.1          1.26.1    wheel
packaging          23.1            23.2      wheel
Pillow             10.0.0          10.1.0    wheel
pip                23.1.2          23.3      wheel
pydicom            2.4.1           2.4.3     wheel
PyGithub           1.59.0          2.1.1     wheel
PyJWT              2.7.0           2.8.0     wheel
pyparsing          3.1.0           3.1.1     wheel
scipy              1.11.1          1.11.3    wheel
setuptools         68.0.0          68.2.2    wheel
SimpleITK          2.2.0rc2.dev368 2.3.0     wheel
smmap              5.0.0           5.0.1     wheel
urllib3            2.0.3           2.0.6     wheel
wheel              0.40.0          0.41.2    wheel

  certifi==2023.7.22 --hash=sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9
  # Hashes correspond to the following packages:
  #  - cffi-1.16.0-cp39-cp39-macosx_10_9_x86_64.whl
  #  - cffi-1.16.0-cp39-cp39-macosx_11_0_arm64.whl
  #  - cffi-1.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  #  - cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  #  - cffi-1.16.0-cp39-cp39-musllinux_1_1_x86_64.whl
  #  - cffi-1.16.0-cp39-cp39-win_amd64.whl
  cffi==1.16.0 --hash=sha256:582215a0e9adbe0e379761260553ba11c58943e4bbe9c36430c4ca6ac74b15ed \
               --hash=sha256:b29ebffcf550f9da55bec9e02ad430c992a87e5f512cd63388abb76f1036d8d2 \
               --hash=sha256:9cb4a35b3642fc5c005a6755a5d17c6c8b6bcb6981baf81cea8bfbc8903e8ba8 \
               --hash=sha256:8f8e709127c6c77446a8c0a8c8bf3c8ee706a06cd44b1e827c3e6a2ee6b8c098 \
               --hash=sha256:8895613bcc094d4a1b2dbe179d88d7fb4a15cee43c052e8885783fac397d91fe \
               --hash=sha256:3686dffb02459559c74dd3d81748269ffb0eb027c39a6fc99502de37d501faa8
  chardet==5.2.0 --hash=sha256:e1cf59446890a00105fe7b7912492ea04b6e6f06d4b742b2c788469e34c82970
  # Hashes correspond to the following packages:
  #  - charset_normalizer-3.3.0-cp39-cp39-macosx_10_9_universal2.whl
  #  - charset_normalizer-3.3.0-cp39-cp39-macosx_10_9_x86_64.whl
  #  - charset_normalizer-3.3.0-cp39-cp39-macosx_11_0_arm64.whl
  #  - charset_normalizer-3.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  #  - charset_normalizer-3.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  #  - charset_normalizer-3.3.0-cp39-cp39-musllinux_1_1_aarch64.whl
  #  - charset_normalizer-3.3.0-cp39-cp39-musllinux_1_1_x86_64.whl
  #  - charset_normalizer-3.3.0-cp39-cp39-win_amd64.whl
  #  - charset_normalizer-3.3.0-py3-none-any.whl
  charset-normalizer==3.3.0 --hash=sha256:e0fc42822278451bc13a2e8626cf2218ba570f27856b536e00cfa53099724828 \
                            --hash=sha256:09c77f964f351a7369cc343911e0df63e762e42bac24cd7d18525961c81754f4 \
                            --hash=sha256:12ebea541c44fdc88ccb794a13fe861cc5e35d64ed689513a5c03d05b53b7c82 \
                            --hash=sha256:805dfea4ca10411a5296bcc75638017215a93ffb584c9e344731eef0dcfb026a \
                            --hash=sha256:619d1c96099be5823db34fe89e2582b336b5b074a7f47f819d6b3a57ff7bdb86 \
                            --hash=sha256:93aa7eef6ee71c629b51ef873991d6911b906d7312c6e8e99790c0f33c576f89 \
                            --hash=sha256:153e7b6e724761741e0974fc4dcd406d35ba70b92bfe3fedcb497226c93b9da7 \
                            --hash=sha256:d97d85fa63f315a8bdaba2af9a6a686e0eceab77b3089af45133252618e70884 \
                            --hash=sha256:e46cd37076971c1040fc8c41273a8b3e2c624ce4f2be3f5dfcb7a430c1d3acc2
  # Hashes correspond to the following packages:
  #  - cryptography-41.0.4-cp37-abi3-macosx_10_12_universal2.whl
  #  - cryptography-41.0.4-cp37-abi3-macosx_10_12_x86_64.whl
  #  - cryptography-41.0.4-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  #  - cryptography-41.0.4-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  #  - cryptography-41.0.4-cp37-abi3-manylinux_2_28_aarch64.whl
  #  - cryptography-41.0.4-cp37-abi3-manylinux_2_28_x86_64.whl
  #  - cryptography-41.0.4-cp37-abi3-musllinux_1_1_aarch64.whl
  #  - cryptography-41.0.4-cp37-abi3-musllinux_1_1_x86_64.whl
  #  - cryptography-41.0.4-cp37-abi3-win_amd64.whl
  cryptography==41.0.4 --hash=sha256:80907d3faa55dc5434a16579952ac6da800935cd98d14dbd62f6f042c7f5e839 \
                       --hash=sha256:35c00f637cd0b9d5b6c6bd11b6c3359194a8eba9c46d4e875a3660e3b400005f \
                       --hash=sha256:cecfefa17042941f94ab54f769c8ce0fe14beff2694e9ac684176a2535bf9714 \
                       --hash=sha256:e40211b4923ba5a6dc9769eab704bdb3fbb58d56c5b336d30996c24fcf12aadb \
                       --hash=sha256:23a25c09dfd0d9f28da2352503b23e086f8e78096b9fd585d1d14eca01613e13 \
                       --hash=sha256:2ed09183922d66c4ec5fdaa59b4d14e105c084dd0febd27452de8f6f74704143 \
                       --hash=sha256:5a0f09cefded00e648a127048119f77bc2b2ec61e736660b5789e638f43cc397 \
                       --hash=sha256:9eeb77214afae972a00dee47382d2591abe77bdae166bda672fb1e24702a3860 \
                       --hash=sha256:c880eba5175f4307129784eca96f4e70b88e57aa3f680aeba3bab0e980b0f37d
  GitPython==3.1.37 --hash=sha256:5f4c4187de49616d710a77e98ddf17b4782060a1788df441846bddefbb89ab33
  # Hashes correspond to the following packages:
  #  - numpy-1.26.1-cp39-cp39-macosx_10_9_x86_64.whl
  #  - numpy-1.26.1-cp39-cp39-macosx_11_0_arm64.whl
  #  - numpy-1.26.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  #  - numpy-1.26.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  #  - numpy-1.26.1-cp39-cp39-musllinux_1_1_x86_64.whl
  #  - numpy-1.26.1-cp39-cp39-win_amd64.whl
  numpy==1.26.1 --hash=sha256:bb894accfd16b867d8643fc2ba6c8617c78ba2828051e9a69511644ce86ce83e \
                --hash=sha256:e44ccb93f30c75dfc0c3aa3ce38f33486a75ec9abadabd4e59f114994a9c4617 \
                --hash=sha256:9696aa2e35cc41e398a6d42d147cf326f8f9d81befcb399bc1ed7ffea339b64e \
                --hash=sha256:a5b411040beead47a228bde3b2241100454a6abde9df139ed087bd73fc0a4908 \
                --hash=sha256:1e11668d6f756ca5ef534b5be8653d16c5352cbb210a5c2a79ff288e937010d5 \
                --hash=sha256:59227c981d43425ca5e5c01094d59eb14e8772ce6975d4b2fc1e106a833d5ae2
  packaging==23.2 --hash=sha256:8c491190033a9af7e1d931d0b5dacc2ef47509b34dd0de67ed209b5203fc88c7
  # Hashes correspond to the following packages:
  #  - Pillow-10.1.0-cp39-cp39-macosx_10_10_x86_64.whl
  #  - Pillow-10.1.0-cp39-cp39-macosx_11_0_arm64.whl
  #  - Pillow-10.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  #  - Pillow-10.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  #  - Pillow-10.1.0-cp39-cp39-manylinux_2_28_aarch64.whl
  #  - Pillow-10.1.0-cp39-cp39-manylinux_2_28_x86_64.whl
  #  - Pillow-10.1.0-cp39-cp39-musllinux_1_1_aarch64.whl
  #  - Pillow-10.1.0-cp39-cp39-musllinux_1_1_x86_64.whl
  #  - Pillow-10.1.0-cp39-cp39-win_amd64.whl
  Pillow==10.1.0 --hash=sha256:0a026c188be3b443916179f5d04548092e253beb0c3e2ee0a4e2cdad72f66099 \
                 --hash=sha256:04f6f6149f266a100374ca3cc368b67fb27c4af9f1cc8cb6306d849dcdf12616 \
                 --hash=sha256:bb40c011447712d2e19cc261c82655f75f32cb724788df315ed992a4d65696bb \
                 --hash=sha256:1a8413794b4ad9719346cd9306118450b7b00d9a15846451549314a58ac42219 \
                 --hash=sha256:c9aeea7b63edb7884b031a35305629a7593272b54f429a9869a4f63a1bf04c34 \
                 --hash=sha256:b4005fee46ed9be0b8fb42be0c20e79411533d1fd58edabebc0dd24626882cfd \
                 --hash=sha256:4d0152565c6aa6ebbfb1e5d8624140a440f2b99bf7afaafbdbf6430426497f28 \
                 --hash=sha256:d921bc90b1defa55c9917ca6b6b71430e4286fc9e44c55ead78ca1a9f9eba5f2 \
                 --hash=sha256:cfe96560c6ce2f4c07d6647af2d0f3c54cc33289894ebd88cfbb3bcd5391e256
  pip==23.3 --hash=sha256:bc38bb52bc286514f8f7cb3a1ba5ed100b76aaef29b521d48574329331c5ae7b
  pydicom==2.4.3 --hash=sha256:797e84f7b22e5f8bce403da505935b0787dca33550891f06495d14b3f6c70504
  PyGithub==2.1.1 --hash=sha256:4b528d5d6f35e991ea5fd3f942f58748f24938805cb7fcf24486546637917337
  PyJWT==2.8.0 --hash=sha256:59127c392cc44c2da5bb3192169a91f429924e17aff6534d70fdc02ab3e04320
  pyparsing==3.1.1 --hash=sha256:32c7c0b711493c72ff18a981d24f28aaf9c1fb7ed5e9667c9e84e3db623bdbfb
  # Hashes correspond to the following packages:
  #  - scipy-1.11.3-cp39-cp39-macosx_10_9_x86_64.whl
  #  - scipy-1.11.3-cp39-cp39-macosx_12_0_arm64.whl
  #  - scipy-1.11.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  #  - scipy-1.11.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  #  - scipy-1.11.3-cp39-cp39-musllinux_1_1_x86_64.whl
  #  - scipy-1.11.3-cp39-cp39-win_amd64.whl
  scipy==1.11.3 --hash=sha256:a63d1ec9cadecce838467ce0631c17c15c7197ae61e49429434ba01d618caa83 \
                --hash=sha256:5305792c7110e32ff155aed0df46aa60a60fc6e52cd4ee02cdeb67eaccd5356e \
                --hash=sha256:9ea7f579182d83d00fed0e5c11a4aa5ffe01460444219dedc448a36adf0c3917 \
                --hash=sha256:c77da50c9a91e23beb63c2a711ef9e9ca9a2060442757dffee34ea41847d8156 \
                --hash=sha256:15f237e890c24aef6891c7d008f9ff7e758c6ef39a2b5df264650eb7900403c0 \
                --hash=sha256:4b4bb134c7aa457e26cc6ea482b016fef45db71417d55cc6d8f43d799cdf9ef2
  setuptools==68.2.2 --hash=sha256:b454a35605876da60632df1a60f736524eb73cc47bbc9f3f1ef1b644de74fd2a
  # Hashes correspond to the following packages:
  #  - SimpleITK-2.3.0-cp39-cp39-macosx_10_9_x86_64.whl
  #  - SimpleITK-2.3.0-cp39-cp39-macosx_11_0_arm64.whl
  #  - SimpleITK-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  #  - SimpleITK-2.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  #  - SimpleITK-2.3.0-cp39-cp39-win_amd64.whl
  SimpleITK==2.3.0 --hash=sha256:b98539c76d318367dce83f28d82dec7b7f4342dde8c77fa2b71ba831aedf4488 \
                   --hash=sha256:78500fa9efac6fcfbd8df90fa626ca38218bce0476ee742b225145c7a7ad08d5 \
                   --hash=sha256:3ef545f957a6b4ab1fe8954f6e21f5c53b827d3c44b850eb38c44c075a956c89 \
                   --hash=sha256:9e12b494a7a49a593e6460895fe8a927b230e1754e498c9136ad8b743a082745 \
                   --hash=sha256:9d117d630f6fba215ed63aac30d9def974fa5752175c36c4ab6ed9c762b2da2a
  smmap==5.0.1 --hash=sha256:e6d8668fa5f93e706934a62d7b4db19c8d9eb8cf2adbb75ef1b675aa332b69da
  urllib3==2.0.6 --hash=sha256:7a7c7003b000adf9e7ca2a377c9688bbc54ed41b985789ed576570342a375cd2
  wheel==0.41.2 --hash=sha256:75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8

Regarding the update of numpy from 1.25.1 to 1.26.1, the 1.26.x series is a continuation of the 1.25.x series with the major change being a new build system for the package. See https://numpy.org/doc/stable/release/1.26.0-notes.html

Testing Results:

Windows

• Build ✔️
• Tests ✔️

[jamesobutler]

The tests on Windows are equivalent to the current Preview release test report on Windows (https://slicer.cdash.org/viewTest.php?onlyfailed&buildid=3178343) which has 8 failures.

[jcfr]

Note that I ran a build with a clean portion of python related superbuild packages, but not a completely clean Slicer superbuild.

I will start a clean build locally and then will move forward with approval and integration. The nightly build will allow to confirm a clean build effectively work on the supported platforms.

[lassoan]

FYI, Pillow==10.1.0 caused a bit of trouble in TotalSegmentator. This new minor version of Pillow has been released just two weeks ago. Many Python packages reject yet untested minor versions by default (have requirement now for pilllow<10.1 until they have tested that their package actually works with 10.1). Therefore, if I install almost any scientific Python package then pillow will be downgraded to 10.0. However, if something has already imported pillow then this downgrade fails (file access denied).

While we could think about making downgrades/upgrades more robust (e.g., somehow allow extensions to run pip when Slicer application is not running; or check if we can use venv in Slicer)
However, it may be less work (and could avoid most version incompatibitues and unnecessary uogrades/downgrades) if we just carefully choose what time to install what package: not too soon and not too late.

[jamesobutler]

Many Python packages reject yet untested minor versions by default (have requirement now for pilllow<10.1 until they have tested that their package actually works with 10.1).

I believe in this case scikit-image usually doesn’t have an upper pin. They added the upper pin for pillow only after observing an issue related to imageio. Therefore their approach is not proactive in always having an upper pin where they don’t bump until they have tested that their package actually works with new versions of dependencies. Instead they are reactive and set upper limitations only if they have found some issue.

From the quarterly update of python packages to latest for well over a year, the top packages including scientific packages have worked well with each other. I think issues arise with the packages with a ton of dependencies like TotalSegmentor which uses some smaller packages which may not have the most resources or in the dependency chain packages have questionable version requirements. To handle this situation I think venv usage in Slicer is really going to be the only reasonable way to make sure 3rd party packages can install their correct versions that may be incompatible with Slicer core versions that have already been imported. @lassoan Did automated SlicerTotalSegmentor test cases on cDash show that installation failed due to the permission error?

[jamesobutler]

I think SlicerTotalSegmentor also has the ability to package specific Python versions at build time rather than installing at runtime. Some discussion of strict version requirements being part of #7171.

[lassoan]

Prohibiting new minor release might not be a general policy but I see it frequently enough that it is something to plan for. The fact is that it may take 1-2 months for a new major.minor release to be tests and stabilized, so we should define some embargo period for them. We can assume patch releases are fine and we can install them immediately.

To be precise, TotalSegmentator only has one significant dependency: nnunet. Nnunet is the one that has lots of other dependencies. There are several other extensions that use nnunet, which is probably the most successful image segmentation model, so it deserves attention (make sure it can be easily installed).

Bundling Python packages with an extension is good for taking care of missing wheels, but I don't see how it can resolve incompatibities.

It would be great if you could test if venv works within Slicer. In most of the deep learning processing modules it is fine if we run the computation in a separate process. If that process could run in a separate Python environment then we would have less dependency conflicts to worry about. We would probably still want to install huge packages, such as pytorch, in a shared environment, but incompatibilities due to SimpleITK and other random issues between small packages could be taken care of.

We don't run any automated tests for totalsegmentator, as we don't have CI on a GPU-equipped computer (although it could be set up). The main problem is probably testing extensions that rely on other extensions and that installing Python packages can mess up the Python environment for other purposes extensions.

[jamesobutler]

Yes it would be nice to setup automated testing within the individual extension repo using a build tree when needed. For SlicerTotalSegmentor being Python based running tests by downloading latest Slicer version and running the self tests should be fairly easy to implement. Then the testing would be localized (not impacting other Slicer extension builds) and results reported in the repo github actions. The purpose of the testing to make sure SlicerTotalSegmentor is always working if it is one of the most commonly used extensions.