Update dependencies, drop Python 3.6 and enable Python 3.9

[ziotom78]

This PR addresses issue #135 with the following changes:

• Drop support for Python 3.6 (whose EOL is 2021-12-31)
• Remove the dependency on dataclasses, which was used only because of Python 3.6
• Bump Numba dependency to 0.54 to add support for Python 3.9, and update all other relevant dependencies
• Bump ducc0 dependency to 0.20
• Bump Black dependency to address compilation errors with Python 3.9 (caution: if you experience issues with cache files after this change, you have to remove cache files from ~/.cache/black)
• Make the optional Poetry dependencies named docs mandatory, as they are required to run doctests since a long time

[ziotom78]

Personal note: as I had to reinstall Anaconda on my laptop and am now running Python 3.9, I would like to merge this ASAP, so I can stop using a dedicated conda environment to run tests on litebird_sim.

[ziotom78]

Unfortunately the toast-cmb package on PyPI (which provides TOAST 2.3.12) is currently not built for Python 3.9, and this makes the tests for this PR fail with Python 3.9.

Note that the tests do not fail because of the missing TOAST wheel, as in this case they just start building TOAST from source. However, some time later the test fails because of an error caused by functions related to atm, see the failed job here:

ImportError: cannot import name 'atm_sim_compute_slice' from 'toast._libtoast'
(/home/runner/.cache/pypoetry/virtualenvs/litebird-sim-7fRgHyCh-py3.9/lib/python3.9/site-packages/toast/_libtoast.cpython-39-x86_64-linux-gnu.so)

There are a few possibilities here, in increasing order of complexity:

1. Continue updating our build scripts till the toast-cmb dependency compiles and runs fine.
2. Ask the TOAST developers to add a new Python3.9-compatible wheel build for TOAST2 and create a new patch release (2.3.13?).
3. Given that the toast3 branch already includes py39 in the matrix used to build wheels, keep this PR frozen till TOAST3 is released and we update the dependency on toast-cmb.

As I said, I would really like to make our code usable with Python 3.9 ASAP, as this is the default version that is installed by Anaconda as of today (and Python 3.10 is going to be released in a few weeks…). @tskisner, @keskitalo, @giuspugl, @ickc, what do you think?

[ickc]

Another possible solution if you can consider making TOAST an optional dependency is How to use skip and xfail to deal with tests that cannot succeed — pytest documentation.

[ickc]

Also, I'm a bit confused: you mentioned Anaconda's default Python is 3.9 but also the PyPI package toast-cmb. While it is valid operations, mixing conda packages and PyPI packages are discouraged. They should install the one from conda instead.

If version constraint is working as expected, installing toast via conda with Python 3.9 should either downgrade the Python or warn you that there are version conflicts (although the stderr may not be obvious to reflect the problem is from the Python version mismatch.)

[tskisner]

Hi @ziotom78 , toast-2.x works fine with python 3.9, and there are py39 conda packages available for the latest 2.3.x release. I agree that wheels would be nice. Our plan for the next couple weeks is to tag a release candidate of toast-3.0 and make a legacy 2.4 branch (with a 2.4.0 release). The 2.4 branch will have critical bug fixes but no new development. If it is useful I could tag a 2.3.13 release. The wheels are automatically created and uploaded on tags.

[tskisner]

Just to follow up, I'm in the process of tagging toast 2.3.13 with formal support for python 3.9 and dropping 3.6. I'll ping this issue once updated wheels are on PyPI and updated packages on conda-forge.

[ziotom78]

Just to follow up, I'm in the process of tagging toast 2.3.13 with formal support for python 3.9 and dropping 3.6. I'll ping this issue once updated wheels are on PyPI and updated packages on conda-forge.

Wow, thanks a lot Ted, this was really quick!

Also, I'm a bit confused: you mentioned Anaconda's default Python is 3.9 but also the PyPI package toast-cmb. While it is valid operations, mixing conda packages and PyPI packages are discouraged. They should install the one from conda instead.

No, I am not mixing packages, I was mentioning Anaconda just because it's the most used Python distribution in our field, and people who have installed it in the last months are getting Python 3.9: that's the reason why I opened this PR.

[tskisner]

Hi folks, sorry for the delay- I had to sort out a conda-forge build issue on macos.

NOTE: we have taken control of the "toast" package on PyPI (it was an abandoned package by someone else). This means that we now have the same package name on PyPI (https://pypi.org/project/toast/) and on conda-forge. This 2.3.14 release has binary wheels for linux and macos for python 3.7-3.9 (3.6 is no longer supported). The package on conda-forge supports the same ranges of python versions and linux / macos. The linux packages also have variants for MKL and OpenBLAS.

And of course MPI is completely optional and only used if you also have the mpi4py package installed.

There will soon be early versions of toast-3.0 with a different API, so if you are placing the current version of toast in your requirements, then I recommend marking that as toast<3 or in poetry as ^2.3, since after the release of toast-3.0 we will only apply bugfixes to 2.x.

[ziotom78]

That's fantastic, thanks a lot @tskisner , now all the test pass and I have been able to update a few dependencies.

[ziotom78]

I'll merge this in a couple of days, unless somebody wants to comment. Be aware that this is a breaking change, as we are dropping support for Python 3.6. But I feel confident that this is the right move, as Python 3.6 is going to be deprecated by the end of 2021.