GTC 2.0

[jborbely]

This discussion is intended to be a roadmap to a GTC 2.0 release. As an initial starting point, this (incomplete) list of topics may be considered.

Remove Python < 3.8 support

• Cleanup GitHub Actions (Update the test matrix on GitHub Actions #38)
• Remove the pickle and older json (de)serialize functions for an Archive (Remove deprecated Archive formats #39)
• Replace unicode string literals, u'', with just the string (delete the u prefix) (use implicit unicode strings #40)
• Remove redundant __future__ statements (print_function and division – which includes removing object.__[r]div__) (Remove redundant future statements #41)
• Use f-strings (Convert to f-string #42)
• Add support for numpy 2.0 (add support for numpy 2.0 #46)
• Classes no longer need to inherit from object (use new-class style by default #47)
• Delete the PY2 constant (Delete the PY2, PY27 and PY35 constants #48)
• Cleanup try..except imports and remove Python 2 builtin names (izip, xrange, long) (cleanup imports and builtin names #49)
• Bring the tests that use the @ matrix multiplication syntax into the uarray unittest module (move matrix multiplication operator tests to test_uncertain_array.py #50)
• Use math.isfinite instead of if math.isnan(x) or math.isinf(x):
• Remove functions where, by default in Python 3, the object is already a view (e.g., Vector.iteritems() and friends)

Modernize the source code

• Use pyproject.toml instead of running setup.py
• Implement type annotations (e.g., mypy). This will help to catch development errors, but also will improve end-user experience in their IDE through tools like auto-completion. This may include using a NamedTuple rather than using namedtuple
• Use a linter and a code formatter to help catch development errors and stylistic problems (e.g., ruff)
• Use an external tool to help provide a consistent (for local development) way that: tests are run, docs are built, GTC is published, etc... Some options are using either nox or hatch
• Consider switching to a src layout

Refactoring

• Perhaps create an _internal folder and dump all non-user-facing code in there. As a developer, we would then not need to worry about deprecating things for anything inside _internal
• Clean up Archive. Harmonize the serialisation model with the language used in the code

Performance

• Firstly, need to define benchmarks (write scripts)
• Benchmark the use of bisect for a Vector

New features

• Squeezed uncertain numbers
• SVD
• Decide what to do about TODO: unary + operator #2

Documentation

• Do the docs need a re-write?
• Select a different theme?
• Use mkdocs (Markdown based) instead of sphinx (reStructuredText based)?

[jborbely]

Bare minimum initial steps could be to

1. create a 1.x branch from develop
2. bump to version 2.0.0.dev0 in develop
3. agree to only branch from (and back into) develop during the 2.0 journey and that no one pushes to develop directly

[BlairDHall]

I suggest that a review of the documentation be initiated independently of the 2.0 dev branch. This will allow smaller interative improvements (like extra examples where they are useful) to be merged before this next big release. The question of using a different documentation theme and markup can accompany the bigger 2.0 release.

So I suggest we create a specific doc-review branch from develop for the documentation clean up.

[jborbely]

Creating a doc-review branch sounds like a good idea. I do think we should be careful with this branch, as I can see it not being merged for a while and a lot of changes will happen in it. Careful meaning that only files in the docs folder may be touched in this branch (or create something like a new docs2 folder). If, for example, someone finds a spelling/grammar mistake in a docstring or in a comment in a .py file then that fix does not belong in doc-review but rather in a new branch that can quickly be merged back into develop after an easy (and quick) review.

[BlairDHall]

I think it can be more targeted, e.g. rev-doc-type-a. I see that as being achieveable in a shorter time frame and merging into dev perhaps even main. I don't think this work can be confined to the docs, because many docstrings need attention.

[jborbely]

Sounds good (more targeted). I mentioned the docstrings explicitly because, if we switch to using type annotations, then all docstrings will be changed (not necessarily the content but the format) so we shouldn't have docstrings evolving in different branches without fairly quick merges.

It sounds like all of the chore-type changes (e.g., removing Python 2) should happen sooner than later. If you want, I can be in charge of the chore-type tasks. Essentially, code cleanup/modernizing and not fixing/adding stuff.

[BlairDHall]

Sounds good. Always happy when people offer to do chores :-)

[jborbely]

I created a 1.x branch from develop.

I will work on a PR to cleanup GitHub Actions to support Python 3.8+ only

[jborbely]

Do we still want to test with 32-bit Python on Windows? The scipy maintainers have stopped providing win32 wheels on PyPI (since scipy-1.9.1, 27 Aug 2022, CPython 3.9, Windows x86) and the maintainers have no interest in building wheels for 32-bit Python anymore. There are workarounds, https://github.com/cgohlke/numpy-mkl-wheels, that we currently use to install scipy when we run the tests on Windows with Python x86 on github.

[BlairDHall]

It doesn't bother me. Do we have any MSLers running GTC on ancient lab computers? I suppose in that case they stay with earlier releases.

[jborbely]

There may be people still reliant on 32-bit Python on Windows (maybe their code depends on some other 32-bit library), but they will have pip install issues that we cannot anticipate. It is trivial for our tests in GitHub Actions (GHA) to continue to test 32-bit Python on Windows since we are aware of the issue and a way to bypass it. So, I can easy edit GHA to support 32-bit Python on Windows and I suppose that answers my original question "yes, keep testing".

Perhaps the question we should be asking ourselves is, should pip install GTC work for people still reliant on 32-bit Python on Windows? Right now, unless someone uses Python 3.9 or below, they will need to have the necessary tools to build scipy from source to install GTC (even when installing GTC 1.5.0) – in a way, we have already excluded these people for about 2 years now (and no one has raised an issue). Python 3.9 reaches end-of-life in October 2025, at which point 32-bit scipy will officially become unsupported.

GTC requires scipy in reporting.py, for coverage-factor stuff

GTC/GTC/reporting.py

Line 55 in 4dc510a

from scipy import special, optimize

so is depending on scipy for these functions worth it? I'm not sure if there is an equivalent numpy function or if the scipy algorithm could be copy-pasted directly into GTC code (i.e., how much of the relevant scipy code is written as a C extension).

The numpy maintainers still provide win32 wheels (for now).

@BlairDHall What is your view of GTC requiring scipy as a dependency?

[BlairDHall]

I am not aware of other code that we could use for those functions.

If we did not provide coverage factor functions then perhaps we could avoid scipy. We would then force users to find their own solution. Uncertainty coverage factors do not have to be very accurate, whereas those scipy functions will be carefully crafted. I think scipy will be using C implementations. It was nice to hand over responsibility for providing them to someone else.

We also use a scipy function in a minimisation routine, but perhaps that could be brought back into GTC (making it slower?).

So, keeping scipy is convenient for us and makes GTC better than it would otherwise be.

[jborbely]

Makes sense. We will keep scipy as a dependency and keep testing 32-bit Python on Windows on GHA.