Bump minimum required Python to 3.8

[takluyver]

We're generally quite conservative with dependencies, because h5py is often used in big institutional environments that may not upgrade quickly. But Python 3.8 is 3.5 years old by now, so I think it's reasonable to drop 3.7.

I've removed some test jobs, and moved some others onto newer versions of Python, trying to keep a decent coverage of different combinations.

[takluyver]

Nice, I managed to get a failure on each of the 3 CI services. 😆

[takluyver]

• Appveyor: I've hopefully figured this out 🤞 , newer Python versions are only on a newer Windows image. Now to wait for it to build everything again.
• Travis: Trying to persuade pip not to hide the error message from building numpy 😕
• Azure: ctypes finds libhdf5.dylib on Python 3.8, 3.9, 3.10, but for some reason not on 3.11. The 'What's new' page for 3.11 does not mention any changes to ctypes.

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (10f089a) 89.79% compared to head (b83572e) 89.79%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2262   +/-   ##
=======================================
  Coverage   89.79%   89.79%           
=======================================
  Files          17       17           
  Lines        2390     2390           
=======================================
  Hits         2146     2146           
  Misses        244      244           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[takluyver]

Whew. This has been an entirely disproportionate pain. It seems that for some reason, the ppc64le build on Travis was hiding stderr. It took me ages to realise this - partly because why on earth would a CI service hide stderr, and partly because pip sends part of its logging to stdout, so it looked like messages going through logging were showing up.

I eventually added some shell 2>&1 redirects and got the error messages from NumPy. I spotted some warnings about not finding Blas, so I added that, but it didn't change the error.

This looks like the crucial piece:

  INFO: building 'numpy.core._simd' extension
  INFO: compiling C dispatch-able sources
  INFO: CCompilerOpt.parse_targets[1853] : looking for '@targets' inside ->  build/src.linux-ppc64le-3.9/numpy/core/src/_simd/_simd.dispatch.c
  INFO: CCompilerOpt._generate_config[2593] : generate dispatched config ->  build/src.linux-ppc64le-3.9/numpy/core/src/_simd/_simd.dispatch.h
  INFO: CCompilerOpt._wrap_target[2553] : wrap dispatch-able target ->  build/src.linux-ppc64le-3.9/numpy/core/src/_simd/_simd.dispatch.vsx3.c
  INFO: C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
  creating build/temp.linux-ppc64le-3.9/build/src.linux-ppc64le-3.9/numpy/core/src/_simd
  INFO: compile options: '-DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Ibuild/src.linux-ppc64le-3.9/numpy/core/src/_simd -Inumpy/core/include -Ibuild/src.linux-ppc64le-3.9/numpy/core/include/numpy -Ibuild/src.linux-ppc64le-3.9/numpy/distutils/include -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/home/travis/virtualenv/python3.9.15/include -I/opt/python/3.9.15/include/python3.9 -Ibuild/src.linux-ppc64le-3.9/numpy/core/src/common -Ibuild/src.linux-ppc64le-3.9/numpy/core/src/npymath -c'
  extra options: '-mcpu=power9 -mtune=power9'
  INFO: gcc: build/src.linux-ppc64le-3.9/numpy/core/src/_simd/_simd.dispatch.vsx3.c
  INFO: C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
  INFO: compile options: '-DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Ibuild/src.linux-ppc64le-3.9/numpy/core/src/_simd -Inumpy/core/include -Ibuild/src.linux-ppc64le-3.9/numpy/core/include/numpy -Ibuild/src.linux-ppc64le-3.9/numpy/distutils/include -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/home/travis/virtualenv/python3.9.15/include -I/opt/python/3.9.15/include/python3.9 -Ibuild/src.linux-ppc64le-3.9/numpy/core/src/common -Ibuild/src.linux-ppc64le-3.9/numpy/core/src/npymath -c'
  extra options: '-mcpu=power8'
  INFO: gcc: build/src.linux-ppc64le-3.9/numpy/core/src/_simd/_simd.dispatch.c
  numpy/core/src/_simd/_simd.dispatch.c.src: In function ‘simd__intrin_extract0_f64’:
  numpy/core/src/_simd/_simd.dispatch.c.src:250:1: error: insn does not satisfy its constraints:
   SIMD_IMPL_INTRIN_1(extract0_@sfx@, @sfx@, v@sfx@)
   ^~~~~~~~~~~~~~~~~~
  (insn 176 164 156 3 (set (reg:DF 31 31 [orig:157 _3 ] [157])
          (mem/j/c:DF (plus:DI (reg:DI 10 10 [231])
                  (reg:DI 5 5 [216])) [38  S8 A8])) "numpy/core/src/_simd/_simd.dispatch.c.src":250 515 {*movdf_hardfloat64}
       (expr_list:REG_DEAD (reg:DI 5 5 [216])
          (nil)))
  numpy/core/src/_simd/_simd.dispatch.c.src:250:1: internal compiler error: in extract_constrain_insn, at recog.c:2213
  Please submit a full bug report,
  with preprocessed source if appropriate.
  See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
  error: Command "gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Ibuild/src.linux-ppc64le-3.9/numpy/core/src/_simd -Inumpy/core/include -Ibuild/src.linux-ppc64le-3.9/numpy/core/include/numpy -Ibuild/src.linux-ppc64le-3.9/numpy/distutils/include -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/home/travis/virtualenv/python3.9.15/include -I/opt/python/3.9.15/include/python3.9 -Ibuild/src.linux-ppc64le-3.9/numpy/core/src/common -Ibuild/src.linux-ppc64le-3.9/numpy/core/src/npymath -c build/src.linux-ppc64le-3.9/numpy/core/src/_simd/_simd.dispatch.c -o build/temp.linux-ppc64le-3.9/build/src.linux-ppc64le-3.9/numpy/core/src/_simd/_simd.dispatch.o -MMD -MF build/temp.linux-ppc64le-3.9/build/src.linux-ppc64le-3.9/numpy/core/src/_simd/_simd.dispatch.o.d -mcpu=power8" failed with exit status 1

There's a vaguely related looking error (not the same one) reported in a NumPy issue here: numpy/numpy#19643

That hinted that the issue was a new version of NumPy with an old compiler (Travis is using Ubuntu 18.04). I spotted in the release notes for 1.23 "Add support for VSX4/Power10", so I tried numpy<1.23, which seems to work. We can try with a newer Ubuntu on another day. 😓

[takluyver]

Oh, and I gave up on working out the Mac OS Python 3.11 + Homebrew thing for now; those tests can run on Python 3.8 and 3.10 instead. 🤷

[tacaswell]

I'm torn between squash merging to hide the thrashing and not to record the thrash!

Leave that to @takluyver 's discretion.

[takluyver]

I'm going for squashing - the pain is recorded here, I don't think it needs to be in the commit history too. 🙂

[ajelenak]

Kudos @takluyver on the perseverance in fixing all these problems! 🚀