Test on Python 3.11, switch CI to use `micromamba` #1758

mattwthompson · 2022-10-27T15:05:32Z

No description provided.

mattwthompson · 2022-11-05T20:33:19Z

2022-11-05T20:30:26.5151946Z ##[group]Run if [ "$INTRISICS" = "generic" ]; then
2022-11-05T20:30:26.5152463Z �[36;1mif [ "$INTRISICS" = "generic" ]; then�[0m
2022-11-05T20:30:26.5152892Z �[36;1m  nix build .#$PYTHON.pkgs.mdtraj-generic --print-build-logs�[0m
2022-11-05T20:30:26.5153264Z �[36;1melse�[0m
2022-11-05T20:30:26.5153633Z �[36;1m  nix build .#$PYTHON.pkgs.mdtraj --print-build-logs�[0m
2022-11-05T20:30:26.5153926Z �[36;1mfi�[0m
2022-11-05T20:30:26.5209250Z shell: /usr/bin/bash -e {0}
2022-11-05T20:30:26.5209548Z env:
2022-11-05T20:30:26.5209855Z   PYTHON: python311
2022-11-05T20:30:26.5210163Z   INTRINSICS: native
2022-11-05T20:30:26.5210456Z ##[endgroup]
2022-11-05T20:30:28.0953297Z copying path '/nix/store/0pypqlnsmw5k1bkh2njpiyrdp7kiqymn-source' from 'https://cache.nixos.org'...
2022-11-05T20:30:33.7852582Z error: flake 'git+file:///home/runner/work/mdtraj/mdtraj?shallow=1' does not provide attribute 'packages.x86_64-linux.python311.pkgs.mdtraj', 'legacyPackages.x86_64-linux.python311.pkgs.mdtraj' or 'python311.pkgs.mdtraj'
2022-11-05T20:30:33.7894877Z ##[error]Process completed with exit code 1.

I think this indicates it needs to be re-built in Nix world, which I don't have the time to learn for this PR.

mattwthompson · 2022-11-05T20:45:50Z

But this patch should work within conda infrastructure - see conda-forge/mdtraj-feedstock#45 where I yoinked this from

mattwthompson · 2022-11-06T02:07:11Z

I could use some help, not sure where to start with this test. It's failing only on 3.11 but not for any obvious reason I can track down to differences in Python versions.

(py311) [mdtraj] pytest tests/test_trajectory.py::test_image_molecules                                             20:57:38  ☁  py311 ☀
========================================================= test session starts ==========================================================
platform darwin -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0 -- /Users/mattthompson/mambaforge/envs/py311/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/mattthompson/software/mdtraj, configfile: pytest.ini
plugins: xdist-3.0.2
collected 1 item

tests/test_trajectory.py::test_image_molecules Fatal Python error: Segmentation fault

Current thread 0x0000000202cf6600 (most recent call first):
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/mdtraj/core/trajectory.py", line 2063 in image_molecules
  File "/Users/mattthompson/software/mdtraj/tests/test_trajectory.py", line 730 in test_image_molecules
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/python.py", line 1789 in runtest
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/runner.py", line 339 in from_call
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/main.py", line 324 in _main
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/Users/mattthompson/mambaforge/envs/py311/bin/pytest", line 10 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, mdtraj.formats.dcd, mdtraj.formats.binpos, mdtraj.formats.xtc, mdtraj.formats.trr, mdtraj.formats.dtr, mdtraj.formats.tng, mdtraj._rmsd, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_lapack, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, mdtraj._lprmsd, mdtraj.geometry._geometry, mdtraj.geometry.drid, mdtraj.geometry.neighbors, mdtraj.geometry.neighborlist, numpy.linalg.lapack_lite, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.tslib, pandas._libs.lib, pandas._libs.hashing, pandas._libs.ops, numexpr.interpreter, pandas._libs.arrays, pandas._libs.index, pandas._libs.join, pandas._libs.sparse, pandas._libs.reduction, pandas._libs.indexing, pandas._libs.internals, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.tslibs.strptime, pandas._libs.groupby, pandas._libs.testing, pandas._libs.parsers, pandas._libs.json (total: 104)
[1]    21620 segmentation fault  pytest tests/test_trajectory.py::test_image_molecules

mattwthompson · 2022-11-06T02:28:59Z

The tests involving a TNG format (which I had not heard of before today) are failing on macOS:

FAILED tests/test_tng.py::test_load_topology - ValueError: This file does not contain topology information, and no Topology was provided.

But these run fine on my machine:

(py311) [mdtraj] pytest tests/test_tng.py                                                                          21:22:36  ☁  py311 ☀
========================================================= test session starts ==========================================================
platform darwin -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0 -- /Users/mattthompson/mambaforge/envs/py311/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/mattthompson/software/mdtraj, configfile: pytest.ini
plugins: xdist-3.0.2
collected 3 items

tests/test_tng.py::test_load_trajectory PASSED                                                                                   [ 33%]
tests/test_tng.py::test_load_topology PASSED                                                                                     [ 66%]
tests/test_tng.py::test_write PASSED                                                                                             [100%]

=========================================================== warnings summary ===========================================================
../../mambaforge/envs/py311/lib/python3.11/site-packages/mdtraj/formats/__init__.py:6
  /Users/mattthompson/mambaforge/envs/py311/lib/python3.11/site-packages/mdtraj/formats/__init__.py:6: DeprecationWarning: 'xdrlib' is deprecated and slated for removal in Python 3.13
    from .xtc import XTCTrajectoryFile

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================== 3 passed, 1 warning in 0.90s =====================================================
(py311) [mdtraj]                                                                                                   21:22:42  ☁  py311 ☀

@peastman I see your name in a lot of formats/tng/ files - any idea what might be going on here? I think mdtraj/formats/tng/tng.pyx should be re-cythonized in these builds, though not tracked in the git tree. The imports in this file don't give much of a hint as to what might have changed in the new Python version.

peastman · 2022-11-10T18:28:19Z

No idea, unfortunately. I haven't looked at that code in years. Also, most of it wasn't code I wrote, just that I merged in.

mattwthompson · 2022-11-10T18:30:32Z

It was passing for years before recent shake-ups, so it's probably not an actual issue with the code.

I'll do some forensics at some point when I get them time - but I'm not squatting on that, anybody is welcome to contribute themselves before I get to it!

tbabej · 2022-12-28T18:38:05Z

I was able to build both current master and 1.9.7 on Python 3.11 and run the test suite successfully. No issues encountered other than the (already mentioned above) segmentation fault with test_image_molecules.

mattwthompson · 2023-06-10T19:43:50Z

One day I should go through and remove the 2/3 gates

$ grep -r PY3 | wc -l
      52

mattwthompson · 2023-06-10T20:01:46Z

Haven't tried getting this working on Windows, it might be possible by switching something over to PowerShell (?). There's a lot of info in the README: https://github.com/mamba-org/setup-micromamba

Start testing on Python 3.11

2d55d77

mattwthompson mentioned this pull request Nov 3, 2022

Will mdtraj run on Python 3.11? #1754

Closed

Force recythonization

70e42b9

mattwthompson added 3 commits November 5, 2022 20:39

Switch CI to Micromamba

d1e6676

Fix typo

f691247

Fiddle with paths

510d3de

mattwthompson changed the title ~~Support Python 3.11~~ Support Python 3.11, switch tests to use conda Nov 6, 2022

Do not use pytest-xdist

6845e6a

mattwthompson mentioned this pull request Dec 28, 2022

The 1.9.7 tarball contains code that does not build on Python 3.11 #1761

Closed

mattwthompson added 3 commits June 10, 2023 14:39

Merge remote-tracking branch 'origin/master' into py311

e055db4

Switch to

c959cda

Maybe read the README

50c131c

mattwthompson mentioned this pull request Jun 10, 2023

Use oldest-supported-numpy instead of numpy version pinning for build system #1751

Merged

Skip segfaulting test

d92fb9c

mattwthompson changed the title ~~Support Python 3.11, switch tests to use conda~~ Test on Python 3.11, switch CI to use micromamba Jun 10, 2023

mattwthompson added 2 commits June 10, 2023 15:05

Skip Windows

988e6bd

Skip a macOS-only failure

249d2ee

This was referenced Jun 10, 2023

Get CI working on Windows #1786

Open

Remove Python 2 compatibility #1787

Open

Remove or update TNG module #1788

Closed

mattwthompson merged commit dc22ce8 into master Jun 10, 2023
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test on Python 3.11, switch CI to use `micromamba` #1758

Test on Python 3.11, switch CI to use `micromamba` #1758

mattwthompson commented Oct 27, 2022

mattwthompson commented Nov 5, 2022

mattwthompson commented Nov 5, 2022

mattwthompson commented Nov 6, 2022

mattwthompson commented Nov 6, 2022

peastman commented Nov 10, 2022

mattwthompson commented Nov 10, 2022

tbabej commented Dec 28, 2022

mattwthompson commented Jun 10, 2023

mattwthompson commented Jun 10, 2023

Test on Python 3.11, switch CI to use micromamba #1758

Test on Python 3.11, switch CI to use micromamba #1758

Conversation

mattwthompson commented Oct 27, 2022

mattwthompson commented Nov 5, 2022

mattwthompson commented Nov 5, 2022

mattwthompson commented Nov 6, 2022

mattwthompson commented Nov 6, 2022

peastman commented Nov 10, 2022

mattwthompson commented Nov 10, 2022

tbabej commented Dec 28, 2022

mattwthompson commented Jun 10, 2023

mattwthompson commented Jun 10, 2023

Test on Python 3.11, switch CI to use `micromamba` #1758

Test on Python 3.11, switch CI to use `micromamba` #1758