import astropy;astropy.test() hangs under macOS

[Schamschula]

I recently took over as MacPorts maintainer of py-astropy, and was reminded of an open ticket regrading astropy.test():

https://trac.macports.org/ticket/58332

I re-ran this test under Python 3.8, with the current pytest-astropy, and it's dependencies, and ran into the same issue as in comment:16 - using Python 3.7, nine months ago.

/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/_pytest/compat.py:340: PytestDeprecationWarning: The TerminalReporter.writer attribute is deprecated, use TerminalReporter._tw instead at your own risk.
See https://docs.pytest.org/en/stable/deprecations.html#terminalreporter-writer for more information.
return getattr(object, name, default)
==================================================================================================================================================== test session starts =====================================================================================================================================================
platform darwin -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1

Running tests with Astropy version 4.0.1.post1.
Running tests in /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy.

Date: 2020-08-11T19:05:22

Platform: macOS-10.14.6-x86_64-i386-64bit

Executable: /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8

Full Python Version:
3.8.5 (default, Jul 21 2020, 18:31:53)
[Clang 10.0.1 (clang-1001.0.46.4)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions:
Numpy: 1.19.1
Scipy: 1.5.2
Matplotlib: 3.3.0
h5py: not available
Pandas: 1.1.0
Cython: 0.29.21
Scikit-image: 0.17.2
asdf: not available
astropy-helpers: 4.0.1

Using Astropy options: remote_data: none.

rootdir: /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy
plugins: cov-2.10.0, arraydiff-0.3, astropy-header-0.1.2, mock-3.2.0, hypothesis-5.23.8, remotedata-0.3.2, openfiles-0.5.0, filter-subpackage-0.1.1, xonsh-0.9.19, doctestplus-0.8.0
collected 14025 items / 14 skipped / 14011 selected

../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/_erfa/tests/test_erfa.py ...................... [ 0%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/config/tests/test_configs.py ................ [ 0%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/constants/tests/test_constant.py ........ [ 0%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/constants/tests/test_pickle.py ......... [ 0%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/constants/tests/test_prior_version.py .......... [ 0%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/constants/tests/test_sciencestate.py .. [ 0%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/convolution/tests/test_convolve.py ..................................................................................................................................................................... [ 1%]
...........................................................................................................................................................................................x.x.x.x.x.x.x.x............................................................................................................ [ 3%]
......... [ 3%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/convolution/tests/test_convolve_fft.py ................................................................................................................................................................. [ 5%]
...................................................................................................................................................................................................................................................................................................................... [ 7%]
..................... [ 7%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/convolution/tests/test_convolve_kernels.py ................................................................................................................ [ 8%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/convolution/tests/test_convolve_models.py ............ [ 8%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/convolution/tests/test_convolve_nddata.py ... [ 8%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/convolution/tests/test_discretize.py ...s...........s................... [ 8%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/convolution/tests/test_kernel_class.py ................................................................................................................................................................. [ 9%]
.................... [ 9%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/convolution/tests/test_pickle.py .xx.xx.xx [ 9%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_angles.py ............................................ [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_angular_separation.py .... [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_api_ape5.py ....s [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_arrays.py ............. [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_atc_replacements.py .... [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_celestial_transformations.py ............ss.. [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_distance.py ........... [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_earth.py ..............s... [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_finite_difference_velocities.py ......s..xx [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_formatting.py .......... [ 10%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_frames.py ............................s................ [ 11%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_frames_with_velocity.py ................................................ [ 11%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_funcs.py ..... [ 11%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_iau_fullstack.py sssssssssssssssssssssssssssssssssssssssssssssssssss. [ 11%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_intermediate_transformations.py .....sssssssssssssssssssssssssssssss..........ssssssssss.ss [ 12%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_matching.py ............. [ 12%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_matrix_utilities.py .. [ 12%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_name_resolve.py s.ss [ 12%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_pickle.py ....xx.x...xx.x...xx.x. [ 12%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_regression.py .ss.s.....s.sss.s...s....s.... [ 12%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_representation.py ................................................................................s.................. [ 13%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_representation_arithmetic.py ......................................................................................... [ 14%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_representation_methods.py ............ [ 14%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_shape_manipulation.py sssssssss [ 14%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_sites.py .ss.s... [ 14%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_sky_coord.py .................................................................................................................................................................... [ 15%]
.........................................................................................................s.............................................................................................................x...............s............ [ 17%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_sky_coord_velocities.py .........x...........x.x........... [ 17%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_skyoffset_transformations.py ...........s................. [ 17%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_solar_system.py s....ssssssssssssss....ssssssss [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_transformations.py ..........................s...... [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_unit_representation.py s [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/test_velocity_corrs.py ssssssss [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/accuracy/test_altaz_icrs.py ssss [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/accuracy/test_ecliptic.py ......... [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/accuracy/test_fk4_no_e_fk4.py . [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/accuracy/test_fk4_no_e_fk5.py . [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/accuracy/test_galactic_fk4.py . [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/coordinates/tests/accuracy/test_icrs_fk5.py . [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/cosmology/tests/test_cosmology.py .......................................... [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/cosmology/tests/test_pickle.py ... [ 18%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/io/ascii/tests/test_c_reader.py .....

[pllim]

Thanks for reporting this. Did you try the dev version of astropy?

[Schamschula]

No. We try to stay on the release version. Generally, if there is demand, we have -devel ports, currently 5 out of 1638 python ports...

[pllim]

Do we have a MacPorts person, @astropy/coordinators ?

[saimn]

Could you run pytest with -v to see which test is stuck ?

[dhomeier]

Seems Github is sometimes swallowing my comments as well...

From the progress bar it looks like this it failing in test_no_header, but could you run with astropy.test(package='io.ascii', args='-vk c_reader') to confirm?

[Schamschula]

Indeed:

astropy.test(package='io.ascii', args='-vk c_reader')
==================================================================================================================================================== test session starts =====================================================================================================================================================
platform darwin -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8
cachedir: .pytest_cache

Running tests with Astropy version 4.0.1.post1.
Running tests in /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/io/ascii.

Date: 2020-08-17T18:57:15

Platform: macOS-10.14.6-x86_64-i386-64bit

Executable: /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8

Full Python Version:
3.8.5 (default, Jul 21 2020, 18:31:53)
[Clang 10.0.1 (clang-1001.0.46.4)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions:
Numpy: 1.19.1
Scipy: 1.5.2
Matplotlib: 3.3.0
h5py: 2.10.0
Pandas: 1.1.0
Cython: 0.29.21
Scikit-image: 0.17.2
asdf: not available
astropy-helpers: 4.0.1

Using Astropy options: remote_data: none.

hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/marius/Development/MacPorts/ports/.hypothesis/examples')
rootdir: /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/io/ascii
plugins: cov-2.10.0, arraydiff-0.3, astropy-header-0.1.2, mock-3.2.0, hypothesis-5.23.8, remotedata-0.3.2, openfiles-0.5.0, filter-subpackage-0.1.1, xonsh-0.9.19, doctestplus-0.8.0
collecting ... collected 532 items / 393 deselected / 139 selected

../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/io/ascii/tests/test_c_reader.py::test_simple_data[True] PASSED [ 0%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/io/ascii/tests/test_c_reader.py::test_simple_data[False] PASSED [ 1%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/io/ascii/tests/test_c_reader.py::test_read_types PASSED [ 2%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/io/ascii/tests/test_c_reader.py::test_supplied_names[True] PASSED [ 2%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/io/ascii/tests/test_c_reader.py::test_supplied_names[False] PASSED [ 3%]
../../../../../opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/astropy/io/ascii/tests/test_c_reader.py::test_no_header[True]

[dhomeier]

Have you noticed any other problems with your pytest installation when running in parallel mode? Wondering since that test is the first to force a ValueError, and parallel=True has been observed to run into problems on some systems.
You should be able to mark the parallel runs as xfail by launching the tests session with TRAVIS=1 set in your environment, although I am not sure if pytest.xfail will help in case of a hanging test.

[saimn]

I think this issue should be transfered to the main astropy repo. There were some issues with the parallel on Python 3.8 and MacOS, because Python 3.8 changed the default multiprocessing method on MacOS: #8851 #8853
And there is still a performance issue: #9864
However on Python 3.7 (https://trac.macports.org/ticket/58332) you should not have this issue, so there may be something else.

[dhomeier]

Agreed, though the exception the test is trying to trigger is a really basic check at the Python level in FastBasic.__init__ and has not changed in a long time, so I still suspect some pytest-specific problems may be involved.