Tests fail in 1-core VM

[bmwiedemann]

What happened:
running the openSUSE python-distributed package build+test in a 1-core VM failed 2 tests.

distributed/tests/test_diskutils.py .....FF
test_workspace_concurrency
test_workspace_concurrency_intense

What you expected to happen:
all tests should succeed. If there are tests that do not make sense on 1 core, cpu_count() should be used to skip them.

Minimal Complete Verifiable Example:

taskset 1 pytest-3.8 distributed/tests/test_diskutils.py

Anything else we need to know?:

Environment:

• Dask version: 2.25.0
• Python version: 3.8.6
• Operating System: openSUSE-Tumbleweed 20210104
• Install method: source / rpm .spec

This bug was found while working on reproducible builds for openSUSE.

[quasiben]

What were the failures of the test ?

A few additional questions:

• Are you the trying to build official openSuse packages for Dask/Distributed ?
• Is 2.25.0 particularly important? If not, I would suggest upgrading to latest: 2020.12.0

[bmwiedemann]

I thought, it would be easy to reproduce, so I skipped the full log before. Now I uploaded for you https://rb.zq1.de/temp/python-distributed-j1-build-log.txt

Yes, I try to rebuild the official openSUSE packages.
On Debian or openSUSE you can do:

osc co openSUSE:Factory/python-distributed && cd $_
taskset 1 osc build --noservice -j1

I tried to switch to 2020.12.0 but it seems to not work with the other versions we have in openSUSE Tumbleweed:

 distributed/client.py:32: in <module>
     from dask.utils import ensure_dict, format_bytes, funcname, stringify
 E   ImportError: cannot import name 'stringify' from 'dask.utils' (/usr/lib/python3.8/site-packages/dask/utils.py)

[jacobtomlinson]

If you switch to 2020.12.0 you probably also need dask at the same version.

[jrbourbeau]

If you switch to 2020.12.0 you probably also need dask at the same version.

Correct, the stringify function in the ImportError was added to Dask in 2020.12.0

[bmwiedemann]

Got it to build at https://build.opensuse.org/package/show/home:bmwiedemann:reproducible:dask/python-distributed
1-core log at https://rb.zq1.de/temp/python-distributed-2020.12.0-j1-build-log.txt

[bmwiedemann]

also interesting:
https://build.opensuse.org/package/live_build_log/home:bmwiedemann:reproducible:dask/python-dask:test/openSUSE_Tumbleweed/x86_64 fails:

[  136s] =================================== FAILURES ===================================
[  136s] ______________________________ test_deterministic ______________________________
[  136s] 
[  136s]     def test_deterministic():
[  136s]         pytest.importorskip("mimesis")
[  136s]     
[  136s]         b = dask.datasets.make_people(seed=123)
[  136s] >       assert b.take(1)[0]["name"] == ("Leandro", "Orr")
[  136s] E       AssertionError: assert ('Luke', 'Ellis') == ('Leandro', 'Orr')

could that be influenced by our PYTHONHASHSEED env?

[jacobtomlinson]

could that be influenced by our PYTHONHASHSEED env?

Probably