[ci] pin dask and distributed in CI jobs

[jameslamb]

This PR is a temporary fix to unblock continuous integration jobs in this project, which are currently blocked by #4285.

[StrikerRUS]

Thank you for the fast fix!

[StrikerRUS]

Just curios: maybe you know why they have switched from semantic versioning to the date-based (or how is it called) one?
https://pypi.org/project/dask/#history

LightGBM/python-package/setup.py

Lines 346 to 349 in adf36d7

	'dask': [
	'dask[array]>=2.0.0',
	'dask[dataframe]>=2.0.0',
	'dask[distributed]>=2.0.0',

[jameslamb]

Just curios: maybe you know why they have switched from semantic versioning to the date-based (or how is it called) one?

Ah, yep! They call this calver ("calendar versioning"). I remember the main discussion was in this issue: dask/community#100.

It's a bit hard for me to follow the conversation there since it also references prior conversations, but I think this comment from dask/community#93 (comment) is probably close to the reason they decided to adopt calver:

Switching to CalVer is a strong signal that versions are checkpoints in time that have been tested more thoroughly and created purposefully with knowledge of changes throughout the community, but nothing more.

[StrikerRUS]

Ooooh, "nice"!

dask=2021.4.0 -> python[version='>=3.7']

They have already dropped support for Python 3.6.

FYI, dependabot will be able to handle such cases with slight workarounds:
https://github.com/BayesWitnesses/m2cgen/blob/f027e71cd0c2e6c9d59b0808d0897332b5895c25/requirements-test.txt#L16-L17

For now, I'm not sure what to do... Maybe temporarily pin to the latest version with Python 3.6 support?

[jameslamb]

They have already dropped support for Python 3.6.

Ah! I forgot about that! There was some discussion last year about it in dask/community#66.

Seems that numpy and pandas did that too.

• numpy: https://github.com/numpy/numpy/blob/3cf4bab1adcaa867e414a78dc112b5b4b37cf024/setup.py#L401
• pandas: https://github.com/pandas-dev/pandas/blob/master/setup.cfg#L37

I'll fix this PR to change the pin on 3.6. I believe that if we just remove pinning conda will handle providing the latest version that works with 3.6 for us. Seems like all Python 3.6 jobs on #4283, for example, succeeded without the dask and distributed pinning, because they got older versions.

For example, on Linux_latest:

dask-2021.3.0              |     pyhd3eb1b0_0           5 KB
dask-core-2021.3.0         |     pyhd3eb1b0_0         659 KB
distributed-2021.3.0       |   py36h06a4308_0         1.0 MB

https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=10078&view=logs&j=275189f9-c769-596a-7ef9-49fb48a9ab70&t=3a9e7a4a-04e6-52a0-67ea-6e8f6cfda74f

[StrikerRUS]

I'll fix this PR to change the pin on 3.6.

Thank you!

Seems that according to dask/community#155 (comment) versions will be pinned internally and we will be able to revert this workaround quite soon 🙂 .

[jameslamb]

A bunch of the R jobs and the CUDA 10.0 job failed because of what looks like a brief outage on GitLab (where we get the Eigen submodule from)

  Submodule path 'external_libs/compute': checked out '36c89134d4013b2e5e45bc55656a18bd6141995a'
  Error: fatal: unable to access 'https://gitlab.com/libeigen/eigen.git/': The requested URL returned error: 500
  Unable to fetch in submodule path 'external_libs/eigen'; trying to directly fetch 8ba1b0f41a7950dc3e1d4ed75859e36c73311235:
  Error: fatal: unable to access 'https://gitlab.com/libeigen/eigen.git/': The requested URL returned error: 502
  Fetched in submodule path 'external_libs/eigen', but it did not contain 8ba1b0f41a7950dc3e1d4ed75859e36c73311235. Direct fetching of that commit failed.
  Error: The process '/usr/local/bin/git' failed with exit code 1

I've restarted them manually

[jameslamb]

Ok seems this worked! And I double-checked the logs from the most recent builds to be sure the Dask tests are running everywhere we expect them to. All looks ok.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.