WIP: [ci] remove pin on dask and distributed in CI (fixes #4285) #4307
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I checked https://repo.anaconda.com/pkgs/main/noarch/ this morning and |
|
I've learned some interesting things investigating this tonight, but I'm still really unsure what is happening. On conda channels, there are two packages:
This is described at https://docs.dask.org/en/latest/install.html#conda. So, said another way, the conda package However, I've found that sometimes a dependency on the conda package In a Python 3.8 environment, installing docker run \
--env CONDA_ENV=base \
-it continuumio/miniconda3 \
/bin/bash \
-c "conda install -q -y -n base dask-core &&
conda install -q -y -n base distributed &&
conda list --explicit --name base"full environment (click me)In a Python 3.8 environment, installing docker run \
--env CONDA_ENV=base \
-it continuumio/miniconda3 \
/bin/bash \
-c "conda install -q -y -n base dask-core distributed &&
conda list --explicit --name base"full environment (click me)And I don't know a good communication channel to report this to. Unlike the Note that using only docker run \
--env CONDA_ENV=base \
-it continuumio/miniconda3 \
/bin/bash \
-c "conda install -q -y -c conda-forge -n base dask-core distributed &&
conda list --explicit --name base"full environment (click me)Given all of the preceding information and the fact the the decision was made in #4054 to not use |
|
Investigating #4285 (comment)
It looks like they are always released to PyPI on the same day.
|
|
Hi, James. Just out of curiosity, why is installing from |
see #4054 (review) |
|
A relevant issue was just opened in the Dask community repo: dask/community#160 And a corresponding one was opened at |
|
good news: yesterday Anaconda published bad news: in environments with version 2021.5.1 of all those libraries, LightGBM's tests are still breaking For example, here's It looks like we got the newest versions of the relevant We got We got full environment (click me)I need to do some investigation to understand why these tests are breaking. |
|
Hi James. It seems that the I did some modifications to [<1x3 sparse matrix of type '<class 'numpy.float64'>'
with 3 stored elements in Compressed Sparse Row format>, <1x3 s...format>, <1x3 sparse matrix of type '<class 'numpy.float64'>'
with 3 stored elements in Compressed Sparse Row format>]My modification was: elif isinstance(data, dask_Array):
predict_fn = partial(
_predict_part,
model=model,
raw_score=raw_score,
pred_proba=pred_proba,
pred_leaf=pred_leaf,
pred_contrib=pred_contrib,
)
data_row = data[[0]].compute() # maybe model.client.compute(data[[0]])?
pred_row = predict_fn(data_row)
chunks = (data.chunks[0],)
if len(pred_row.shape) > 1:
chunks += (pred_row.shape[1],)
else:
kwargs['drop_axis'] = 1
return data.map_blocks(
predict_fn,
chunks=chunks,
meta=np.array([], dtype=dtype),
**kwargs
)I was hoping |
|
Thanks @jmoralez ! Great investigation. Are you interested in submitting a PR with that fix? If not or if you don't have the time, I can do that as well. I have some other ideas too, now that you've helped us narrow it down to that place in the code:
|
|
Sure, I can make a PR. Should I include the changes you've made here to the environment as well? |
yes please, thank you |
jameslamb
changed the title
|
Closing this in favor of #4351 |
|
This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#4288 introduced explicit pins on
daskanddistributedin this project's CI jobs, due to dask/distributed#4819.Starting from
dask/distributed2021.5.0, Dask maintainers decided to pindistributedlikedask=={version}to avoid such issues in the future. See dask/community#155 (comment)dask-coreanddistributed2021.5.0 are now available from the default conda channels. As soon asdaskis available there as well, I think the workaround from #4288 can be reverted.distributed 2021.5.0: https://repo.anaconda.com/pkgs/main/linux-64/dask-core 2021.5.0: https://repo.anaconda.com/pkgs/main/noarch/dask (currently 2021.4.0): https://repo.anaconda.com/pkgs/main/noarch/Just opening this PR for visibility. I think it will fail until
daskis uploaded too