-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: [ci] remove pin on dask and distributed in CI (fixes #4285) #4307
Conversation
I checked https://repo.anaconda.com/pkgs/main/noarch/ this morning and |
I've learned some interesting things investigating this tonight, but I'm still really unsure what is happening. On conda channels, there are two packages:
This is described at https://docs.dask.org/en/latest/install.html#conda. So, said another way, the conda package However, I've found that sometimes a dependency on the conda package In a Python 3.8 environment, installing docker run \
--env CONDA_ENV=base \
-it continuumio/miniconda3 \
/bin/bash \
-c "conda install -q -y -n base dask-core &&
conda install -q -y -n base distributed &&
conda list --explicit --name base"
full environment (click me)
In a Python 3.8 environment, installing docker run \
--env CONDA_ENV=base \
-it continuumio/miniconda3 \
/bin/bash \
-c "conda install -q -y -n base dask-core distributed &&
conda list --explicit --name base"
full environment (click me)
And I don't know a good communication channel to report this to. Unlike the Note that using only docker run \
--env CONDA_ENV=base \
-it continuumio/miniconda3 \
/bin/bash \
-c "conda install -q -y -c conda-forge -n base dask-core distributed &&
conda list --explicit --name base"
full environment (click me)
Given all of the preceding information and the fact the the decision was made in #4054 to not use |
Investigating #4285 (comment)
It looks like they are always released to PyPI on the same day.
|
Hi, James. Just out of curiosity, why is installing from |
see #4054 (review) |
A relevant issue was just opened in the Dask community repo: dask/community#160 And a corresponding one was opened at |
good news: yesterday Anaconda published bad news: in environments with version 2021.5.1 of all those libraries, LightGBM's tests are still breaking For example, here's
It looks like we got the newest versions of the relevant
We got We got full environment (click me)
I need to do some investigation to understand why these tests are breaking. |
Hi James. It seems that the I did some modifications to [<1x3 sparse matrix of type '<class 'numpy.float64'>'
with 3 stored elements in Compressed Sparse Row format>, <1x3 s...format>, <1x3 sparse matrix of type '<class 'numpy.float64'>'
with 3 stored elements in Compressed Sparse Row format>] My modification was: elif isinstance(data, dask_Array):
predict_fn = partial(
_predict_part,
model=model,
raw_score=raw_score,
pred_proba=pred_proba,
pred_leaf=pred_leaf,
pred_contrib=pred_contrib,
)
data_row = data[[0]].compute() # maybe model.client.compute(data[[0]])?
pred_row = predict_fn(data_row)
chunks = (data.chunks[0],)
if len(pred_row.shape) > 1:
chunks += (pred_row.shape[1],)
else:
kwargs['drop_axis'] = 1
return data.map_blocks(
predict_fn,
chunks=chunks,
meta=np.array([], dtype=dtype),
**kwargs
) I was hoping |
Thanks @jmoralez ! Great investigation. Are you interested in submitting a PR with that fix? If not or if you don't have the time, I can do that as well. I have some other ideas too, now that you've helped us narrow it down to that place in the code:
|
Sure, I can make a PR. Should I include the changes you've made here to the environment as well? |
yes please, thank you |
Closing this in favor of #4351 |
This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this. |
#4288 introduced explicit pins on
dask
anddistributed
in this project's CI jobs, due to dask/distributed#4819.Starting from
dask
/distributed
2021.5.0, Dask maintainers decided to pindistributed
likedask=={version}
to avoid such issues in the future. See dask/community#155 (comment)dask-core
anddistributed
2021.5.0 are now available from the default conda channels. As soon asdask
is available there as well, I think the workaround from #4288 can be reverted.distributed 2021.5.0
: https://repo.anaconda.com/pkgs/main/linux-64/dask-core 2021.5.0
: https://repo.anaconda.com/pkgs/main/noarch/dask (currently 2021.4.0)
: https://repo.anaconda.com/pkgs/main/noarch/Just opening this PR for visibility. I think it will fail until
dask
is uploaded too