Separate array and dataframe mindeps builds#8079
Conversation
|
There's the |
github-actions
bot
added
array
dataframe
dispatch
github-actions
bot
removed
dataframe
dispatch
…ay-dataframe-mindeps
dask/array/percentile.py
Outdated
| -------- | ||
| numpy.percentile : Numpy's equivalent Percentile function | ||
| """ | ||
| from dask.dataframe.dispatch import _percentile |
There was a problem hiding this comment.
This seems to be not going through the dispatch _percentile and instead going through local _percentile method. Will it not result in dispatch of the registered backends never being used ?
There was a problem hiding this comment.
That's correct. I was initially concerned with making whatever were needed to remove the pandas dependency in dask.array. I agree we may need further updates to get back the functionality added in #8029.
Since dask.dataframe depends on NumPy and pandas, we can import dask.array code inside dask.dataframe, but not the other way around. Is there a way we could move that dispatching logic from dask.dataframe into dask.array instead?
There was a problem hiding this comment.
Is there a way we could move that dispatching logic from
dask.dataframeintodask.arrayinstead?
I think we can move these to dask.array dispatch and backend: https://github.com/dask/dask/blob/main/dask/dataframe/dispatch.py#L25-L29
&
https://github.com/dask/dask/blob/main/dask/dataframe/backends.py#L538-L540
There was a problem hiding this comment.
Would you mind pushing up a PR which handles the dask.dataframe -> dask.array migration?
There was a problem hiding this comment.
Thanks @galipremsagar! Just left comments over in #8083
…ay-dataframe-mindeps
galipremsagar
left a comment
galipremsagar
left a comment
There was a problem hiding this comment.
Thanks @jrbourbeau !
|
Thanks for reviewing @galipremsagar! Just a note to others that gpuCI will start passing once rapidsai/cudf#9118 is in |
2 participants
This PR separates our existing array + dataframe mindeps CI build into separate builds for array and dataframe. This is to ensure that
dask.arrayanddask.dataframeonly depend on their specified minimum dependencies (i.e.numpyandnumpy+pandas, respectively).xref #8078