Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Dask connector #199

Closed
wants to merge 2 commits into from
Closed

WIP: Dask connector #199

wants to merge 2 commits into from

Conversation

martindurant
Copy link
Contributor

cc @avaranovich
Let's put this code in here, if MS is amenable. I would appreciate if you would push your testing strategy.


This checklist is used to make sure that common guidelines for a pull request are followed.

Description of the change

Adds dask shim to ADLFS
.
This allows parallel access to Azure across a dask cluster of workers,
giving access to commands like

import dask.dataframe as dd
df = dd.read_parquet('adl://mybucket/path/to/data.parquet')

General Guidelines

  • The PR has modified HISTORY.rst with an appropriate description of the change and a version increment.
  • The PR has supporting test coverage that confirm the expected behavior and protects against regressions, including necessary recordings.
  • Links to associated bugs, if any, are in the description.

Martin Durant added 2 commits October 23, 2017 13:02
This allows parallel access to Azure across a dask cluster of workers,
giving access to commands like

```python
import dask.dataframe as dd
df = dd.read_parquet('adl://mybucket/path/to/data.parquet')
```

Code from https://github.com/oxygenanywhere/dask/blob/master/dask/bytes/adl.py
@msftclas
Copy link

msftclas commented Oct 23, 2017

CLA assistant check
All CLA requirements met.

@martindurant martindurant changed the title WIP: Dasky WIP: Dask connector Oct 23, 2017
@coveralls
Copy link

coveralls commented Oct 23, 2017

Coverage Status

Coverage decreased (-3.3%) to 80.956% when pulling 2d033da on martindurant:dasky into d18e47d on Azure:master.

@akharit akharit changed the base branch from master to dev September 25, 2018 19:25
@martindurant
Copy link
Contributor Author

This PR is stale, closing now.
It would be far better at this point to refactor ADLFS to depend on fsspec for conformance with other filesystems and compliance with pyarrow, Intake and dask.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants