base dask on fsspec

[martindurant]

• Tests added / passed
• Passes black dask / flake8 dask

lots of red in the +/-!

[mrocklin]

Side thought: You should write about this. The message might be something like "We (I) have invested a lot of time building out the bytes handling in Dask in a nicely modular way that can be helpful to others. We found that, surprisingly, it was helpful to others. This includes Intake, X, Y, and Z (zarr, rapids, ...?). We've now moved the last bits from Dask so that you too can use these systems without depending on Dask explicitly (though of course it's a very lightweight dependency).

[martindurant]

@mrocklin : yep, eventually. I posted the code here more for backup and to show the dev group that this thing really would work.

[martindurant]

Merger here requires mergers on other repos and possibly releases; but all tests pass locally now.

[martindurant]

OK, so errors are due to two main things:

• s3fs not having been released with fsspec
• gzipfile et al being seekable with py37, but not giving "operation not supported" earlier

[mrocklin]

Merger here requires mergers on other repos and possibly releases; but all tests pass locally now.

Woot!

[martindurant]

So it would be a good idea to make a decision now on read_* functions with compression. Should blocksize!=None be a straight exception? None of the compression engines support real random access normally (even if some can). Currently, random access is allowed and works on py37, but it is scanning through the file to find the right offset. That could be useful in some scenarios, but terrible in most.
If an early exception, I would probably put it in logical_size, since that's where the first scan would occur, to find the end of the uncompressed data.

[martindurant]

Just a single failure on https://travis-ci.org/dask/dask/jobs/557843869#L1002 around compression with chunksize; but quite a few windows path errors I'll need to do some work on.

[mrocklin]

So it would be a good idea to make a decision now on read_* functions with compression. Should blocksize!=None be a straight exception? None of the compression engines support real random access normally (even if some can). Currently, random access is allowed and works on py37, but it is scanning through the file to find the right offset. That could be useful in some scenarios, but terrible in most.

When you say it like that, erring seems reasonable to me. I haven't thought about this very deeply though. cc'ing @jcrist who might have thoughts?

[jcrist]

For read_csv we already warn and then switch to using blocksize=None:

dask/dask/dataframe/io/csv.py

Lines 380 to 387 in 3eb2dc8

	if blocksize and compression not in seekable_files:
	warn(
	"Warning %s compression does not support breaking apart files\n"
	"Please ensure that each individual file can fit in memory and\n"
	"use the keyword ``blocksize=None to remove this message``\n"
	"Setting ``blocksize=None``" % compression
	)
	blocksize = None

Seems reasonable to me to convert this to an error everywhere.

[martindurant]

OK, actually passes Travis for py37 (linting not yet done).
Now back to windows paths...

[martindurant]

green on windows, only flake errors on py37 :)

[martindurant]

py37: passing (with s3fs from master)
py36: failing s3 only (s3fs from release)
py35: failing s3 (s3fs from release) and something in kwargs to compression
appveyor: ony failure is unrelated partd thing

[martindurant]

Summary:

• I recommend merging this.

Outstanding:

• the test failure on appveyor is due to something with partd, which I'm pretty sure is unrelated
• the appveyor and py37 tests use master versions, which currently are the same as released versions. This can be reverted, but wanted to be sure that's the consensus.

[TomAugspurger]

the test failure on appveyor is due to something with partd, which I'm pretty sure is unrelated

Most likely #5104

the appveyor and py37 tests use master versions, which currently are the same as released versions. This can be reverted, but wanted to be sure that's the consensus.

Could they master versions be moved to the nightly upstream build? I've almost got that passing, just one remaining pandas failure that I'll try to fix today.

[martindurant]

Could they master versions be moved to the nightly upstream build?

Sure, I'm ok with that - then for this PR, should go back to released versions only in all builds.

[rjzamora]

PR#4995 now has these changes merged in, and all tests are passing.

[martindurant]

Will merge this evening, unless there are objections.

[jrbourbeau]

Woo! Thanks for all your work on this @martindurant :)