You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Dask reported some failures starting with the pyarrow 2.0 release, and specifically on Windows: dask/dask#6754
After some investigation, it seems that this is due to the ParquetDatasetPiece its path attribute now returning a path with a mixture of \\ and / in it.
It specifically happens when dask is passing a posix-style base path pointing to the dataset base directory (so using all /), and passing an fsspec-based (local) filesystem.
From a debugging output during one of the dask tests:
So you can see that the result here has a mix of \\ and /. Using pyarrow 1.0, this was consistently using /.
The reason for the change is that in pyarrow 2.0 we started to replace fsspec LocalFileSystem with our own LocalFileSystem (assuming for a local filesystem that should be equivalent). But it seems that our own LocalFileSystem has a pathsep} property that equals to os.path.sep, which is \\ on Windows (
So note that while this started being broken in pyarrow 2.0 when using fsspec filesystem, this was already "broken" before when using our own local filesystem (or when not passing any filesystem). But, 1) dask always passes an fsspec filesystem, and 2) dask uses the piece's path as dictionary key and is thus especially sensitive to the change (using it as a file path to read something in, it will probably still work even with the mixture of path separators).
Joris Van den Bossche / @jorisvandenbossche:
The actual trigger that caused this bug to surface was not a change in pyarrow 2.0 itself, but is that fssec stopped subclassing pyarrow.filesystem.FileSystem when pyarrow >= 2.0 is installed (because we now better handle fsspec filesystems, and because pyarrow.filesystem is deprecated, so this was a good change).
But the code that swaps a fsspec LocalFileSystem with our own LocalFileSystem was only reached when the passed filesystems was not a pyarrow filesystem subclass. So with pyarrow 1.0 (where fsspec-based filesystems subclassed pyarrow), this code was never run in practice .. But now with pyarrow 2.0 and the latest fsspec, we started running this.
Dask reported some failures starting with the pyarrow 2.0 release, and specifically on Windows: dask/dask#6754
After some investigation, it seems that this is due to the
ParquetDatasetPiece
itspath
attribute now returning a path with a mixture of \\ and / in it.It specifically happens when dask is passing a posix-style base path pointing to the dataset base directory (so using all
/
), and passing an fsspec-based (local) filesystem.From a debugging output during one of the dask tests:
So you can see that the result here has a mix of \\ and
/
. Using pyarrow 1.0, this was consistently using/
.The reason for the change is that in pyarrow 2.0 we started to replace fsspec LocalFileSystem with our own LocalFileSystem (assuming for a local filesystem that should be equivalent). But it seems that our own LocalFileSystem has a
pathsep
} property that equals toos.path.sep
, which is \\ on Windows (arrow/python/pyarrow/filesystem.py
Lines 304 to 306 in 9231976
So note that while this started being broken in pyarrow 2.0 when using fsspec filesystem, this was already "broken" before when using our own local filesystem (or when not passing any filesystem). But, 1) dask always passes an fsspec filesystem, and 2) dask uses the piece's path as dictionary key and is thus especially sensitive to the change (using it as a file path to read something in, it will probably still work even with the mixture of path separators).
Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Joris Van den Bossche / @jorisvandenbossche
PRs and other links:
Note: This issue was originally created as ARROW-10462. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: