[Python] test_fs fails when run from a different drive on Windows

[asfimport]

path = "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_construct_from_single_fil0\single-file"
_, path = FileSystem.from_uri(path)
path == "/Users/VssAdministrator/AppData/Local/Temp/pytest-of-VssAdministrator/pytest-0/test_construct_from_single_fil0/single-file"

Reporter: Krisztian Szucs / @kszucs
Assignee: Antoine Pitrou / @pitrou

PRs and other links:

• GitHub Pull Request #6911

_{Note: This issue was originally created as ARROW-8406. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
I cannot reproduce:

>>> path = r"C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_construct_from_single_fil0\single-file"
>>> fs.FileSystem.from_uri(path)
(<pyarrow._fs.LocalFileSystem at 0xbfe81b8>,
 'C:/Users/VssAdministrator/AppData/Local/Temp/pytest-of-VssAdministrator/pytest-0/test_construct_from_single_fil0/single-file')

[asfimport]

Antoine Pitrou / @pitrou:
Where did you see this?

[asfimport]

Antoine Pitrou / @pitrou:
Nevertheless, I can reproduce an error on test_fs.py when running from "Z:"

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
AFAIK, on CI this happened on Azure (but not on Appveyor), where on Azure the files are on the D:/ drive:

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@pitrou I am not sure it is just the filesystem test that needs fixing.

There are other tests failing (in test_parquet), see eg https://dev.azure.com/ursa-labs/crossbow/_build/results?buildId=9714&view=logs&j=4c86bc1b-1091-5192-4404-c74dfaad23e7&t=ec99a26b-0264-5e86-36fb-9cfd0ca0f9f3&l=3910

[asfimport]

Antoine Pitrou / @pitrou:
I cannot reproduce the test_parquet failure here.

[asfimport]

Antoine Pitrou / @pitrou:
In any case, the first step to understanding the Azure failure would be for Datasets to raise an error when a file doesn't exist. Otherwise we're poking in the dark.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

I cannot reproduce the test_parquet failure here.

Are you using a different drive?

[asfimport]

Antoine Pitrou / @pitrou:
Ok, I tried putting ARROW_TEST_DATA and PARQUET_TEST_DATA on a different drive, and I can still not reproduce. This may have to with the fact that Arrow itself is on a different drive (though I'm not sure why).

[asfimport]

Krisztian Szucs / @kszucs:
@pitrou could you try with the path prefixed with the sceme? file://C:/Users/VssAdministrator...

[asfimport]

Antoine Pitrou / @pitrou:
Try what exactly?

[asfimport]

Krisztian Szucs / @kszucs:

>>> FileSystem.from_uri('D:/something.parquet')
(<pyarrow._fs.LocalFileSystem object at 0x000001D673635330>, 'D:/something.parquet')
>>> FileSystem.from_uri('D:\something.parquet')
(<pyarrow._fs.LocalFileSystem object at 0x000001D672D625B0>, 'D:/something.parquet')
>>> FileSystem.from_uri('file://D:/something.parquet')
(<pyarrow._fs.LocalFileSystem object at 0x000001D672D625B0>, '/something.parquet')
>>> FileSystem.from_uri('file://D:\something.parquet')
(<pyarrow._fs.LocalFileSystem object at 0x000001D672D6FFB0>, '/something.parquet')

[asfimport]

Krisztian Szucs / @kszucs:
Issue resolved by pull request 6911
#6911