LocalFileSystem fix _strip_protocol for root directory #1477

fleming79 · 2023-12-21T01:05:32Z

This modifies LocalFileSystem to fix #945.

All local tests pass on Windows.

nicholasjng · 2023-12-21T09:08:01Z

I'd like to see this go into the AbstractFileSystem._strip_protocol base class method as well - I've had trouble with it in the case of lakeFS, where stripping away trailing slashes has some implications on URI parsing and API call results.

martindurant · 2023-12-21T14:21:03Z

I'd like to see this go into the AbstractFileSystem._strip_protocol base class

The trouble here is, that for real filesystems (local, ftp, ssh, ...), files cannot e d with "/", even though directory names are sometimes printed with the extra character. So for those, stripping is always the right thing to do. Similarly, in s3 and gcs, "/" is the "separation" character used with the remote API and also has a special place, if not exactly the same. So I think I would for now recommend that implementations that need it should provide their own _strip_protocol rather than changing the default one.

nicholasjng · 2023-12-21T18:29:31Z

The trouble here is, that for real filesystems (local, ftp, ssh, ...), files cannot e d with "/", even though directory names are sometimes printed with the extra character. So for those, stripping is always the right thing to do. Similarly, in s3 and gcs, "/" is the "separation" character used with the remote API and also has a special place, if not exactly the same. So I think I would for now recommend that implementations that need it should provide their own _strip_protocol rather than changing the default one.

I see your point, but on the other hand, trailing slashes are also the defining hint of directories in many implementations - in a different world, in those implementations, all file-ops could raise IsADirectoryErrors on trailing-slash filenames if they are not applied recursively.

In my case, I am working with URIs similiar to S3, of the structure lakefs://<bucket>/<version>/<resource>, where an empty resource marks the root of the bucket (i.e. / as root marker). I cannot just simply return the root marker there since the bucket and version are still in front, so the filename or cls.root_marker check will not do what I want.

But since the subclass implementation works fine in our case, I see no harm in keeping that in place for now.

fleming79 · 2023-12-22T01:40:24Z

My apologies if the updates have been noisy. I'm a novice when it comes to git and pull requests.

The other history has been removed when I switched to using a nested function so that remove_trailing_slash is always respected.

I note there is an error for test_fsmap_error_on_protocol_keys.

def test_fsmap_access_with_suffix(tmp_path):
    tmp_path.joinpath("b").mkdir()
    tmp_path.joinpath("b", "a").write_bytes(b"data")
    m = fsspec.get_mapper(f"file://{tmp_path}")

    with pytest.raises(KeyError):
        _ = m["b/"]

    assert m["b/a/"] == b"data"

I haven't used FSMap before but wonder if the line in the test that is failing:

assert m["b/a/"] == b"data"`

should be without the trailing separator?

assert m["b/a"] == b"data"'

martindurant · 2023-12-22T21:14:14Z

fsspec/implementations/local.py

+    return path.rstrip("/") if remove_trailing_slash else path
+
+
+def _make_path_posix(path, sep=os.sep):


Why split up the function? The windows-root check is a rare one and should ideally be checked after more common things. This would be a second case within "# windows full path"?

My reasoning was that the original make_path_posix function has several return statements. Wrapping the original function enables detection of special edge cases but also adds support for stripping the trailing suffix where appropriate.

martindurant · 2023-12-22T21:16:08Z

fsspec/implementations/local.py

+    path = _make_path_posix(path, sep)
+    if os.name == "nt" and len(path) == 3 and path[1] == ":":
+        return path  # windows root
+    return path.rstrip("/") if remove_trailing_slash else path


Presumably this i what breaks the mapper test; the key in question I think would have hit "if path.startswith("/"):" previously.

The trouble here is, that for real filesystems (local, ftp, ssh, ...), files cannot e d with "/", even though directory names are sometimes printed with the extra character.

The filename ends with a trailing '/'. Is it supposed to?

I should mention that I also added that check in open, however the issue still occurs if the check is disabled.

The issue can be resolved by stripping the trailing suffix in mapping.py->__getitem__. Doing doesn't appear to cause any regression.

k = self._key_to_str(key).rstrip("/")

…th_posix. make_path_posix with some optimisation tweaks such as change re.match to a string comparison. Should also properly format concatenated windows paths as posix.

…with all arguments

…rsions

test_fsmap_access_with_suffix - enable for windows.

- now converts to string if not.

fleming79 · 2024-01-03T06:26:22Z

make_path_posix has been divided into a native posix version and nt (windows) version. It has also been ordered with assumptions made about the most common types of paths.
A test, test_parent for testing the results of the method: _parent has also been added. Do you think _parent of a root should return the root, or should it raise an error?

fsspec/implementations/local.py

martindurant · 2024-01-04T16:16:52Z

fsspec/mapping.py

-        self.root = fs._strip_protocol(root).rstrip("/")
+        try:
+            self.root = fs._strip_protocol(root, remove_trailing_slash=True)
+        except TypeError:


This is unfortunate; in what case does rstrip fail? In this case, I think we might be explicitly joining on the separator anyway.

On Windows if the root is something like c:/, rstrip will cause a root of c: which is treated like '.', the current working directory instead of the root.
On posix rstrip will make / change to .

This is the only place that the special case for LocalFileSystem leaks, and I don't like it :|. I think it may be better to let through strange behaviour if the user thinks they want to may their whole C drive (and gets only cwd files instead).

I've changed this back.

But without .rstrip("/")

make_path_posix - removed handling of `file://' (should be done in _strip_protocol).

martindurant · 2024-01-09T15:23:18Z

py38 does not support removeprefix

AttributeError: 'str' object has no attribute 'removeprefix'

fleming79 · 2024-01-24T21:12:29Z

I think sep probably shouldn't be an argument for the function make_path_posix since having it there advertises that it is capable of producing posix for either system, but it relies on system specific functions for much of its behaviour. The associated tests can be made os specific (some previously were already).

disable test_jupyter.test_simple for windows

fleming79 · 2024-01-24T22:09:52Z

The latest changes remove sep accordingly.

martindurant · 2024-01-31T21:15:24Z

fsspec/implementations/tests/test_jupyter.py

@@ -38,6 +38,7 @@ def jupyter(tmpdir):
        P.terminate()


+@pytest.mark.skipif(WIN, reason="Subprocess gets stuck in a loop")


I didn't notice this before. Any idea what's going on?

I found the problem - Something to do with the temp dir passed to Jupyterlab in the launch call showing up as an invalid path. Surrounding the path with "" fixed it. The test now passes when I run it on Windows.

cmd = f'jupyter notebook --notebook-dir="{tmpdir}" --no-browser --port=5566'

This line was to be removed according to the thread above

Quite right - I made the changes but forgot to push them. It is done now.

martindurant · 2024-02-05T21:42:54Z

please merge from master to pick up the constraint on pytest that should fix CI

fleming79 · 2024-03-01T21:39:29Z

Did you mean something other than to sync the fork?

martindurant · 2024-03-01T21:45:10Z

That's exactly what I meant - sorry I didn't get a notification

fleming79 · 2024-03-01T22:27:05Z

No problem.
I fixed the Ruff lint issue, but there were also errors with 'dbfs'. I'm not sure how they could be related to this PR.

martindurant · 2024-03-02T01:56:18Z

@nils-braun , @a24lorie , any idea why DBFS should start failing, but only for some python versions? It is also happening on other PRs.

@fleming79 , you may wish to take the commit from #1533 , which should at least squash the intermittent SMB failures.

a24lorie · 2024-03-04T11:45:46Z

@nils-braun , @a24lorie , any idea why DBFS should start failing, but only for some python versions? It is also happening on other PRs.

@martindurant I have found this to be an issue with the pytest-vcr and urllib3 version 2.x with python 3.10 and 3.11.
Downgrading urllib3 to version 1.26.18 in Python 3.10 removes this error

martindurant · 2024-03-04T14:40:34Z

Please update your fork following #1533

fleming79 · 2024-03-07T20:40:04Z

I synced the fork with GitHub.

martindurant

Sorry this has been taking so long...

I wonder, since we now are essentially calling _strip_protocol from within LocalFileSystem, would it be easy to split off the optional remove_trailing_slash= into a separate method, to maintain the previous signature? I am still not sure what the best default is, since in most places we think of paths not having terminal sep characters even when they refer to directories.

fsspec/implementations/local.py

martindurant · 2024-03-08T21:35:16Z

fsspec/implementations/tests/test_jupyter.py

@@ -38,6 +38,7 @@ def jupyter(tmpdir):
        P.terminate()


+@pytest.mark.skipif(WIN, reason="Subprocess gets stuck in a loop")


This line was to be removed according to the thread above

fsspec/implementations/tests/test_local.py

martindurant · 2024-03-08T21:42:02Z

fsspec/mapping.py

-        self.root = fs._strip_protocol(root).rstrip("/")
+        try:
+            self.root = fs._strip_protocol(root, remove_trailing_slash=True)
+        except TypeError:


This is the only place that the special case for LocalFileSystem leaks, and I don't like it :|. I think it may be better to let through strange behaviour if the user thinks they want to may their whole C drive (and gets only cwd files instead).

…ol_no_authority

fleming79 · 2024-03-15T21:09:17Z

The fork has been synced, with no other changes.

martindurant · 2024-03-18T18:30:49Z

:)

fleming79 force-pushed the master branch from c8ec7af to c64eb0d Compare December 22, 2023 00:37

Fix _strip_protocol for windows root directory

93d92c8

fleming79 force-pushed the master branch from b1edd0e to 93d92c8 Compare December 22, 2023 01:09

martindurant reviewed Dec 22, 2023

View reviewed changes

fleming79 added 4 commits December 23, 2023 11:10

Update tests for windows root and remove_trailing_slash

f6d44d4

FSMap._key_to_str now always strips trailing separator

5b77125

make_path_posix now uses sep to identify os type

ffb2eb8

LocalFileSystem - switch from using os.name to sep

00b742f

fleming79 force-pushed the master branch from 46bd1ce to 3af29ad Compare December 23, 2023 06:57

local.py - make_path_posix removed temporary nested function _make_pa…

9d01d9b

…th_posix. make_path_posix with some optimisation tweaks such as change re.match to a string comparison. Should also properly format concatenated windows paths as posix.

fleming79 force-pushed the master branch from 3af29ad to 9d01d9b Compare December 23, 2023 11:25

fleming79 added 7 commits December 24, 2023 12:12

Fixed LocalFileSystem._parent & added test

e15955d

make_path_posix - made slightly faster in recursive calls by calling …

c0c7399

…with all arguments

Divide make_path_posix and _parent method into native posix and nt ve…

7abb0c4

…rsions

FSMap - use new remove_trailing_slash to get root

6e2df0f

test_fsmap_access_with_suffix - enable for windows.

make_path_posix

16c2c82

- now converts to string if not.

Merge branch 'master' of https://github.com/fsspec/filesystem_spec

0b4d069

make_path_posix - use stringify_path

9fa75a9

martindurant reviewed Jan 4, 2024

View reviewed changes

fleming79 and others added 4 commits January 5, 2024 08:47

Merge branch 'fsspec:master' into master

4b25084

LocalFileSystem._strip_protocol - will now lstrip for 'local' on NT.

97ee256

make_path_posix - removed handling of `file://' (should be done in _strip_protocol).

_strip_protocol fix for 3.8 string missing removeprefix

a0ee144

Merge branch 'fsspec:master' into master

a68e536

Fix version check

41147e6

Alan Fleming and others added 2 commits January 25, 2024 08:36

remove sep as argument for make_path_posix , _strip_protocol, _parent

778cd95

fix test_make_path_posix

7570de8

disable test_jupyter.test_simple for windows

martindurant reviewed Jan 31, 2024

View reviewed changes

fleming79 added 4 commits February 6, 2024 13:53

Merge branch 'fsspec:master' into master

502413f

Merge branch 'fsspec:master' into master

64433f5

Merge branch 'fsspec:master' into master

fe93b1d

Merge branch 'fsspec:master' into master

ac990da

Fix ruff lint PIE810

e0de0a6

fleming79 added 2 commits March 5, 2024 18:53

Merge branch 'fsspec:master' into master

05f073a

Merge branch 'fsspec:master' into master

7799d4e

martindurant reviewed Mar 8, 2024

View reviewed changes

fleming79 and others added 5 commits March 9, 2024 10:17

Re-enable jupyter test_simple for windows

c2352bf

Merge branch 'fsspec:master' into master

006718d

Merge branch 'master' of https://github.com/fleming79/filesystem_spec

ec56ea4

Revert setting of root in 'FSMap' and added line to test_strip_protoc…

03b3106

…ol_no_authority

Merge branch 'fsspec:master' into master

082b544

martindurant merged commit 1653552 into fsspec:master Mar 18, 2024
10 checks passed

ap-- mentioned this pull request Apr 7, 2024

LocalFileSystem restore _strip_protocol signature #1567

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LocalFileSystem fix _strip_protocol for root directory #1477

LocalFileSystem fix _strip_protocol for root directory #1477

fleming79 commented Dec 21, 2023

nicholasjng commented Dec 21, 2023

martindurant commented Dec 21, 2023

nicholasjng commented Dec 21, 2023 •

edited

Loading

fleming79 commented Dec 22, 2023

martindurant Dec 22, 2023

fleming79 Dec 22, 2023

martindurant Dec 22, 2023

fleming79 Dec 22, 2023 •

edited

Loading

fleming79 commented Jan 3, 2024

martindurant Jan 4, 2024

fleming79 Jan 4, 2024

martindurant Mar 8, 2024

fleming79 Mar 9, 2024

fleming79 Mar 10, 2024

martindurant commented Jan 9, 2024

fleming79 commented Jan 24, 2024 •

edited

Loading

fleming79 commented Jan 24, 2024

martindurant Jan 31, 2024

fleming79 Feb 1, 2024

martindurant Mar 8, 2024

fleming79 Mar 8, 2024

martindurant commented Feb 5, 2024

fleming79 commented Mar 1, 2024

martindurant commented Mar 1, 2024

fleming79 commented Mar 1, 2024

martindurant commented Mar 2, 2024 •

edited

Loading

a24lorie commented Mar 4, 2024 •

edited

Loading

martindurant commented Mar 4, 2024

fleming79 commented Mar 7, 2024

martindurant left a comment

martindurant Mar 8, 2024

martindurant Mar 8, 2024

fleming79 commented Mar 15, 2024

martindurant commented Mar 18, 2024

		return path.rstrip("/") if remove_trailing_slash else path


		def _make_path_posix(path, sep=os.sep):

		@@ -38,6 +38,7 @@ def jupyter(tmpdir):
		P.terminate()


		@pytest.mark.skipif(WIN, reason="Subprocess gets stuck in a loop")

LocalFileSystem fix _strip_protocol for root directory #1477

LocalFileSystem fix _strip_protocol for root directory #1477

Conversation

fleming79 commented Dec 21, 2023

nicholasjng commented Dec 21, 2023

martindurant commented Dec 21, 2023

nicholasjng commented Dec 21, 2023 • edited Loading

fleming79 commented Dec 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fleming79 Dec 22, 2023 • edited Loading

Choose a reason for hiding this comment

fleming79 commented Jan 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martindurant commented Jan 9, 2024

fleming79 commented Jan 24, 2024 • edited Loading

fleming79 commented Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martindurant commented Feb 5, 2024

fleming79 commented Mar 1, 2024

martindurant commented Mar 1, 2024

fleming79 commented Mar 1, 2024

martindurant commented Mar 2, 2024 • edited Loading

a24lorie commented Mar 4, 2024 • edited Loading

martindurant commented Mar 4, 2024

fleming79 commented Mar 7, 2024

martindurant left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fleming79 commented Mar 15, 2024

martindurant commented Mar 18, 2024

nicholasjng commented Dec 21, 2023 •

edited

Loading

fleming79 Dec 22, 2023 •

edited

Loading

fleming79 commented Jan 24, 2024 •

edited

Loading

martindurant commented Mar 2, 2024 •

edited

Loading

a24lorie commented Mar 4, 2024 •

edited

Loading