Fix ArrowFSWrapper parent paths for S3#2072
Conversation
ArrowFSWrapper inherited the "/" root marker for every pyarrow backend, so put() sent S3 create_dir() a path beginning with a separator. Use an instance-specific empty marker for S3 and resolve parent paths with it. Add a regression covering the path passed to the wrapped backend. Fixes fsspec#2015
|
This specific issue was seen before - can you please have a search in the issues/PR? I thought it had been solved... |
|
@martindurant I found the earlier work: #1464 and the closed #1467. That PR used a dynamic subclass plus This PR is narrower: it sets an instance |
| return super()._parent(path) | ||
|
|
||
| path = self._strip_protocol(path).lstrip("/") | ||
| return path.rsplit("/", 1)[0] if "/" in path else self.root_marker |
There was a problem hiding this comment.
root_maker is guaranteed to be "", no?
There was a problem hiding this comment.
Yes. In the empty-root branch it is guaranteed to be ""; 971351d now returns the literal directly.
| global PYARROW_VERSION | ||
| PYARROW_VERSION = get_package_version_without_import("pyarrow") | ||
| self.fs = fs | ||
| if fs.type_name == "s3": |
There was a problem hiding this comment.
No. GCS has the same key-style path semantics; the earlier #1467 regression matrix covered both S3 and GCS. 971351d now selects an empty root marker for both and parameterizes the regression over both backends.
2 participants
ArrowFSWrapper always inherited
/as its root marker, soput()derived/bucket/...parents that pyarrow's S3 backend rejects. S3 wrappers now use an empty instance root marker and resolve parents from that marker; other Arrow backends keep their existing behavior.Fixes #2015
Tested with the Arrow, spec, core, and mapping test modules (284 passed, 69 skipped, 1 xfailed) and Ruff.