ENH: add fsspec support #34266

martindurant · 2020-05-19T19:59:53Z

Supersedes #33549
closes #33452

martindurant · 2020-05-19T20:11:59Z

I see some parquet changes here, will need a bit of work...

TomAugspurger

We'll want a whatsnew in 1.1.0.rst. You can add a subsection to enhancements for this.

TomAugspurger · 2020-05-19T20:11:49Z

environment.yml

@@ -98,7 +98,7 @@ dependencies:

  - pyqt>=5.9.2  # pandas.read_clipboard
  - pytables>=3.4.2  # pandas.read_hdf, DataFrame.to_hdf
-  - s3fs  # pandas.read_csv... when using 's3://...' path
+  - s3fs  # pandas.read_csv... when using 's3://...' path (also brings in fsspec)


I think explicitly list fsspec here, if we're import it.

What should we set the minimum supported version to? We'll add this to a few places in the library (compat._optional, docs, ...).

Can update this, and the minimum versions?

TomAugspurger · 2020-05-19T20:13:53Z

pandas/io/common.py

+    try:
+        import fsspec  # noqa: F401
+
+        return isinstance(url, str) and ("::" in url or "://" in url)


Should this also check if there's an fsspec-compatible implementation for that protocol? What's the behavior / error message if you have fsspec installed but not s3fs and do read_csv("s3://...")?

fsspec gives specific error messages for known protocols that are not available due to missing dependency (s3fs would be such a one), and a different message is the protocol is completely unknown.

Does that error surface to the user? Or does pandas swallow it somewhere along the way?

TomAugspurger · 2020-05-19T20:14:36Z

pandas/io/common.py

 def get_filepath_or_buffer(
    filepath_or_buffer: FilePathOrBuffer,
    encoding: Optional[str] = None,
    compression: Optional[str] = None,
    mode: Optional[str] = None,
+    **storage_options,


Add a type? Dict[str, Any] I think?

Does Dict[str, Any] work?

TomAugspurger · 2020-05-19T20:14:43Z

pandas/io/common.py

@@ -175,6 +175,7 @@ def get_filepath_or_buffer(
    compression : {{'gzip', 'bz2', 'zip', 'xz', None}}, optional
    encoding : the encoding to use to decode bytes, default is 'utf-8'
    mode : str, optional
+    storage_options: passed on to fsspec, if using it


Suggested change

storage_options: passed on to fsspec, if using it

**storage_options : dict, optional

passed on to fsspec.open, if using it.

add a versionadded tag (1.1)

Just FYI, this isn't in the public API.

At some point we'll add this keyword to all of our IO routines, which would benefit from the versionadded. But that can be a separate PR.

TomAugspurger · 2020-05-19T20:16:11Z

pandas/tests/io/test_fsspec.py

+
+def test_to_csv(cleared_fs):
+    df1.to_csv("memory://test/test.csv", index=True)
+    gc.collect()  # pandas does not explicitly close file buffers


(why) is this necessary?

I'll double check - maybe this was only need by the previous version of fsspec

TomAugspurger · 2020-05-19T20:16:38Z

pandas/tests/io/test_fsspec.py

+
+
+@pytest.fixture
+@td.skip_if_installed("fsspec")


Why is this skipped if fsspec is installed?

Ah, should be the opposite - but why did the tests run for me??

pandas/tests/io/test_gcs.py

pandas/tests/io/test_pickle.py

TomAugspurger · 2020-05-19T20:18:07Z

pandas/tests/io/test_s3.py

@@ -1,25 +0,0 @@
-from io import BytesIO


Again: ideally we keep the old tests, aside from the is_s3_url one.

pep8speaks · 2020-05-20T16:24:34Z

Hello @martindurant! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-06-19 18:15:56 UTC

martindurant · 2020-05-21T20:49:43Z

(NB: some failures are due to v0.7.4 of fsspec only being on conda-forge)

martindurant · 2020-05-21T20:51:09Z

but I will need some help with

pandas/tests/io/test_fsspec.py:18: error: Item "None" of "Optional[str]" has no attribute "encode"

(i.e., I know that, given the input, the output is str and not None, but how do I tell mypy that)

jreback · 2020-05-25T23:01:50Z

pandas/io/common.py

@@ -175,6 +175,7 @@ def get_filepath_or_buffer(
    compression : {{'gzip', 'bz2', 'zip', 'xz', None}}, optional
    encoding : the encoding to use to decode bytes, default is 'utf-8'
    mode : str, optional
+    storage_options: passed on to fsspec, if using it


add a versionadded tag (1.1)

jreback · 2020-05-25T23:03:53Z

pandas/tests/io/test_fsspec.py

+from pandas.util import _test_decorators as td
+
+from pandas.io.common import is_fsspec_url
+


everything in this module is safe if fsspec is not available?

ok to simply skip the module

you mean fsspec = pytest.importorskip("fsspec")?
test_is_fsspec_url should be moved to test_common in that case.

pandas/tests/io/test_parquet.py

TomAugspurger

Linting error: https://github.com/pandas-dev/pandas/pull/34266/checks?check_run_id=697447968#step:11:10

Can you run mypy on this locally?

CI failures at https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=35892&view=logs&j=98c84d08-34d8-513e-80be-2c581992dd5a&t=f75024c1-aff0-57fb-4f0e-35923b654e09&l=208. Do we need to specify a minimum version of fsspec?

cc @gfyoung if you have a chance to review.

TomAugspurger · 2020-05-26T13:35:56Z

pandas/io/common.py

@@ -175,6 +175,7 @@ def get_filepath_or_buffer(
    compression : {{'gzip', 'bz2', 'zip', 'xz', None}}, optional
    encoding : the encoding to use to decode bytes, default is 'utf-8'
    mode : str, optional
+    storage_options: passed on to fsspec, if using it


Just FYI, this isn't in the public API.

At some point we'll add this keyword to all of our IO routines, which would benefit from the versionadded. But that can be a separate PR.

TomAugspurger · 2020-05-26T13:36:25Z

pandas/io/common.py

+        file_obj = fsspec.open(
+            filepath_or_buffer, mode=mode or "rb", **storage_options
+        ).open()
+        # TODO: both fsspec and pandas handle compression and encoding


What would resolve the TODO here? To not handle compression or encoding in pandas? Can you update the comment to indicate that?

Given that pandas must still handle compression and encoding for local and http, that code will not be deprecated. Therefore, I think it's fine that we don't advertise the fact that fsspec can do that part too, and open everything on the backend as "rb"/"wb", uncompressed. The TODO would be resolved if at some point we decided that fsspec should handle all file ops, which is not likely in the near term.

Agreed, so I think the TODO can be removed.

TomAugspurger · 2020-05-26T21:28:15Z

environment.yml

@@ -98,7 +98,7 @@ dependencies:

  - pyqt>=5.9.2  # pandas.read_clipboard
  - pytables>=3.4.2  # pandas.read_hdf, DataFrame.to_hdf
-  - s3fs  # pandas.read_csv... when using 's3://...' path
+  - s3fs  # pandas.read_csv... when using 's3://...' path (also brings in fsspec)


Can update this, and the minimum versions?

TomAugspurger · 2020-05-26T21:28:56Z

pandas/io/common.py

 def get_filepath_or_buffer(
    filepath_or_buffer: FilePathOrBuffer,
    encoding: Optional[str] = None,
    compression: Optional[str] = None,
    mode: Optional[str] = None,
+    **storage_options,


Does Dict[str, Any] work?

TomAugspurger · 2020-05-26T21:31:44Z

pandas/io/parquet.py

@@ -107,6 +102,11 @@ def write(
        # write_to_dataset does not support a file-like object when
        # a directory path is used, so just pass the path string.
        if partition_cols is not None:
+            if is_fsspec_url(path) and "filesystem" not in kwargs:


Can you leave a comment explaining this "filesystem" not in kwargs check? It's not obvious to me why it's needed.

In fsspec, you can specify the exact protocol you would like beyond that inferred from the URL. Given that we don't pass storage_options through yet, perhaps this gives more flexibility than required and I can remove it.

Sorry, edit on that: this is the filesystem parameter (i.e., an actual instance) to pyarrow. I have no idea if people might currently be using that.

Ah, you're saying the user could pass a filesystem like

df.to_parquet(..., filesystem=filesystem)

That certainly seems possible. Could you ensure that we have a test for that?

pandas/tests/io/test_gcs.py

pandas/io/parquet.py

martindurant · 2020-05-28T13:29:53Z

Linting error: https://github.com/pandas-dev/pandas/pull/34266/checks?check_run_id=697447968#step:11:10

I need help with this. The return of to_csv is Optional[str], but for the input I gave it, it must be str, so that encode is a legal thing to do. How do I specify that? Wrapping it in str() does not seem right.

CI failures

There runs are using an older fsspec from conda defaults. I can ask the conda team to update the version, but we should be certain that this PR doesn't require any further changes in fsspec.

TomAugspurger

Can you add a new subsection to "Enhancements" to doc/source/whatsnew/v1.1.0.rst`?

environment.yml

pandas/io/common.py

TomAugspurger · 2020-05-28T13:40:14Z

pandas/io/common.py

+        file_obj = fsspec.open(
+            filepath_or_buffer, mode=mode or "rb", **storage_options
+        ).open()
+        # TODO: both fsspec and pandas handle compression and encoding


Agreed, so I think the TODO can be removed.

TomAugspurger · 2020-05-28T13:41:29Z

pandas/io/parquet.py

@@ -107,6 +102,11 @@ def write(
        # write_to_dataset does not support a file-like object when
        # a directory path is used, so just pass the path string.
        if partition_cols is not None:
+            if is_fsspec_url(path) and "filesystem" not in kwargs:


Ah, you're saying the user could pass a filesystem like

df.to_parquet(..., filesystem=filesystem)

That certainly seems possible. Could you ensure that we have a test for that?

pandas/tests/io/test_fsspec.py

pandas/io/parquet.py

pandas/tests/io/test_parquet.py

Added test with filesystem= Added whatsnew Changed imports, updated comments

doc/source/whatsnew/v1.1.0.rst

TomAugspurger · 2020-05-28T20:54:46Z

doc/source/whatsnew/v1.1.0.rst

+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For reading and writing to filesystems other than local and reading from HTTP(S),
+the optional dependency ``fsspec`` will be used to dispatch operations. This will give unchanged


Link to the original issue at the end of the first sentence.

Also not fixed yet.

pandas/compat/_optional.py

martindurant · 2020-05-29T17:39:35Z

Note: fsspec 0.7.4 will be up on conda defaults soon. At that point I'll remove the WIP tag.

TomAugspurger

I don't know that we'll need to wait for the fsspec to be updated on defaults. You can update the various ci/deps/*.yml files to include fsspec>=0.7.4 wherever we include s3fs or gcsfs already. That should force it to use conda-forge till defaults is updated.

Can you also run scripts/generate_pip_deps_from_conda.py?

doc/source/whatsnew/v1.1.0.rst

TomAugspurger · 2020-05-29T18:17:20Z

doc/source/whatsnew/v1.1.0.rst

+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For reading and writing to filesystems other than local and reading from HTTP(S),
+the optional dependency ``fsspec`` will be used to dispatch operations. This will give unchanged


Also not fixed yet.

martindurant · 2020-05-29T18:28:56Z

Those two points were fixed on my local branch, not yet pushed.

doc/source/whatsnew/v1.1.0.rst

TomAugspurger · 2020-06-12T21:08:59Z

I think one of the test failures in https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=37143&view=logs&j=bef1c175-2c1b-51ae-044a-2437c76fc339&t=770e7bb1-09f5-5ebf-b63b-578d2906aac9 is real.

    @tm.network
    @td.skip_if_no("pyarrow")
    def test_parquet_read_from_url(self, df_compat):
        url = (
            "https://raw.githubusercontent.com/pandas-dev/pandas/"
            "master/pandas/tests/io/data/parquet/simple.parquet"
        )
>       df = pd.read_parquet(url)

pandas/tests/io/test_parquet.py:584: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/io/parquet.py:316: in read_parquet
    return impl.read(path, columns=columns, **kwargs)
pandas/io/parquet.py:131: in read
    import_optional_dependency("fsspec")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Should we not be ussing fsspec when reading from a URL?

martindurant · 2020-06-12T21:11:11Z

Should we not be using fsspec when reading from a URL?

In order not to change current behaviour, http(s) is excepted in get buffer, because it was not handled by fsspec previously.

doc/source/whatsnew/v1.1.0.rst

TomAugspurger · 2020-06-16T14:59:17Z

@jorisvandenbossche are you good with this now that #34500 is in (once the merge issue in the whatsnew is sorted out)?

jorisvandenbossche · 2020-06-18T07:35:21Z

pandas/io/parquet.py

-            self.api.parquet.write_table(
-                table, file_obj_or_path, compression=compression, **kwargs
-            )
+            self.api.parquet.write_table(table, path, compression=compression, **kwargs)


It might have been discussed before in this PR, but why this change from file_obj_or_path to path ? Because with this change, file_obj_or_path is basically never used?

You know what, this can be made simpler...

The reason is, pyarrow can take a filesystem+path and do the right thing, and for the partitioning case, this is the only way. I think we can deal without opening the file in the pandas code at all. I'll push that suggestion, and add comments for what the blocks do to be clearer.

The new code above certainly looks fine, I am only wondering if there were things that get_filepath_or_buffer was doing before we might now be missing.

It gets a buffer for urls, but I don't think this actually works for writing.
In addition, it also seems to do a os.path.expanduser on strings. I suppose this is not tested (and not sure many people would rely on that), but that might lead to a regression.

You want to add expanduser to the function? I wouldn't bother going through get_filepath_or_buffer just for this, and as you say, it isn't tested or documented anywhere. In fact, test may be a little tricky for the various OSs, I'm not sure you can guarantee that "~/afile"-> "${HOME}/afile".

Confirmed that it works on master

In [2]: df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}) In [3]: df.to_parquet("~/test-4.parquet")

and raises on this branch.

In [3]: df.to_parquet("~/test-5.parquet") --------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <ipython-input-3-c7dcbbaae894> in <module> ----> 1 df.to_parquet("~/test-5.parquet") ~/sandbox/pandas/pandas/util/_decorators.py in wrapper(*args, **kwargs) 197 else: 198 kwargs[new_arg_name] = new_arg_value --> 199 return func(*args, **kwargs) 200 201 return cast(F, wrapper) ~/sandbox/pandas/pandas/core/frame.py in to_parquet(self, path, engine, compression, index, partition_cols, **kwargs) 2323 index=index, 2324 partition_cols=partition_cols, -> 2325 **kwargs, 2326 ) 2327 ~/sandbox/pandas/pandas/io/parquet.py in to_parquet(df, path, engine, compression, index, partition_cols, **kwargs) 266 index=index, 267 partition_cols=partition_cols, --> 268 **kwargs, 269 ) 270 ~/sandbox/pandas/pandas/io/parquet.py in write(self, df, path, compression, index, partition_cols, **kwargs) 118 else: 119 # write to single output file --> 120 self.api.parquet.write_table(table, path, compression=compression, **kwargs) 121 122 def read(self, path, columns=None, **kwargs): ~/Envs/pandas-dev/lib/python3.7/site-packages/pyarrow/parquet.py in write_table(table, where, row_group_size, version, use_dictionary, compression, write_statistics, use_deprecated_int96_timestamps, coerce_timestamps, allow_truncated_timestamps, data_page_size, flavor, filesystem, compression_level, **kwargs) 1341 use_deprecated_int96_timestamps=use_int96, 1342 compression_level=compression_level, -> 1343 **kwargs) as writer: 1344 writer.write_table(table, row_group_size=row_group_size) 1345 except Exception: ~/Envs/pandas-dev/lib/python3.7/site-packages/pyarrow/parquet.py in __init__(self, where, schema, filesystem, flavor, version, use_dictionary, compression, write_statistics, use_deprecated_int96_timestamps, compression_level, **options) 434 filesystem, path = resolve_filesystem_and_path(where, filesystem) 435 if filesystem is not None: --> 436 sink = self.file_handle = filesystem.open(path, 'wb') 437 else: 438 sink = where ~/Envs/pandas-dev/lib/python3.7/site-packages/pyarrow/filesystem.py in open(self, path, mode) 242 """ 243 path = _stringify_path(path) --> 244 return open(path, mode=mode) 245 246 @property FileNotFoundError: [Errno 2] No such file or directory: '~/test-5.parquet'

pandas consistently expands ~ to the homedir, so we'll need to ensure we don't regress on this.

but how do we test this reliably?

We should have similar tests for csv.

We test the common utility:

pandas/pandas/tests/io/test_common.py

Lines 55 to 68 in 80ba4c4

def test_expand_user(self):

filename = "~/sometest"

expanded_name = icom._expand_user(filename)

assert expanded_name != filename

assert os.path.isabs(expanded_name)

assert os.path.expanduser(filename) == expanded_name

def test_expand_user_normal_path(self):

filename = "/somefolder/sometest"

expanded_name = icom._expand_user(filename)

assert expanded_name == filename

assert os.path.expanduser(filename) == expanded_name

and then probaby suppose that all IO methods use those common utilities.

A little weak...
I'll put in something. By the way, I expect that expanding the home dir didn't work before either when partitioning.

jorisvandenbossche · 2020-06-18T07:39:20Z

pandas/io/parquet.py

-        parquet_ds = self.api.parquet.ParquetDataset(
-            path, filesystem=get_fs_for_path(path), **kwargs
-        )
+        if is_fsspec_url(path) and "filesystem" not in kwargs:


Can you check this comment?
(I will try if I can write a test that would catch it)

TomAugspurger · 2020-06-18T16:01:37Z

pandas/tests/io/json/test_pandas.py

-            obj.key for obj in s3_resource.Bucket("pandas-test").objects.all()
-        )
+        timeout = 5
+        while True:


Hmm, this is concerning. Do you know what's causing it? I would think that everything is synchronous.

I don't know why... S3 is not supposed to be immediately consistent, maybe the botocore caches or moto doesn't update it's index immediately, or something like that.

martindurant · 2020-06-18T17:01:12Z

(travis is passing, but has not updated in the list above, at least for me)

martindurant · 2020-06-22T20:52:21Z

@jorisvandenbossche , anything else?

jorisvandenbossche · 2020-06-23T13:06:49Z

pandas/io/parquet.py

-        parquet_ds = self.api.parquet.ParquetDataset(
-            path, filesystem=get_fs_for_path(path), **kwargs
-        )
+        if is_fsspec_url(path) and "filesystem" not in kwargs:


ping for this one

jorisvandenbossche · 2020-06-23T13:09:20Z

pandas/io/parquet.py

+        else:
+            fs = kwargs.pop("filesystem", None)
+            should_close = False
+            path = _expand_user(path)


No blocking comment (could also be as a follow-up), but for my understanding (and could maybe use a code comment): what was the reason again that this whole is_fsspec_url(path): .. else: .. block cannot be handled by get_filepath_or_buffer ?
(which would eg handle the _expand_user as well)

That we are not getting an open file, we are passing a path and filesystem to arrow for it to handle. This is important for the partitioning case, but works for the simple case too. Wasn't it your suggestion?

Wasn't it your suggestion?

I don't think so, since this was already there from the beginning of the PR before my first review ;) Not that it matters much.

Anyway, to answer my own question (I think): on a second look, this is simply because get_filepath_or_buffer doesn't return a filesystem (and which is indeed needed here, as you indicated). I was just confused about that for a moment.

jorisvandenbossche · 2020-06-23T13:43:42Z

Thanks!

martindurant · 2020-06-23T13:51:47Z

Phew :)

TomAugspurger · 2020-06-23T13:57:28Z

Thanks!

Adapt a mock for the changed implementation at pandas, see pandas-dev/pandas#34266. Closes #51.

Julian de Ruiter and others added 2 commits April 14, 2020 22:37

Add remote file io using fsspec.

94e717f

Attempt refactor and clean

fd7e072

TomAugspurger mentioned this pull request May 19, 2020

[WIP] Add remote file io using fsspec. #33549

Closed

5 tasks

TomAugspurger reviewed May 19, 2020

View reviewed changes

Merge branch 'master' into feature/add-fsspec-support

302ba13

Martin Durant added 4 commits May 21, 2020 14:37

readd and adapt s3/gcs tests

9e6d3b2

remove gc from test

4564c8d

Simpler is_fsspec

0654537

add test

8d45cbb

jreback requested changes May 25, 2020

View reviewed changes

jreback added IO Data IO issues that don't fit into a more specific label IO Google labels May 25, 2020

TomAugspurger reviewed May 26, 2020

View reviewed changes

Answered most points

006e736

TomAugspurger reviewed May 28, 2020

View reviewed changes

Martin Durant added 2 commits May 28, 2020 14:47

Implemented suggestions

724ebd8

Added test with filesystem= Added whatsnew Changed imports, updated comments

lint

9da1689

TomAugspurger reviewed May 28, 2020

View reviewed changes

Add versions info

a595411

TomAugspurger reviewed May 29, 2020

View reviewed changes

Update some deps

6dd1e92

martindurant commented May 29, 2020

View reviewed changes

doc/source/whatsnew/v1.1.0.rst Outdated Show resolved Hide resolved

issue link syntax

6e13df7

Merge branch 'master' into feature/add-fsspec-support

b3e2cd2

TomAugspurger reviewed Jun 16, 2020

View reviewed changes

doc/source/whatsnew/v1.1.0.rst Outdated Show resolved Hide resolved

redo whatsnew

4977a00

jorisvandenbossche reviewed Jun 18, 2020

View reviewed changes

Martin Durant added 2 commits June 18, 2020 09:44

simplify parquet write

29a9785

Retry S3 file probe with timeout, in test_to_s3

565031b

TomAugspurger reviewed Jun 18, 2020

View reviewed changes

TomAugspurger mentioned this pull request Jun 18, 2020

[WIP]REGR: Fixed reading from public S3 buckets with credentials #34866

Closed

Martin Durant added 2 commits June 19, 2020 13:08

expand user in non-fsspec paths for parquet; add test for this

606ce11

reorder imports!

60b80a6

This was referenced Jun 19, 2020

BUG/TST: Read from Public s3 Bucket Without Creds #34877

Merged

BUG: to_parquet write partitioned DataFrame to local filesystem for other filesystems (e.g. S3) #34841

Closed

kylase mentioned this pull request Jun 20, 2020

Infer filesystem from path when writing a partitioned DataFrame to remote file systems using pyarrow #34842

Closed

5 tasks

martindurant mentioned this pull request Jun 21, 2020

Setting a custom endpoint_url fsspec/s3fs#120

Closed

jorisvandenbossche reviewed Jun 23, 2020

View reviewed changes

jorisvandenbossche approved these changes Jun 23, 2020

View reviewed changes

jorisvandenbossche merged commit 38f4af9 into pandas-dev:master Jun 23, 2020

martindurant deleted the feature/add-fsspec-support branch June 23, 2020 13:51

alecglassford mentioned this pull request Jul 30, 2020

DOC: s3fs is required when using read_csv with an S3 URI #35206

Open

This was referenced Sep 25, 2020

tests failing with ValueError: I/O operation on closed file gerritholl/fogtools#51

Closed

Tests failing with ValueError: I/O operation on closed file gerritholl/fogtools#52

Closed

gerritholl added a commit to gerritholl/fogtools that referenced this pull request Sep 25, 2020

Adapt mock for changed pandas implementation

8c146e9

Adapt a mock for the changed implementation at pandas, see pandas-dev/pandas#34266. Closes #51.

	storage_options: passed on to fsspec, if using it
	**storage_options : dict, optional
	passed on to fsspec.open, if using it.

		from pandas.util import _test_decorators as td

		from pandas.io.common import is_fsspec_url

	def test_expand_user(self):
	filename = "~/sometest"
	expanded_name = icom._expand_user(filename)

	assert expanded_name != filename
	assert os.path.isabs(expanded_name)
	assert os.path.expanduser(filename) == expanded_name

	def test_expand_user_normal_path(self):
	filename = "/somefolder/sometest"
	expanded_name = icom._expand_user(filename)

	assert expanded_name == filename
	assert os.path.expanduser(filename) == expanded_name



		@pytest.fixture
		@td.skip_if_installed("fsspec")

ENH: add fsspec support #34266

ENH: add fsspec support #34266

Conversation

martindurant commented May 19, 2020 • edited

martindurant commented May 19, 2020

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pep8speaks commented May 20, 2020 • edited

Comment last updated at 2020-06-19 18:15:56 UTC

martindurant commented May 21, 2020

martindurant commented May 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martindurant commented May 28, 2020

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martindurant commented May 29, 2020

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martindurant commented May 29, 2020

TomAugspurger commented Jun 12, 2020

martindurant commented Jun 12, 2020

TomAugspurger commented Jun 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martindurant commented Jun 18, 2020

martindurant commented Jun 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Jun 23, 2020

martindurant commented Jun 23, 2020

TomAugspurger commented Jun 23, 2020

martindurant commented May 19, 2020 •

edited

pep8speaks commented May 20, 2020 •

edited