ARROW-7311: [Python] Return filesystem and path from URI

[kszucs]

This should supersede #5977

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7311

[fhoering]

Ok. Thanks for the PR. Sorry for the delay on the other PR. I will close it.

What about the ergonomics of this new API in Python ? Do you think we can just wrap it to fit the old API ? Or replace the methods by shorter versions like ls, read, write ?

[kszucs]

@fhoering the unix like filesystem methods might not fit every backend, like S3 or HDFS. I think we'll need to deprecate the old filesystem API in favor of the new one.

cc @pitrou

[fhoering]

OK. Obviously no need to do exactly unix like but at least use a concise format. When thinking of filesystem everybody somehow thinks of dos/unix filesystem command and S3/HDFS file system shells also wrap to this. So imo the most intuitive is just stick to that.

I also would like to upstream additional helper methods from https://github.com/dask/hdfs3. Already wrapped some of them in my project (criteo/cluster-pack#23, https://github.com/criteo/cluster-pack/blob/master/cluster_pack/filesystem.py). My goal is to make all this disappear and use only pyarrow's filesystem.

So I will create a jira ticket and document what I think could be improved. I can also work on that if it is agreed and PR will be accepted.

[pitrou]

+1

[kszucs]

@fhoering sounds good to me, please create a jira ticket.

[jorisvandenbossche]

Thanks @kszucs !