ARROW-8766: [Python] Allow implementing filesystems in Python

[pitrou]

No description provided.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-8766

[jorisvandenbossche]

A first test case is working, using fsspec's in-memory filesystem:

In [1]: import pyarrow.parquet as pq
   ...: import pyarrow.dataset as ds
   ...:
   ...: from pyarrow.fs import PyFileSystem
   ...: from pyarrow.tests.test_fs import FSSpecHandler

In [2]: import fsspec

In [3]: memfs = fsspec.filesystem("memory")

In [4]: table = pa.table({'a': [1, 2, 3]})

In [5]: with memfs.open("test", "wb") as f:
   ...:     pq.write_table(table, f)
   ...:

In [6]: dataset = ds.dataset("test", filesystem=PyFileSystem(FSSpecHandler(memfs)))

In [7]: dataset.to_table(filter=ds.field('a') > 1).to_pandas()
Out[7]:
   a
0  2
1  3

I only have a bit trouble finding a robust way to get a file handle for an open file from the fsspec filesystem, but so that is not related to this PR ;) For the rest it was quite easy to get this working!

[pitrou]

Note to self: should instead add a separate mtime_ns argument.

[jorisvandenbossche]

The WIP attempt to create a Handler for fsspec is here (which the example I posted above is using): jorisvandenbossche@14ac33d. With that, reading files is working. Although there are some incosistencies in the file-like objects the open() method creates, so it will probably not work for all fsspec filesystems.
(writing is still TODO).

[jorisvandenbossche]

@pitrou looks all good to me. Went through the code, and tested it further with the fsspec handler, and all seems to be working nicely.
Only the open_append_stream, I am not yet fully sure how to do this (if possible). The PythonFile class might not yet support an append (mode="a") writable mode?

[jorisvandenbossche]

Does this need some doc comments? (I don't know how "public" this is, in the end it is only to be used in the python bindings I think, and there the PyFileSystem class has docstrings for python users, so that might be sufficient)

[pitrou]

It's not public as a C++ API, indeed, but if there's some stuff there that needs explaining, feel free to point it out and I'll add some comments.

[jorisvandenbossche]

Maybe just one-liner indicating this is only used to implement pyarrow.fs.PyFileSystem could be added then, but not that important

[jorisvandenbossche]

I suppose methods like this are expected to just raise the appropriate FileNotFoundError in case src does not exist?

[pitrou]

Right.

[jorisvandenbossche]

The PythonFile class might not yet support an append (mode="a") writable mode?

Recognizing append mode as a "write" mode seems to work:

--- a/python/pyarrow/io.pxi
+++ b/python/pyarrow/io.pxi
@@ -654,6 +654,8 @@ cdef class PythonFile(NativeFile):
             kind = 'w'
         elif inferred_mode.startswith('r'):
             kind = 'r'
+        elif inferred_mode.startswith('a'):
+            kind = 'w'
         else:
             raise ValueError('Invalid file mode: {0}'.format(mode))

(if that's a proper fix, can include that in my follow-up PR)

[pitrou]

Recognizing append mode as a "write" mode seems to work:

You'll have to check that writing indeed appends at the end rather than e.g. truncating.

More generally, while I originally added the append-open method, I'm not sure it will ever be useful, so in the meantime if an implementation wants to raise NotImplementedError, it's fine to me.

[jorisvandenbossche]

You'll have to check that writing indeed appends at the end rather than e.g. truncating.

At least the tests pass (I added a fsspec local filesytem handler to the list of filesytem fixtures), and from a small local test it seems to be appending.

Now, if it's not actually used within arrow at the moment (eg for enabling reading/writing parquet files or datasets with those filesystems, we won't need it), then it is also not very important to get this working.