GitFileSystem info call very slow

[mxmlnkn]

E.g. try this:

git clone https://github.com/mxmlnkn/ratarmount
cd ratarmount

import time
import fsspec

f = fsspec.open("git://v0.15.2@/")

t0 = time.time()
files = f.fs.listdir("/tests/", detail = False)
t1 = time.time()
print(f"Listdir took {t1 - t0:.3f} s")

t0 = time.time()
[f.fs.info(name) for name in files]
t1 = time.time()
print(f"ls -la took {t1 - t0:.3f} s")

Listdir took 0.167 s and returned 117 entries
ls -la took 17.916 s

Note that this pattern may be avoidable by using detail = True, but it nevertheless appears via the FUSE wrapper, which results in a simple ls -la taking almost 20 seconds.

Looking ta the git implementation makes it clear that it is experimental and very barebones. The info call is implemented via the ls call, resulting in a O(n^2) complexity for my use case where n is the number of files.

The same is true for many other methods such as exists. This is a very core issue. It may be easy to use for the maintainer, but this leads to pain for the users. Imho, functions that should work O(1) should only be implemented by default via functions that themselves are O(1) in time and space.

Also, it might be a good idea to cache the tree. I'm unsure how fast resolve_refish is.

[martindurant]

I suppose the assumption was that this should be "fast". For remote FSs, we have .dircache to store listings, so we could use that for git too - it is guaranteed to be read-only, after all.