db.table_names() comes up empty on GCS

[wjones127]

From a user report:

>>> import lancedb
>>> db = lancedb.connect("gs://xxxx/lance")
>>> db.table_names()
[]
>>> db.open_table("meta")
LanceTable(meta)
>>> t = db.open_table("meta")
>>> len(t.to_pandas())
11
>>>

We use PyArrow filesystems under the hood to introspect the bucket:

lancedb/python/lancedb/db.py

Lines 296 to 298 in b085d9a

	paths = filesystem.get_file_info(
	fs.FileSelector(get_uri_location(self.uri))
	)

However, PyArrow's GCSFileSystem is known to report common prefixes as empty if there isn't a "directory marker": apache/arrow#32403 object-store-rs doesn't create these directory markers.

I suspect that is the underlying issue. We could either create the directory markers or we could use some other mechanism to list the bucket.

[sergiocorreia]

FWIW, I get the same error on a Windows installation (Win10+Py3.12), and for a different reason

Given a path C:/foo/bar.lancedb, get_uri_location() returns /foo/bar.lancedb which then will obviously fail when passed to filesystem.get_file_info() because it's not a valid path or file.

If instead of doing

paths = filesystem.get_file_info( 
     fs.FileSelector(get_uri_location(self.uri)) 
 )

We do:

paths = filesystem.get_file_info( 
     fs.FileSelector(self.uri)
 )

Then the code works correctly, on Windows.

[westonpace]

The table_names feature should be fixed in #1059