Firestore: coll_ref.list_documents() retrieves complete documents first, then returns doc_refs?

[mikespub]

The current implementation of CollectionReference.list_documents() seems to be seriously flawed.

Reading the API documentation at https://googleapis.dev/python/firestore/latest/collection.html it returns a sequence of doc_refs, which sounds like a good way to get (or count) all document references in a collection, without retrieving any unneeded data.
[Note: this is a bit like what a keys_only() query would do in Datastore mode, which doesn't really seem to have an equivalent select().stream() in native mode -> actually there is an equivalent, it's just not well documented: select([]) corresponds to select(["__name__"])]

So you would expect list_documents() to only retrieve doc_refs from Firestore in the back-end as well.

But what seems to happen in practice is that all document data are retrieved completely (with all fields) by the API, and then _item_to_document_ref() returns a doc_ref based on the document.

So this is actually the worst possible combination: you loose a lot of time/bandwidth retrieving all the document data and then simply discard it.

I'm not sure how this is implemented in other languages, but either list_documents() should continue returning only doc_refs but then only actually retrieve those doc_refs in the back-end. Or it if that's not technically possible for some reason, it should return actual docs, possibly with selectable field_paths like query().

Thanks,

Mike.

[crwilcox]

I walked over this code path. There are not multiple calls happening. The firestore api call 'list documents is called and then the iterator transforms the response to a document references from a a 'google.cloud.firestore_v1.types.document.Document'. This doesn't require additional data transfer, it is more about providing a different interface as a user of the python library. I think the primary concern is that we transport fields but you aren't concerned with field data, just the document id in the collection.

Focusing on avoiding returning unwanted data, the RPC for listDocuments supports a field mask, which if set, will cause only matching fields to be returned. You could get 'close' to a keysonly type response

For example, modifying the code within list_documents, we can get the following, which would emulate keysonly like behavior.

       request, kwargs = self._prep_list_documents(page_size, retry, timeout)
        
        mask = {"field_paths": None}
        request['mask'] = mask
        iterator = self._client._firestore_api.list_documents(
            request=request, metadata=self._client._rpc_metadata, **kwargs,
        )
        return (_item_to_document_ref(self, i) for i in iterator)

We would need to talk more on the client library team about how to add on to our API for something like this.

[crwilcox]

Updated above, sending None or "" as the value for field_paths will result in no fields being sent.

[crwilcox]

After taking a bit a time away from this, I think with the way we have eposed list_documents in python, returning references, not snapshots, we can actually always mask the data to limit wire xfer. PR to demonstrate the fix.

[crwilcox]

PR is merged. This will cut down on data transfer. I do want to note, this doesn't change the 'reads' from a cloud side, but it will keep the payloads much smaller 👍

[mikespub]

Thanks