Firestore: Consistent read transactions across documents/collections/queries

[tlunter]

What feature would you like to see?

When trying to read data from multiple documents, collections or queries, we'd like to perform those requests within a single transaction. The gRPC API allows for beginning a read-only transaction and passing an identifier to GetDocument, RunQuery, ListDocuments, etc.

How would you use it?

Our application makes use of state that exists in a global document per customer, and a few sub-collections. One of those sub-collections state is directly tied to the global document. Without a consistent read of both the document and sub-collection, we could end up with an inconsistent rendering of our UI. Since the current client just does BatchRead within a transaction for each .get() with no transaction identifier, we expect the values can drift. Similarly, we can't rely on listeners because it is not guaranteed that when we get listen events for both the sub-collection and document that we're at a consistent state.

The system during busy hours will have writes in a bursty behavior but will be close in time. Getting a consistent view when we go to fetch data is paramount for our customers to see the right rendering.

Exploratory work

Given the gRPC API is already stubbed into the SDK, I was able to add a top level method called readTransaction that operated very similar to the current write transaction. It first starts the transaction and provides methods like .get(), .list(), .query() that makes use of the Firestore objects. Given the existing Query object mimic/generates a similar form as the StructuredQuery object, it wasn't that difficult to use the same existing API and just use a different transform in the ReadTransaction class.

We're today using the REST API in conjunction with this SDK to solve our use case.

Thanks!

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[samtstern]

@tlunter thanks for filing this. I have heard a few more requests for "read transactions" lately so you're not alone. I'm not sure if there are particular issues with mobile that makes this infeasible or inadvisable, will defer to @wilhuff and others.

[mikelehen]

@tlunter One quick question:

we can't rely on listeners because it is not guaranteed that when we get listen events for both the sub-collection and document that we're at a consistent state.

If we were to have a Firestore.addAllSnapshotsAreConsistentListener() that let you register a listener that would fire each time all of your snapshot listeners are at a consistent point (i.e. it might be a good time for you to render your UI), would that be useful and sufficient to solve your use case?

[tlunter]

@mikelehen it's possible. In reality what we're looking for is a way to get an event for every transaction we perform on via the admin SDK. If those events for all documents/collections fired in the same snapshot that would work. The benefit of using a listener is we can make use of the SDK persistence, where a separate transaction wouldn't work.

What would the criteria be for consistency from the SDKs side?

[mikelehen]

@tlunter Any time a snapshot listener receives a new snapshot, we'd shortly thereafter raise a "all snapshots are consistent" event... but if the write that generated the snapshot also affects any other snapshot listeners you have, we'd make sure that those snapshot events got raised before the "all snapshots are consistent" event. So if an atomic write by your backend affects two listeners, those listeners would each get a snapshot event updating them, and then you'd get a "all snapshots are consistent" event. So you could just buffer snapshots between each "all snapshots are consistent" events are whatever.

Internally we already have all the means to do this (The Firestore listen protocol provides strong global consistency guarantees across all listeners already). We're just not surfacing this through the API. So this would probably be an easier API for us to expose than consistent read transactions... and as you point out, it would work offline.

[tlunter]

@mikelehen if I enable logging, should I see events come through that include multiple collections/documents? Or is that logging at your current snapshot listener level. As long as we can guarantee that the data snapshot is consistent when reading from the collection and the document, that should work.

[mikelehen]

@tlunter Our logging is more at the current snapshot listener level (there's probably a log message for every individual document update regardless of what write / listener caused it)... But when you see a log message like "Starting transaction: Apply remote event" that would roughly correspond to when we would raise the "all snapshot listeners are consistent" event... except the log message might be a bit extra noisy (we'd avoid raising the event if nothing changed).

[wilhuff]

One aspect of this conversation that went by that didn't get addressed is concerning:

In reality what we're looking for is a way to get an event for every transaction we perform on via the admin SDK

This is not possible today and we can't make this possible. Adding a read transaction API wouldn't help either.

Part of what makes the system efficient is that the server is allowed to consolidate changes into one larger update. Consider a client that's been disconnected for a week. When such a client resumes a query, instead of replaying every change that's happened over the week, the server just gives the delta to the current state.

[tlunter]

@wilhuff that definitely makes sense. We're ok with combined transactions, as long as we don't get partial data for those transactions.

[tlunter]

Hi @wilhuff, I was wondering if there was any news regarding an approach to some of the solutions discussed here?