NDB: Transactions

[chrisrossi]

Thinking out loud/brain dump, for transactions in NDB.

At the Datastore API level, a transaction is begun with a BeginTransaction call which returns a transaction id. By passing the transaction id back as a read option, read calls can present a consistent view of the database at the time the transaction was begun. Commit ends the transaction, by applying all of the mutations passed in with the CommitRequest. The Commit call is actually the only call which can change the database state, and can be called with or without a transaction. Because of this, when in a transaction, all changes to database state must be gathered into a single Commit call to end the transaction.

This explains why, in the high level documentation for Datastore, we read: "[The] consistent snapshot view also extends to reads after writes inside transactions. Unlike with most databases, queries and gets inside a Cloud Datastore transaction do not see the results of previous writes inside that transaction." The reason, of course, is that the writes haven't been sent to the back end yet—they're only sent at commit time.

The practical upshot for NDB is that if we're in a transaction, any writes need to gathered for the eventual call to Commit at the end of the transaction. For methods like Model._put(), which return the (possibly new) key for the written entity, we'll need to make calls to AllocateIds to prefetch new keys before committing.

Question: What did old NDB do, in terms of reads after writes in a transaction? Is it the same behavior as modern datastore, or did it show the writes in subsequent reads? Note, that there's possible interaction with one or both of the caching layers here. Since the semantics of caching in old NDB are not well understood yet, it's not obvious what the interaction might be.

[chrisrossi]

In NDB, there is a propagation transaction option that seems to be derived from the older implementation of datastore that it used. There is no longer any such thing in google.cloud.datastore, so I propose dropping it from NDB.

https://cloud.google.com/appengine/docs/standard/python/ndb/functions#context_options

Thoughts @andrewsg ?

[chrisrossi]

google.cloud.datastore handles transactions like this:

with client.transaction():
    do_some_stuff()

This is beautifully Pythonic. Unfortunately, it's not obvious to me how I can make the context manager cooperate with the async/tasklets stuff. The NDB way of doing this is a little more clunky:

def some_task():
    do_some_thing()

result = ndb.transaction(some_task)

My initial thought was to implement this in such a way that the original interface is still supported but a newer, more Pythonic interface is available and encouraged. Absent a brilliant brainstorm, though, there doesn't seem to be a way to do this datastore-style with async/tasklets. It might be worth revisiting post-MVP, if we're interested in improving NDB moving into the future.

[andrewsg]

Interesting. It almost sounds like the transaction concept is mostly implemented client-side, with only the "reads consistent with the time the transaction started" as a kind of snapshot functionality that's implemented server-side.

I agree that while the pythonic context manager strategy looks great, our first priority is backwards compatibility so if there's not a super easy way to support both, we should just keep track of it as an idea for the future and move on. But there are some gaps in my knowledge here; why does the ndb version have a "result" to the transaction where the datastore one does not? And what is the potential conflict with tasklets that is making it difficult to support both methods?

[chrisrossi]

Interesting. It almost sounds like the transaction concept is mostly implemented client-side, with only the "reads consistent with the time the transaction started" as a kind of snapshot functionality that's implemented server-side.

That is precisely the state of affairs, yes. It is odd. This is the first "database" I've encountered that won't let you add to a transaction incrementally. I overheard some conversation in a meeting the other week that led me to believe that Spanner works in the same way. So I guess it's a pattern for Google.

why does the ndb version have a "result" to the transaction where the datastore one does not?

Because it's calling a function that could return something. I could have made it clearer:

def some_task():
    do_some_thing()
    result = some_computation()
    return result

result = ndb.transaction(some_task)

And what is the potential conflict with tasklets that is making it difficult to support both methods?

I'll take a stab at explaining this. It might help to look at the this code here:

https://github.com/chrisrossi/google-cloud-python/blob/ndb-transaction/ndb/src/google/cloud/ndb/transaction.py#L26

Because we're in a tasklet, when we get to something that might block, like the call to begin_transaction, we can yield which lets that piece run asynchronously. Our tasklet will resume when the eventloop has a result back for whatever call was yielded.

Now, imagine that we converted that into a context manager. Everything before the call to callback() would happen in the __enter__ method, and everything after would be the __exit__ method. callback() would just be whatever code was executed in the with block that uses the context manager.

We can't make an asynchronous call to begin_transaction in the __enter__ method, because when the __enter__ method finishes, the code in the with body is going to be executed. And we can't have a future for our transaction id. We need the actual transaction id before getting that far. There's no way to delay the execution of the with block except by blocking in the __enter__ method--we can't yield out to the eventloop, because we can't turn this into a tasklet. Likewise, when exiting the context, before moving on, we need for the transaction to actually have been committed or rolled back--we can't do anything useful with a future here, either.

With the transaction function call, the function that gets called can be a tasklet, or not, and it will work just fine. It's compatible with asynchronous and synchronous functions. We can write a context manager version, but we'd have to restrict ourselves to making synchronous calls in it.

[andrewsg]

Got it. That does make sense. Thank you for the great explanation.

[chrisrossi]

Implemented in #31. Still need to implement retry.

The PR punts on "propagation"--it doesn't allow nested transactions. We may have to circle back around to that. I'm not sure.

[chrisrossi]

The "propagation" feature was actually part of the old Datastore RPC library. There doesn't seem to be any equivalent in modern Datastore. It came with a huge list of nasty sounding caveats in the original implementation, so I'm not surprised. Let's hope no one needs that. I've documented as a "No Longer Implemented".

[chrisrossi]

Done