-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Alternator: support "transactions" feature #5064
Comments
In https://docs.google.com/document/d/1i4yjF5OSAazAY_-T8CBce9-2ykW4twx_E_Nt2zDoOVs/edit#heading=h.qmq6kez54pdq I mentioned that:
This may be a good way to have full support for this feature - albeit somewhat ugly and slow - quickly, without huge investments in Scylla infrastructures to support transactions in CQL. Amazon have good design document https://github.com/awslabs/dynamodb-transactions/blob/master/DESIGN.md on how they implemented full-fledged multi-item cross-table transactions over pre-existing DynamoDB features, and I'm optimistic we can do the same while emulating their new transaction API. I'm optimistic that this can be done in 1-2 weeks instead of the original 1-2 month estimate (which itself was optimistic) for a from-scratch implementation of this feature. |
I summarized the option of providing transactions over existing features in a new design document https://docs.google.com/document/d/1d0RmuWPkGmoKPXwJ2mY6HLhf40otuMvfOKNUYcfn2Cg/edit#. There are good news and bad news. The good news is that this I think this technique can work. The bad news is that the performance will be bad (e.g., a transactional write will be 7-9 times more expensive than a regular LWT write, while the ratio is 2 in DynamoDB), and some edge cases involving races of transactional and non-transactional operations, or interaction of transactions and GSI or streams, will be tricker to solve. |
Our LWT implementation already supports conditional batches: "The entire conditional batch has an isolated view of the database and is executed using all-or-nothing principle.". We could use this existing capability to implement the DynamoDB Transactions feature with one important limitation - that all operations in a transaction must be on the same partition. Although this limited feature is not as powerful as the full DynamoDB feature (that can cross partitions and even tables), it will probably be good enough for some users - and we should hopefully be able to provide this subset quickly and most importantly - efficiently (it will have the same performance as the current CAS-based writes). |
Amazon's FAST '19 presentation by Doug Terry on how DynamoDB transactions were implemented: https://www.usenix.org/conference/fast19/presentation/terry Some additional interesting links:
|
Is there a core scylladb issue describing the requirements to support this issue? I can't find anything that requests cross-table transaction support in the issue list - perhaps I just can't think of the correct search phrase? |
I don't think there is such an issue open. I agree that it would also be nice to have multi-item transactions in CQL and that whatever core feature we'll end up implementing may also be used in CQL to implement a similar transaction feature. The reason why I opened this issue specifically for Alternator (ScyllaDB's DynamoDB-compatible API) is that it is a requirement for full DynamoDB compatibility, and that feature has known behavior (defined by Amazon). For the CQL feature it would be a completely new feature and we'll need to design it and determine how exactly it should behave. Or maybe it should be compatible with the planned Cassandra feature - https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15%3A+General+Purpose+Transactions (a.k.a. "Accord"). UPDATE: CEP-15: General Purpose Transactions was officially accepted in Cassandra, there is an issue - https://issues.apache.org/jira/browse/CASSANDRA-17092, and will probably miss Cassandra 5.0 and be in Cassandra 5.1. So to maintain Cassandra compatibility, the CQL feature was already defined for us. |
Thanks for the background. From what you say, it seems to me that it would
now be useful to open a separate issue for multi-item transaction support
in CQL. Such an issue would be a good place to capture requirements like
supporting this Alternator extension but also provide a place for
discussing the CQL specific design aspects. I think that such a feature is
very desirable in its own right and the probability of starting to really
tackle issues such as this one will be much higher if we really recognize
the benefits it could bring to the whole product.
…On Sun, 29 Jan 2023 at 09:26, nyh ***@***.***> wrote:
Is there a core scylladb issue describing the requirements to support this
issue?
I don't think there is such an issue open. I agree that it would also be
nice to have multi-item transactions in CQL and that whatever core feature
we'll end up implementing may also be used in CQL to implement a similar
transaction feature. The reason why I opened this issue specifically for
Alternator (ScyllaDB's DynamoDB-compatible API) is that it is a requirement
for full DynamoDB compatibility, and that feature has known behavior
(defined by Amazon). For the CQL feature it would be a completely new
feature and we'll need to design it and determine how exactly it should
behave. Or maybe it should be compatible with the planned Cassandra feature
-
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15%3A+General+Purpose+Transactions
—
Reply to this email directly, view it on GitHub
<#5064 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADWOAUPPSHHGG2XPNC6PJTWUYZT3ANCNFSM4IYIEL4Q>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Originally, DynamoDB transactions were limited to contain 25 read or write requests. Starting in September 2022, this limit was increased to 100 operations in a single transaction. |
"Transactions" are a new DynamoDB feature (announced in November 2018) which allows atomic (all-or-nothing) updates to one or more tables in the same region. This feature is unrelated to the older single-item conditional updates feature - which in Scylla are called lightweight transactions, and which we already support but without the correct isolation guarantees (see issue #5054).
See https://docs.google.com/document/d/1i4yjF5OSAazAY_-T8CBce9-2ykW4twx_E_Nt2zDoOVs/edit# (section "Transactions") for some implementation ideas.
The new API requests for this feature:
TransactWriteItems
andTransactGetItems
.The text was updated successfully, but these errors were encountered: