Alternator: CreateTable + Tags should be atomic #6391
Comments
psarna
pushed a commit
that referenced
this issue
May 7, 2020
The "current compatibility with DynamoDB" section in alternator.md is where we should list very briefly our state of compatibility - it's not the right place to explain implementation details or track obscure bugs. I've significantly shortened the "Tags" section because, in brief, we do fully support tags and should say that we do. I moved the two bugs mentioned in the text into the bug tracker: Refs #6389 Refs #6391 Signed-off-by: Nadav Har'El <nyh@scylladb.com> Message-Id: <20200507125022.22608-1-nyh@scylladb.com>
denesb
pushed a commit
that referenced
this issue
Feb 10, 2022
This patch adds a "--raft" option to test/alternator/run to enable the
experimental Raft-based schema changes ("--experimental-features=raft")
when running Scylla for the tests. This is the same option we added to
test/cql-pytest/run in a previous patch.
Note that we still don't have any Alternator tests that pass or fail
differently in these two modes - these will probably come later as we
fix issues #9868 and #6391. But in order to work on fixing those issues
we need to be able to run the tests in Raft mode.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20220209123144.321344-1-nyh@scylladb.com>
denesb
pushed a commit
that referenced
this issue
Feb 14, 2022
We add reproducing tests for two known Alternator issues, #6391 and #9868, which involve the non-atomicity of table creation. Creating a table currently involves multiple steps - creating a keyspace, a table, materialized views, and tags. If some of these steps succeed and some fail, we get an InternalServerError and potentially leave behind some half-built table. Both issues will be solved by making better use of the new Raft-based capabilities of making multiple modifications to the schema atomically, but this patch doesn't fix the problem - it just proves it exist. The new tests involve two threads - one repeatedly trying to create a table with a GSI or with tags - and the other thread repeatedly trying to delete the same table under its feet. Both bugs are reproduced almost immediately. Note that like all test/alternator tests, the new tests are usually run on just one node. So when we fix the bug and these tests begin to pass, it will not be a proof that concurrent schema modification works safely on *different* nodes. To prove that, we will also need a multi-node test. However, this test can prove that we used Raft-based schema modification correctly - and if we assume that the Raft-based schema modification feature is itself correct, then we can be sure that CreateTable will be correct also across multiple nodes. Although it won't hurt to check it directly. Refs #6391 Refs #9868 Signed-off-by: Nadav Har'El <nyh@scylladb.com> Message-Id: <20220207223100.207074-1-nyh@scylladb.com>
nyh
added a commit
to nyh/scylla
that referenced
this issue
Feb 14, 2022
We add reproducing tests for two known Alternator issues, scylladb#6391 and scylladb#9868, which involve the non-atomicity of table creation. Creating a table currently involves multiple steps - creating a keyspace, a table, materialized views, and tags. If some of these steps succeed and some fail, we get an InternalServerError and potentially leave behind some half-built table. Both issues will be solved by making better use of the new Raft-based capabilities of making multiple modifications to the schema atomically, but this patch doesn't fix the problem - it just proves it exist. The new tests involve two threads - one repeatedly trying to create a table with a GSI or with tags - and the other thread repeatedly trying to delete the same table under its feet. Both bugs are reproduced almost immediately. Note that like all test/alternator tests, the new tests are usually run on just one node. So when we fix the bug and these tests begin to pass, it will not be a proof that concurrent schema modification works safely on *different* nodes. To prove that, we will also need a multi-node test. However, this test can prove that we used Raft-based schema modification correctly - and if we assume that the Raft-based schema modification feature is itself correct, then we can be sure that CreateTable will be correct also across multiple nodes. Although it won't hurt to check it directly. Refs scylladb#6391 Refs scylladb#9868 Signed-off-by: Nadav Har'El <nyh@scylladb.com>
|
Already in branch 5.1. Not backporting to 5.0 because it relies on the new experimental Raft-based schema changes reorganization of the code. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The
CreateTableoperation has aTagsparameter used to set tags on the newly created table. However, the way we do this today is as two separate operations: first the table is created, then the tags are added. It is, at least in theory, possible for the table creation to succeed but then the tagging to fail - and the result is that the operation will seem to fail but still leave behind a new table, missing its tags.If the tagging fails we should probably delete the newly-created table. Or even better, we should find a way to atomically create a table already already with its tags and not as two independent operations.
The text was updated successfully, but these errors were encountered: