Alternator: allow adding a GSI to a pre-existing table

[nyh]

This is a cleanup of issue #5022 which asked to implement the UpdateTable operation and its ability to add a GSI to an already-existing table

Whereas DynamoDB requires that any LSIs must be created up-front when the table is first created, DynamoDB does allow a user to add a GSI to a preexisting table - and also to remove a GSI from a table - using the UpdateTable operation. Alternator doesn't support these UpdateTable operations, and allowing them is the purpose of this issue.

We have xfailing tests for this issue in test_gsi.py: test_gsi_backfill and test_gsi_delete test adding a new GSI, and deleting a GSI, respectively.

The reason why we don't yet support this feature wasn't a small oversight, but rather a deep problem related to the interaction of Alternator's schema-less item representation, and our materialized-views implementation:

We implement a GSI as a materialized view. A materialized view's key must be a real Scylla column. But in Alternator, most attributes except the key attributes aren't a real Scylla column but rather elements of a single map :attrs, so such an attribute cannot be used as the GSI key. Today, when we create the GSI up-front, together with the table, we make the GSI key a real column. But when adding a GSI to an existing table, the GSI key would most likely be an element of the map and not a real column.

I think the best way to add this feature is as follows:

We can add to our materialized views implementation the ability to use as a key a specific member of a map, e.g., :attrs[hello]. This is related to the Cassandra 4.0 map value select feature #7751 (although Cassandra doesn't support this for materialized views) and to #5440 (although that was on CQL secondary-index, not materialized views). But in this issue there is one extra snag: In Alternator, the map's value isn't typed (e.g., it isn't an integer), it is a serialized JSON. So we'll need to add an ability for the materialized views code to run an Alternator-specific de-serialization function. This function will need deserialize the JSON and "skip" items whose value has the "wrong" type (the right type for this GSI's key is declared when adding the GSI).

We will need to use the new mechanism not just when adding a GSI to an existing table, but also when creating a new table with GSI. I.e., we cannot continue to create the GSI's key as a real Scylla column as we do today, not even in a new table. The reason is that if a GSI's key is a real column, it has a fixed type - but if we want to support later deleting the GSI, the user must be able after this deletion to use in this column values of any type, not just the type enforced while the GSI existed. Note that if we want, we can continue to use real Scylla columns for LSI keys, because unlike GSIs, LSIs cannot be added or deleted after creating the table.

Here is an alternative implementation idea, which I believe is worse: Adding a GSI to an existing table is already expected to be a fairly long process (it is called "backfilling" in DynamoDB, or "view building" in Scylla). So we can also spend time on moving data between the map and new a real column which will be the GSI key. The problem is that this process is slow, ugly, and race-prone (this copying process can race with expressions reading or modifying this column). We also need to reverse this process when a GSI is removed - because after a GSI is removed DynamoDB allows a user to set this column to any value of any type - something which isn't allowed on a real schema column that must have a pre-determined type.

When we implement adding a GSI to an existing table, we should remember that while the backfilling ("view building") is in progress, DescribeTable should return for this index IndexStatus=CREATING and Backfilling=True. We don't have a test for that yet. It's not obvious how to test for this, because the backfilling stage can be so short we won't notice it is happening. Perhaps we can use injection to ensure it is paused and only resumed when we want.

[nyh]

@fee-mendes noticed that if a user tries to add a GSI, using a GlobalSecondaryIndexUpdates parameter and also (as needed) a AttributeDefinitions, Alternator reports the unsupported "AttributeDefinitions" which is confusing and not explaining what is really not supported. We should improve the error message.