Alternator: missing fields in return of DescribeTable #5026

nyh · 2019-09-15T13:15:27Z

The DescribeTable operation returns only a subset of the fields it's supposed to return. While this is enough for many use cases - which only use DescribeTable to tell when creating a table actually finished - it might not be enough in different use cases.

What is missing - with xfailing tests for each already in alternator-test/test_describe_table.py:

The table's schema in KeySchema, AttributeDefinitions (see test_describe_table_schema).
ItemCount (see test_describe_table_item_count).
TableSizeBytes (see test_describe_table_size).
ProvisionedThroughput (see test_describe_table_provisioned_throughput).
ARN (see test_describe_table_arn)
TableId (see test_describe_table_id)

Also CreationDateTime is returned incorrectly (see separate issue #5013).

Also BillingModeSummary we return is always the same thing (PAY_PER_REQUEST) regardless of what the user chose when creating the table. This is often good enough, but we should reconsider this.

Moreover, when there are indexes in a table (GSI or LSI), we only provide partial information for those too - see xfailing test test_gsi_describe (we should probably have a similar test for LSI as well).

The DeleteTable operation returns the return type as DescribeTable, so maybe should return additional values as well.

The text was updated successfully, but these errors were encountered:

tzach · 2019-09-16T18:06:23Z

@nyh I do see

"TableId": "dbfd89f0-d86a-11e9-b3a5-000000000002"

On the Alternator create table response

nyh · 2019-11-20T16:30:48Z

@tzach reported to the mailing list a case where the AttributeDefinitions field was expected and caused an application to fail:

dynamodb-admin is an open-source node base UI project for DynamoDB.
I was able to create a table with it, but I try to view the data, I got the following error message below. Scylla itself does not report any issue.

TypeError: /usr/lib/node_modules/dynamodb-admin/views/scan.ejs:106
    104|     >
    105|       <datalist id="attributes">
 >> 106|         <% for (let AttributeDefinition of Table.AttributeDefinitions) { %>
    107|           <option value="<%= AttributeDefinition.AttributeName %>" />
    108|         <% } %>
    109|       </datalist>

Table.AttributeDefinitions is not iterable
    at eval (/usr/lib/node_modules/dynamodb-admin/views/scan.ejs:59:46)
    at scan (/usr/lib/node_modules/dynamodb-admin/node_modules/ejs/lib/ejs.js:682:17)
    at tryHandleCache (/usr/lib/node_modules/dynamodb-admin/node_modules/ejs/lib/ejs.js:254:36)
    at View.exports.renderFile [as engine] (/usr/lib/node_modules/dynamodb-admin/node_modules/ejs/lib/ejs.js:485:10)
    at View.render (/usr/lib/node_modules/dynamodb-admin/node_modules/express/lib/view.js:135:8)
    at tryRender (/usr/lib/node_modules/dynamodb-admin/node_modules/express/lib/application.js:640:10)
    at Function.render (/usr/lib/node_modules/dynamodb-admin/node_modules/express/lib/application.js:592:3)
    at ServerResponse.render (/usr/lib/node_modules/dynamodb-admin/node_modules/express/lib/response.js:1012:7)
    at describeTable.then.description (/usr/lib/node_modules/dynamodb-admin/lib/backend.js:404:13)
    at process._tickCallback (internal/process/next_tick.js:68:7)

One of the fields still missing in DescribeTable's response (Refs #5026) was the table's schema - KeySchema and AttributeDefinitions. This patch adds this missing feature, and enables the previously-xfailing test test_describe_table_schema. A complication of this patch is that in a table with secondary indexes, we need to return not just the base table's schema, but also the indexes' schema. The existing tests did not cover that feature, so we add here two more tests in test_gsi.py for that. One of these secondary-index schema tests, test_gsi_2_describe_table_schema, still fails, because it outputs a range-key which Scylla added to a view because of its own implementation needs, but wasn't in the user's definition of the GSI. I opened a separate issue #5320 for that. Signed-off-by: Nadav Har'El <nyh@scylladb.com>

nyh · 2019-11-21T15:17:35Z

KeySchema, AttributeDefinitions were done in commit 6672059 ("alternator: make DescribeTable return table schema"), but one small corner case remains (in a specific case involving a GSI) and moved to a separate issue #5320.

One of the fields still missing in DescribeTable's response (Refs #5026) was the table's schema - KeySchema and AttributeDefinitions. This patch adds this missing feature, and enables the previously-xfailing test test_describe_table_schema. A complication of this patch is that in a table with secondary indexes, we need to return not just the base table's schema, but also the indexes' schema. The existing tests did not cover that feature, so we add here two more tests in test_gsi.py for that. One of these secondary-index schema tests, test_gsi_2_describe_table_schema, still fails, because it outputs a range-key which Scylla added to a view because of its own implementation needs, but wasn't in the user's definition of the GSI. I opened a separate issue #5320 for that. Signed-off-by: Nadav Har'El <nyh@scylladb.com>

fruch · 2020-02-07T09:08:10Z

@nyh, I've encountered a real example that 6672059 is actually helps with
I've was playing with pynamodb
and it does assume having the KeySchema
https://github.com/pynamodb/PynamoDB/blob/master/pynamodb/connection/base.py#L101

I was using scylla 3.2.0, hence didn't had that. I guess I should use the latest docker for my this specific experiment

nyh · 2020-11-04T11:00:27Z

The ARN in the response was also already fixed, in commit 8c17b5a

This issue is becoming a mess, it's hard to tell what is still missing. I should split it to separate items.

nyh · 2020-11-04T16:05:01Z

Created issue #7550 for ItemCount, #7551 for TableSizeBytes and #11222 for ProvisionedThroughput.
Now, the remaining things to cover in new issues before this issue can be closed are:

BillingModeSummary we return is always the same thing (PAY_PER_REQUEST) regardless of what the user chose when creating the table. This is often good enough, but we should reconsider this.
When there are indexes in a table (GSI or LSI), we only provide partial information for those too - see xfailing test test_gsi_describe (we should probably have a similar test for LSI as well).
The DeleteTable operation returns the return type as DescribeTable, so maybe should return additional values as well.

DescribeTable should return a UUID "TableId" in its reponse. We alread had it for CreateTable, and now this patch adds it to DescribeTable. The test for this feature is no longer xfail. Moreover, I improved the test to not only check that the TableId field is present - it should also match the documented regular expression (the standard representation of a UUID). Refs #5026 Signed-off-by: Nadav Har'El <nyh@scylladb.com> Message-Id: <20201104114234.363046-1-nyh@scylladb.com>

nyh · 2022-09-06T16:43:52Z

We have individual issues about all remaining missing fields:

Alternator: missing ItemCount field in return of DescribeTable #7550 for ItemCount in base tables and indexes,
Alternator: missing TableSizeBytes field in return of DescribeTable #7551 for TableSizeBytes (for base tables)
Alternator: missing IndexSizeBytes field in return of DescribeTable #11466 for IndexSizeBytes (for indexes)
Alternator: missing IndexSizeBytes field in return of DescribeTable #11466 for Projection (for indexes)
Alternator: missing IndexStatus field in return of DescribeTable #11471 for IndexStatus (for indexes)
Alternator: missing fields in return of DeleteTable #11472 for missing fields in DeleteTable's response.

So closing this issue.

nyh added the area/alternator Alternator related Issues label Sep 15, 2019

slivne added this to the 3.x milestone Sep 22, 2019

nyh assigned nyh and unassigned nyh Nov 20, 2019

fruch mentioned this issue Oct 21, 2020

Fix YCSB case fruch/hydra-kcl#2

Merged

nyh mentioned this issue Sep 6, 2022

DynamoDB API create table with secondary index adds extra KeySchema #11455

Closed

1 task

nyh added the user request label Sep 6, 2022

nyh closed this as completed Sep 6, 2022

nyh mentioned this issue Sep 28, 2022

Alternator: return full table description on return of DeleteTable #11628

Closed

DoronArazii modified the milestones: 5.x, 5.2 Nov 8, 2022

julienrf mentioned this issue Jul 12, 2024

Alternator: incomplete attribute definitions in return of DescribeTable #19716

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alternator: missing fields in return of DescribeTable #5026

Alternator: missing fields in return of DescribeTable #5026

nyh commented Sep 15, 2019 •

edited

Loading

tzach commented Sep 16, 2019

nyh commented Nov 20, 2019 •

edited

Loading

nyh commented Nov 21, 2019

fruch commented Feb 7, 2020

nyh commented Nov 4, 2020

nyh commented Nov 4, 2020 •

edited

Loading

nyh commented Sep 6, 2022 •

edited

Loading

Alternator: missing fields in return of DescribeTable #5026

Alternator: missing fields in return of DescribeTable #5026

Comments

nyh commented Sep 15, 2019 • edited Loading

tzach commented Sep 16, 2019

nyh commented Nov 20, 2019 • edited Loading

nyh commented Nov 21, 2019

fruch commented Feb 7, 2020

nyh commented Nov 4, 2020

nyh commented Nov 4, 2020 • edited Loading

nyh commented Sep 6, 2022 • edited Loading

nyh commented Sep 15, 2019 •

edited

Loading

nyh commented Nov 20, 2019 •

edited

Loading

nyh commented Nov 4, 2020 •

edited

Loading

nyh commented Sep 6, 2022 •

edited

Loading