Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

scalar-schemaを使ってパーティションキーあり、クラスタリーキーなし、セカンダリインデックスありの条件のテーブルをAmazon DynamoDBに登録したい #187

Closed
scalar-kato opened this issue Apr 14, 2021 · 5 comments
Closed

scalar-schemaを使ってパーティションキーあり、クラスタリーキーなし、セカンダリインデックスありの条件のテーブルをAmazon DynamoDBに登録したい #187

scalar-kato opened this issue Apr 14, 2021 · 5 comments

Comments

@scalar-kato
Copy link

Is your feature request related to a problem? Please describe.
scalardbのツールを使ってdynamodbにschemaを定義したいのだが特定のパターンのテーブルだと定義できない。
https://github.com/scalar-labs/scalardb/tree/master/tools/scalar-schema

パーティションキーあり、クラスタリーキーなし、セカンダリインデックスありの条件のテーブルをscalar-schemaでdynamodbにテーブル作成しようとすると以下のエラーになった。

./workflow_templates.json
2021-04-09 13:19:44,696 Exception in thread "main" software.amazon.awssdk.services.dynamodb.model.DynamoDbException: One or more parameter values were invalid: Some index key attributes are not defined in AttributeDefinitions. Keys: [deleted], AttributeDefinitions: [concatenatedPartitionKey] (Service: DynamoDb, Status Code: 400, Request ID: 810F9NAKCICUGV9JIB02EIGRFJVV4KQNSO5AEMVJF66Q9ASUAAJG, Extended Request ID: null)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
	at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.createTable(DefaultDynamoDbClient.java:1062)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
	at scalar_schema.dynamo$create_table.invokeStatic(dynamo.clj:196)
	at scalar_schema.dynamo$make_dynamo_operator$reify__3394.create_table(dynamo.clj:210)
	at scalar_schema.operations$create_tables$fn__4030.invoke(operations.clj:23)
	at clojure.core$map$fn__5866.invoke(core.clj:2755)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:51)
	at clojure.lang.Cons.next(Cons.java:39)
	at clojure.lang.RT.next(RT.java:713)
	at clojure.core$next__5386.invokeStatic(core.clj:64)
	at clojure.core$dorun.invokeStatic(core.clj:3142)
	at clojure.core$doall.invokeStatic(core.clj:3148)
	at scalar_schema.operations$create_tables.invokeStatic(operations.clj:19)
	at scalar_schema.core$_main.invokeStatic(core.clj:35)
	at scalar_schema.core$_main.doInvoke(core.clj:32)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at scalar_schema.core.main(Unknown Source)

こちらのJSONデータを使ってscalar-schemaを実行した際に上のエラーになりました。

{
  "keyspacename.workflow_templates": {
    "transaction": true,
    "partition-key": [
      "template_id"
    ],
    "columns": {
      "template_id": "TEXT",
      "template_name": "TEXT",
      "template_desc": "TEXT",
      "owner": "TEXT",
      "members": "TEXT",
      "status": "TEXT",
      "created_at": "BIGINT",
      "created_by": "TEXT",
      "updated_at": "BIGINT",
      "updated_by": "TEXT",
      "template_detail_json": "TEXT",
      "deleted": "BOOLEAN"
    },
    "secondary-index": [
      "deleted"
    ]
  }
}

Describe the solution you'd like
パーティションキーあり、クラスタリーキーなし、セカンダリインデックスありの条件のテーブルをscalar-schemaでdynamodbにテーブルを登録したい

Describe alternatives you've considered

Additional context
@feeblefakie
Copy link
Contributor

@scalar-kato

    "clustering-key": [
    ],

これを足してもだめですか？
（ "clustering-key"のキーは必須です）

@scalar-kato
Copy link
Author

@feeblefakie
提案いただいたjson形式で再度実行しましたが同様のエラーが発生しました。

@yito88 yito88 mentioned this issue Apr 16, 2021
Merged
@yito88
Copy link
Member

yito88 commented Apr 19, 2021 •
edited

@scalar-kato Thank you for reporting this issue!

This error happened when a secondary index would be registered without clustering keys.
It was caused because the condition to make Attribute Definitions was wrong.
I've fixed it at #189.

By the way, I'm not sure why you want to use "BOOLEAN" as a secondary index type.
The query with the index would be too slow because it goes through all records to check deleted records.
For example, you can create a deleted table and insert the keys of records that you want to delete asynchronously.

@feeblefakie
Copy link
Contributor

@scalar-kato I agree with @yito88.
I feel the schema needs to be re-designed.
Using a secondary index for low cardinality attribute is a bad practice in any databases since it will retrieve mos records. (For example, we have 1 million records and 300k records are marked as deleted then the secondary index access will retrieve the 300k records.)

@scalar-kato
Copy link
Author

@yito88 @feeblefakie
Thank you for the response!
I now understand that secondary indexes should not be used for columns with low cardinality, and I will fix the DB schema in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@feeblefakie @yito88 @scalar-kato