Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: creating gsi under on-demand mode #85

Merged
merged 5 commits into from
Nov 14, 2023

Conversation

Julian-Chu
Copy link
Contributor

@Julian-Chu Julian-Chu commented Nov 7, 2023

Issue #, if available:

Description of changes:

  • fix empty NonKeyAttributes validation issue
  • fix GSI creation issue when using pay-per-request mode
2023-04-19T17:31:29.932+0200    ERROR   Reconciler error        {"controller": "table", "controllerGroup": "dynamodb.services.k8s.aws", "controllerKind": "Table", "Table": {"name":"ack-demo-table-provisioned-global-secondary-indexes","namespace":"ack-system"}, "namespace": "ack-system", "name": "ack-demo-table-provisioned-global-secondary-indexes", "reconcileID": "c7354a77-56c4-4c56-b715-f16b59d1926e", "error": "ValidationException: One or more parameter values were invalid: Neither ReadCapacityUnits nor WriteCapacityUnits can be specified when BillingMode is PAY_PER_REQUEST\n\tstatus code: 400, request id: R901I15N84RUVS24D4HVRP6L3JVV4KQNSO5AEMVJF66Q9ASUAAJG"}

2023-04-19T18:54:54.887+0200    ERROR   Reconciler error        {"controller": "table", "controllerGroup": "dynamodb.services.k8s.aws", "controllerKind": "Table", "Table": {"name":"ack-demo-table-provisioned-global-secondary-indexes","namespace":"ack-system"}, "namespace": "ack-system", "name": "ack-demo-table-provisioned-global-secondary-indexes", "reconcileID": "223f9f39-f2cc-460c-ad78-56fce55b4bf3", "error": "ValidationException: One or more parameter values were invalid: ProvisionedThroughput should not be specified for index: id-index when BillingMode is PAY_PER_REQUEST\n\tstatus code: 400, request id: JO5OGQR8IN74JJ2C62O43ESU43VV4KQNSO5AEMVJF66Q9ASUAAJG"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    conditions:
    - message: |
        InvalidParameter: 3 validation error(s) found.
        - minimum field size of 1, UpdateTableInput.GlobalSecondaryIndexUpdates[0].Create.Projection.NonKeyAttributes.
        - minimum field value of 1, UpdateTableInput.GlobalSecondaryIndexUpdates[0].Create.ProvisionedThroughput.ReadCapacityUnits.
        - minimum field value of 1, UpdateTableInput.GlobalSecondaryIndexUpdates[0].Create.ProvisionedThroughput.WriteCapacityUnits.
      status: "True"
      type: ACK.Terminal

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@ack-prow ack-prow bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 7, 2023
Copy link

ack-prow bot commented Nov 7, 2023

Hi @Julian-Chu. Thanks for your PR.

I'm waiting for a aws-controllers-k8s member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@a-hilaly
Copy link
Member

a-hilaly commented Nov 8, 2023

/ok-to-test

@ack-prow ack-prow bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 8, 2023
Copy link
Member

@a-hilaly a-hilaly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much @Julian-Chu ! This is neat!
I have one questions below.

Comment on lines +236 to +238
if pt == nil {
return nil
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch sir!

Comment on lines +239 to +250
provisionedThroughput := &svcsdk.ProvisionedThroughput{
// ref: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_ProvisionedThroughput.html
// Minimum capacity units is 1 when using provisioned capacity mode
ReadCapacityUnits: aws.Int64(1),
WriteCapacityUnits: aws.Int64(1),
}
if pt.ReadCapacityUnits != nil {
provisionedThroughput.ReadCapacityUnits = aws.Int64(*pt.ReadCapacityUnits)
}

if pt.WriteCapacityUnits != nil {
provisionedThroughput.WriteCapacityUnits = aws.Int64(*pt.WriteCapacityUnits)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts: What if we're not using PROVISIONED capacity mode? Would 0's be allowed? If that's the case would it be worth to evaluate the capacity mode in this function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you mean the PAY_PER_REQUEST mode.
this is tricky part. IIRC, we don't allow to set 0 capacity units for PAY_PER_REQUEST , please see the following message

   conditions:
    - message: |
        InvalidParameter: 3 validation error(s) found.
        - minimum field size of 1, UpdateTableInput.GlobalSecondaryIndexUpdates[0].Create.Projection.NonKeyAttributes.
        - minimum field value of 1, UpdateTableInput.GlobalSecondaryIndexUpdates[0].Create.ProvisionedThroughput.ReadCapacityUnits.
        - minimum field value of 1, UpdateTableInput.GlobalSecondaryIndexUpdates[0].Create.ProvisionedThroughput.WriteCapacityUnits.
      status: "True"
      type: ACK.Terminal

but aws DDB api will set capacity units as 0 when PAY_PER_REQUEST

If you set BillingMode as PROVISIONED, you must specify this property. If you set BillingMode as PAY_PER_REQUEST, you cannot specify this property.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html#cfn-dynamodb-table-provisionedthroughput

If read/write capacity mode is PAY_PER_REQUEST the value is set to 0.

https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_ProvisionedThroughput.html

}
} else {
projection.ProjectionType = aws.String("")
projection.NonKeyAttributes = []*string{}
projection.NonKeyAttributes = nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another nice catch!

@ack-prow ack-prow bot added the approved label Nov 13, 2023
Copy link
Member

@a-hilaly a-hilaly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
/lgtm

@ack-prow ack-prow bot added the lgtm Indicates that a PR is ready to be merged. label Nov 14, 2023
Copy link

ack-prow bot commented Nov 14, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: a-hilaly, Julian-Chu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@a-hilaly
Copy link
Member

/test all

@ack-prow ack-prow bot merged commit b52e89f into aws-controllers-k8s:main Nov 14, 2023
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test.
Projects
None yet
2 participants