aws-dynamodb: Cannot add multiple dynamodb GSI.

[sarawin-pm]

❓ General Issue

I got "Cannot perform more than one GSI creation or deletion in a single update" error when I tried to add multiple GSI to dynamodb.

const mainTable = new dynamodb.Table(this, "MainTable", {
  partitionKey: { name: "pk", type: dynamodb.AttributeType.STRING },
  sortKey: { name: "sk", type: dynamodb.AttributeType.STRING },
  billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
});

mainTable.addGlobalSecondaryIndex({
  indexName: "pk-createdAt-index",
  partitionKey: { name: "pk", type: dynamodb.AttributeType.STRING },
  sortKey: { name: "createdAt", type: dynamodb.AttributeType.STRING },
});

mainTable.addGlobalSecondaryIndex({
  indexName: "__typename-index",
  partitionKey: { name: "__typename", type: dynamodb.AttributeType.STRING },
});

mainTable.addGlobalSecondaryIndex({
  indexName: "__typename-status-index",
  partitionKey: { name: "__typename", type: dynamodb.AttributeType.STRING },
  sortKey: { name: "status", type: dynamodb.AttributeType.STRING },
});

The Question

Until now, I have to uncomment code for each GSI adding section and redeploy multiple times.
How can I add multiple GSI to dynamodb and deploy in single time?

Environment

• CDK CLI Version: 1.73.0 (build eb6f3a9)
• Module Version: 1.73.0
• Node.js Version: v12.18.3
• OS: Ubuntu 18.04.1
• Language (Version): typescript 3.9.7

Other information

I think aws-sdk does not allow to add more than one GSI at the time. Is it possible that we can add one by one until every GSI are added?

[skinny85]

That is very weird... this sounds like a CloudFormation bug to me...?

@sarawin-pm can you see if you get a similar error when deploying purely through CloudFormation?

[hoegertn]

Yes. This is a known limitation in CloudFormation.

[skinny85]

Damn. This seems like a giant problem with the CloudFormation DynamoDB support...

[sarawin-pm]

I have two ideas. I'm not sure is it possible to implement.

1. Can we generate cloud formation multiple times during single cdk deployment? When deployment, check whether it has GSI1 if yes then add GSI2 to cloud formation and repeat until we can have all.
2. Can we generate GSI creation into some aws lambda during cdk deployment? and lambda handles all GSI creation task.

[skinny85]

Yes, we could add a Custom Resource that adds the GSIs one at a time.

But honestly, this seems to me like such an egregious bug with the CloudFormation DynamoDB support, I think it should be solved on the level of the CFN DynamoDB provider.

[laurrentt]

This has been flagged with the CloudFormation team before (aws-cloudformation/cloudformation-coverage-roadmap#229). If you look at this particular answer aws-cloudformation/cloudformation-coverage-roadmap#229 (comment) from @luiseduardocolon who closed the issue, they believe it's the DynamoDB team that should fix this. I don't really care that much about AWS' internal politics and who's right or wrong, but I think the clear loser here is the customer.

This is especially dangerous for disaster recovery scenarios where I would need to quickly deploy the stack in a new environment. I would need some kind of script that deploys it as many times as there are indexes in my table, adding one after the other.

[rix0rrr]

I would strongly encourage someone to create and independently vend a construct that creates multiple GSIs in order.

With the right NPM tags it will automatically show up here: https://awscdk.io/

Once the 3rd party construct has some proven mileage on it, we will consider merging it upstream into the CDK core library.

[shellscape]

Pinging to see if anyone that's been following the issue has workarounds so we don't have to break our deployments up

[Mikko-AM]

We used step functions to create GSI, wait it's completion and then create next GSI. We use separate CDK stack that creates only this one table that needed multiple GSI's.

[ThomasRooney]

A semi-automated workaround I've been using, via a nodejs script invoked by CDK.

1. For all tables that are being deployed:
2. Pull down the stack template JSON via cloudformation.getTemplate. Access the cloudformation resource associated with the dynamodb table
3. Compute a list of indexes to destroy, indexes to create via comparing the desired GSI state against the current one in cloudformation
4. Invoke the stack constructor in CDK passing in the original CF's GSI state with a single index change from the desired state. The indexes to destroy are applied first.

The script is invoked by the CDK runner cdk deploy --all --app "ts-node --prefer-ts-exts scripts/iterative-dynamodb-index-update dev" --path-metadata=false, with multiple GSIs changes being applied via running the script, waiting until the index change is applied, and then re-running it.

Edit May 2022: Open Sourced @ https://github.com/ThomasRooney/reamplify/blob/master/packages/deploy/scripts/iterative-dynamodb-index-update.ts

[stoyan-scava]

I was able to deploy a dynamodb table with more than one GSI with a single CodeBuild.
It seems to me that the dynamo limitation to modify only one GSI is only for cli.

[flochaz]

I would strongly encourage someone to create and independently vend a construct that creates multiple GSIs in order.

With the right NPM tags it will automatically show up here: https://awscdk.io/

Once the 3rd party construct has some proven mileage on it, we will consider merging it upstream into the CDK core library.

Here is a first draft : https://www.npmjs.com/package/aws-dynamodb-table-multi-gsis

[blimmer]

I wanted to call out that you can currently create a table with multiple GSIs defined without any problems or workarounds. The CloudFormation-related issue is when you're mutating more than one index in a single changeset.

I agree that the upstream index mutation logic should be fixed for updates, but I wanted to comment that creates are not affected for folks reading this thread.

[lmiguelmh]

I don't know if what I'm doing is wrong but I tried to follow @blimmer's advice and wasn't able to create a new table with 2 GSIs. I got the same error: "Cannot perform more than one GSI creation or deletion in a single update"

This is a summary of the code that I've used. I didn't find a reference in the Table docs for another way to add a GSI.

some_table = dynamodb.Table(
    ...
)
some_table.add_global_secondary_index(
    ...
)
some_table.add_global_secondary_index(
    ...
)

[ApexOps-Scott]

Same issue Sep 22 using SAM which builds in a codebuild project and deploys straight to cfn. Can add one GSI at a time only. Very frustrating.

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[cfinkelstein]

Yes, same here - would be great to have a proper solution for this issue available.

[wz2b]

Yes, we could add a Custom Resource that adds the GSIs one at a time.

But honestly, this seems to me like such an egregious bug with the CloudFormation DynamoDB support, I think it should be solved on the level of the CFN DynamoDB provider.

How would you make sure your lambda to create the custom resource doesn't run until the table is finished being created?

CreateTable is an asynchronous operation. Upon receiving a CreateTable request, DynamoDB immediately returns a response with a TableStatus of CREATING. After the table is created, DynamoDB sets the TableStatus to ACTIVE. You can perform read and write operations only on an ACTIVE table.

UpdateTable is an asynchronous operation; while it is executing, the table status changes from ACTIVE to UPDATING. While it is UPDATING, you cannot issue another UpdateTable request. When the table returns to the ACTIVE state, the UpdateTable operation is complete.

Suppose you want your table to have two GSIs. If you create the table in CDK then your lambda has to wait for the CreateTable to be finished, then issue one UpdateTable, then wait for THAT to be finished, then issue the second UpdateTable. If the custom resource is in on_create() it can do it all at once (up to 20 indices) but if it's in on_update() then it has to know when DynamoDB is ready for the next action.

How is everyone trying to solve this with custom resources solving this?

[wz2b]

I was able to deploy a dynamodb table with more than one GSI with a single CodeBuild. It seems to me that the dynamo limitation to modify only one GSI is only for cli.

I don't think that's the case. According to the API documentation, it's a DynamoDB limitation. It comes from two different constraints. First, each UpdateTable can only touch one index at a time:

GlobalSecondaryIndexUpdates
An array of one or more global secondary indexes for the table. For each index in the array, you can request one action:
Create - add a new global secondary index to the table.
Update - modify the provisioned throughput settings of an existing global secondary index.
Delete - remove a global secondary index from the table.
You can create or delete only one global secondary index per UpdateTable operation.

Second, UpdateTable is asynchronous, so you can't issue two UpdateTables at the same time - you have to wait for the table status to go from UPDATING back to ACTIVE before you can issue the second one. If you're building an index on a huge table I think that means waiting for it to complete generating the index which could take a long time.

[wz2b]

I don't know if what I'm doing is wrong but I tried to follow @blimmer's advice and wasn't able to create a new table with 2 GSIs. I got the same error: "Cannot perform more than one GSI creation or deletion in a single update"

Me, too. According to the DynamoDB API it should work when creating the table for the first time. Maybe there is a problem in both the DynamoDB API and how CloudFormation handles it.

[mcmurm]

Also running into this issue modifying existing tables that were previously deployed via CloudFormation. Another user previously noted, there are situations (like Disaster Recovery, hotfixes, etc.) where speed and consistency of deployment is critical. This "bug" whether the responsibility of the CloudFormation or DynamoDB product teams needs to be addressed.

[poudelroshan]

Facing the same issue. Would be great to have this "feature" expedited.

[aj84276]

Stuck with this, looking for some workaround.

[rrhurtado]

I am also encountering this problem. I would be very grateful if you could prioritize this feature.

[phillsv87]

I just ran into the same problem and I'm using the latest version of the CDK. AWS please help us out here.

[Naumel]

Looking at https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html#cfn-dynamodb-table-globalsecondaryindexes the needs-cfn label is now incorrect. Updating accordingly.

[sanjayshukla98]

I tried pre-establishing the global secondary indexes manually in the console and then deploying using the SDK. Even when the tables already contain the secondary indices, there is still an error thrown not being able to create more than one GSI at a time.

[corymhall]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

Just wanted to let everyone know that we saw the upvotes, but this is not something we plan on addressing anytime soon. This comment is still our stance.

[shellscape]

@corymhall what's the reasoning?

[Thomas-X]

Ran into this for the ninth time this month again, hope cdk-tf gets more stable soon so I can get out of this cloudformation craphole.

Delegating a fix to a 3rd party solution as it's not some issue with the tool itself is the opposite of open source working

[luc-ferron]

Seriously AWS, for a company that boasts about being customer centric, this issue is a proof that it's just not true.

[BerndWessels]

Is this still an issue? Just asking because the code below seems to work/deploy just fine (haven't tried multiple times though)

import { NestedStack, NestedStackProps } from "aws-cdk-lib";
import { AttributeType, BillingMode, Table } from "aws-cdk-lib/aws-dynamodb";
import { Construct } from "constructs";

/**
 * Stack Properties
 */
export interface DatabaseStackProps extends NestedStackProps {
  readonly environmentName: string;
}

/**
 * Stack
 */
export class DatabaseStack extends NestedStack {
  /**
   * Outputs
   */
  public readonly baseTable: Table;

  /**
   * Construct
   */
  constructor(scope: Construct, id: string, props: DatabaseStackProps) {
    super(scope, id, props);
    /**
     * Props
     */
    const { environmentName } = props;

    /**
     * The base DynamoDB table using the single table design pattern.
     */
    const baseTable = new Table(this, `BaseTable`, {
      partitionKey: { name: "pk", type: AttributeType.STRING },
      sortKey: { name: "sk", type: AttributeType.STRING },
      billingMode: BillingMode.PAY_PER_REQUEST,
    });

    /**
     * The reverse global secondary index for pk sk.
     */
    baseTable.addGlobalSecondaryIndex({
      indexName: "gsir",
      partitionKey: {
        name: "sk",
        type: AttributeType.STRING,
      },
      sortKey: {
        name: "pk",
        type: AttributeType.STRING,
      },
    });

    /**
     * The global secondary index for sk sk1.
     */
    baseTable.addGlobalSecondaryIndex({
      indexName: "gsi1",
      partitionKey: {
        name: "sk",
        type: AttributeType.STRING,
      },
      sortKey: {
        name: "sk1",
        type: AttributeType.STRING,
      },
    });

    /**
     * Output
     */
    this.baseTable = baseTable;
  }
}

[ntippie]

@BerndWessels Is this for creating or updating? Some reports like this one said that you can create a table with multiple GSIs, but you can only add them one-at-a-time for existing tables. So you’d need to deploy your code, then add 2 more GSIs, then deploy a second time.

[BerndWessels]

@ntippie Right, I just ran into "Cannot perform more than one GSI creation or deletion in a single update" while trying to update a dynamo table with multiple GSIs.
So there seems to be still some issues.
Hopefully the lovely CDK team will address this in a timely manner, considering DynamoDB is one of the flagship services for such a long time already.

[phillsv87]

I found a workaround, but it has a big caveat. If your table is empty you can update multiple GSIs at once 😅. Pretty helpful, I know.

[BryanCrotaz]

A semi-automated workaround I've been using, via a nodejs script invoked by CDK.

1. For all tables that are being deployed:
2. Pull down the stack template JSON via cloudformation.getTemplate. Access the cloudformation resource associated with the dynamodb table
3. Compute a list of indexes to destroy, indexes to create via comparing the desired GSI state against the current one in cloudformation
4. Invoke the stack constructor in CDK passing in the original CF's GSI state with a single index change from the desired state. The indexes to destroy are applied first.

The script is invoked by the CDK runner cdk deploy --all --app "ts-node --prefer-ts-exts scripts/iterative-dynamodb-index-update dev" --path-metadata=false, with multiple GSIs changes being applied via running the script, waiting until the index change is applied, and then re-running it.

Edit May 2022: Open Sourced @ https://github.com/ThomasRooney/reamplify/blob/master/packages/deploy/scripts/iterative-dynamodb-index-update.ts

Any instructions on how to set this up (as a custom resource??) on a stack?