Concurrent UpdateItem upserts to the same new item fail with ConditionalCheckFailedException

[justinsoong]

Version: built from source at v0.1.3 (main, 3961c63), Postgres 16 backend, builtin auth, default settings.

Summary

When two or more UpdateItem calls with no ConditionExpression race to create the same item (same table, same key, item does not exist yet), one call succeeds and the others fail with ConditionalCheckFailedException. DynamoDB serializes these writes and all of them succeed.

I found this while evaluating ExtendDB against an existing production DynamoDB workload. Our request flow issues two unconditional upserts to the same configuration item in parallel on first write, which works on DynamoDB but returns a 500 through ExtendDB every time a new item is first created.

Repro

Node, any AWS SDK v3, against a fresh table:

import {DynamoDBClient, CreateTableCommand, UpdateItemCommand} from "@aws-sdk/client-dynamodb";
import {marshall} from "@aws-sdk/util-dynamodb";

const c = new DynamoDBClient({endpoint: "https://localhost:18443", region: "us-east-1"});

await c.send(new CreateTableCommand({
  TableName: "Repro",
  BillingMode: "PAY_PER_REQUEST",
  AttributeDefinitions: [{AttributeName: "pk", AttributeType: "S"}],
  KeySchema: [{AttributeName: "pk", KeyType: "HASH"}],
}));
await new Promise((r) => setTimeout(r, 6000));

const upsert = (i) => c.send(new UpdateItemCommand({
  TableName: "Repro",
  Key: marshall({pk: "same-key"}),
  UpdateExpression: "SET #v = :v",
  ExpressionAttributeNames: {"#v": "value"},
  ExpressionAttributeValues: marshall({":v": `writer-${i}`}),
  ReturnValues: "ALL_NEW",
}));

const results = await Promise.allSettled([upsert(1), upsert(2), upsert(3), upsert(4)]);
results.forEach((r, i) => console.log(i, r.status, r.reason?.name ?? ""));

Observed

0 rejected ConditionalCheckFailedException
1 rejected ConditionalCheckFailedException
2 fulfilled
3 rejected ConditionalCheckFailedException

Sequential create-then-update on the same key works fine (0 failures in 20 runs), so this is specific to concurrent creation of a not-yet-existing item.

Expected

All four calls succeed, as they do on DynamoDB. UpdateItem without a ConditionExpression should never surface ConditionalCheckFailedException.

Suspected cause

The losing writers appear to take the insert path, hit the Postgres unique constraint violation on the primary key, and map it to a conditional failure instead of retrying as an update. An INSERT ... ON CONFLICT DO UPDATE in the storage-postgres upsert path, or a retry of the update path on unique violation, would presumably resolve it.

Happy to provide more detail or test a fix against our workload.

[LeeroyHannigan]

Thanks @justinsoong for the well defined issue. Great catch.