Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch delete does not work #3996

Closed
orfisko opened this issue Jan 23, 2024 · 3 comments
Closed

Batch delete does not work #3996

orfisko opened this issue Jan 23, 2024 · 3 comments
Labels
bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged.

Comments

@orfisko
Copy link

orfisko commented Jan 23, 2024

Describe the bug

I have a dynamodb table containing this record:

{
  "identifier": {
    "cuttingListLineId": "01HMVM5PAFP0AK03X2WS9AT5AK",
    "projectId": "01HMVM5PA44P1X6Z1PMX2TEXCT",
    "cuttingListId": "01HMVM5PA4WXPW0V6JNFKRSYP4"
  },
  "boardSku": "FIN_MEL_UN_0WE31BST_PB_18_L2800W2070",
  "SK": "ITEMTYPE#COSTDRIVER#CUTTINGLIST#01HMVM5PA4WXPW0V6JNFKRSYP4#CUTTINGLISTLINE#01HMVM5PAFP0AK03X2WS9AT5AK#COSTDRIVERTYPE#PANEL",
  "PK": "PROJECT#01HMVM5PA44P1X6Z1PMX2TEXCT",
  "bomItemType": "COSTDRIVER",
  "panelType": "CUTTING_LIST",
  "dimension": {
    "length": "1000",
    "width": "500",
    "measurementUnit": "MILLIMETRE"
  },
  "__module__": "dk_common.model.bom.costdriver",
  "costDriverType": "PANEL",
  "__class__": "PanelCostDriver"
}

The dynamodb table is defined as follows:

TableName: bomservice-dev
AttributeDefinitions:
  - AttributeName: PK
    AttributeType: S
  - AttributeName: SK
    AttributeType: S
  - AttributeName: GSI1PK
    AttributeType: S
  - AttributeName: GSI1SK
    AttributeType: S
KeySchema:
  - AttributeName: PK
    KeyType: HASH
  - AttributeName: SK
    KeyType: RANGE
GlobalSecondaryIndexes:
  - IndexName: GSI1
    KeySchema:
      - AttributeName: GSI1PK
        KeyType: HASH
      - AttributeName: GSI1SK
        KeyType: RANGE
    Projection:
      ProjectionType: ALL
BillingMode: PAY_PER_REQUEST

I have that table deployed on a local dynamodb instance:

CONTAINER ID   IMAGE                          COMMAND                  CREATED       STATUS       PORTS                                       NAMES
f9029346a985   amazon/dynamodb-local:latest   "java -jar DynamoDBL…"   3 weeks ago   Up 2 weeks   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp   dynamodb-local

I have made a generic batch writer function for all my batch writing needs. The end result of that python code boils down to this:

with table.batch_writer() as batch:
     batch.delete_item(Key={"PK": {"S": "PROJECT#01HMVM5PA44P1X6Z1PMX2TEXCT"}, "SK": {"S": "ITEMTYPE#COSTDRIVER#CUTTINGLIST#01HMVM5PA4WXPW0V6JNFKRSYP4#CUTTINGLISTLINE#01HMVM5PAFP0AK03X2WS9AT5AK#COSTDRIVERTYPE#PANEL"}})

When I try to delete now the aforementioned record with this code, it does not work.
If I check the debug logs, there are no unprocessed records nor any other mention of an error whatsoever:

DEBUG:botocore.endpoint:Making request for OperationModel(name=BatchWriteItem) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'DynamoDB_20120810.BatchWriteItem', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'Boto3/1.33.10 md/Botocore#1.33.10 ua/2.0 os/linux#6.1.0-1027-oem md/arch#x86_64 lang/python#3.11.7 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.33.10 Resource'}, 'body': b'{"RequestItems": {"cuttinglistservice-dev": [{"DeleteRequest": {"Key": {"PK": {"S": "PROJECT#01HMVM5PA44P1X6Z1PMX2TEXCT"}, "SK": {"S": "ITEMTYPE#COSTDRIVER#CUTTINGLIST#01HMVM5PA4WXPW0V6JNFKRSYP4#CUTTINGLISTLINE#01HMVM5PAFP0AK03X2WS9AT5AK#COSTDRIVERTYPE#PANEL"}}}}]}}', 'url': 'http://localhost:8000/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7fa54d9634d0>, 'has_streaming_input': False, 'auth_type': None}}  
DEBUG:botocore.parsers:Response body:
b'{"UnprocessedItems":{}}'
DEBUG:botocore.hooks:Event needs-retry.dynamodb.BatchWriteItem: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa54d97c510>
DEBUG:botocore.retryhandler:No retry needed.
DEBUG:botocore.hooks:Event after-call.dynamodb.BatchWriteItem: calling handler <bound method TransformationInjector.inject_attribute_value_output of <boto3.dynamodb.transform.TransformationInjector object at 0x7fa54d97cf10>>
DEBUG:boto3.dynamodb.table:Batch write sent 1, unprocessed: 0

No idea what is going wrong here, but it seems like a bug somewhere in boto3. Any ideas?
Also, I would love to easily catch those unprocessed items, but they seem not to be returned by the batch writer.

Expected Behavior

Well, that it deletes the records for which I specified the keys and does not fail silently on me.

Current Behavior

It does not delete the record nor does it raise an error

Reproduction Steps

If all the stuff mentioned above is deployed, I guess this would suffice:

with table.batch_writer() as batch:
     batch.delete_item(Key={"PK": {"S": "PROJECT#01HMVM5PA44P1X6Z1PMX2TEXCT"}, "SK": {"S": "ITEMTYPE#COSTDRIVER#CUTTINGLIST#01HMVM5PA4WXPW0V6JNFKRSYP4#CUTTINGLISTLINE#01HMVM5PAFP0AK03X2WS9AT5AK#COSTDRIVERTYPE#PANEL"}})

Possible Solution

No testcase for this in the boto3 deployment pipeline?

Additional Information/Context

No response

SDK version used

boto3==1.34.25 botocore==1.34.25

Environment details (OS name and version, etc.)

Python 3.11.7 on Ubuntu 22.04.3 LTS

@orfisko orfisko added bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Jan 23, 2024
@orfisko
Copy link
Author

orfisko commented Jan 23, 2024

This can be closed. I just noticed I was targeting the wrong table. Still, it would be nice if this would not simply pass happily without raising an error though...

@orfisko
Copy link
Author

orfisko commented Jan 23, 2024

Closing since already discussed in
#2584

@orfisko orfisko closed this as completed Jan 23, 2024
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged.
Projects
None yet
Development

No branches or pull requests

1 participant