bug: DynamoDB Streams with Byte property crashes Lambda

[TRAD-Anthony-CKO]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Given I create a DynamoDB Table with DynamoDB streams enabled.

When I PUT an item into that table having a Byte attribute (B)
And the Lambda plugged into that stream triggered.
Then I see a JSON encoding for the Byte attribute instead of a Base64 string

Expected Behavior

Given I create a DynamoDB Table with DynamoDB streams enabled.

When I PUT an item into that table having a Byte attribute (B)
And the Lambda plugged into that stream triggered.
Then I see a Base64 string attribute corresponding to the Byte attribute

Note that this is the expected behavior on AWS.

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

docker compose up

Environment

- OS: MacOsx M1
- LocalStack: 2.2.0 (latest)

Anything else?

When I enable debug mode with DEBUG=1, I can the following on localstack 2.2.0:

localstack    | 2023-08-02T18:13:58.793 DEBUG --- [functhread40] l.s.a.i.docker_runtime_exe : Sending invoke-payload
{
    "invoke-id": "509b6ea0-c342-4d72-9173-f58d8baaa6da",
    "invoked-function-arn": "arn:aws:lambda:eu-west-1:000000000000:function:OMITTED",
    "payload": "{\"Records\": [{\"eventID\": \"c1e58428\", \"eventVersion\": \"1.0\", \"awsRegion\": \"eu-west-1\", \"eventName\": \"INSERT\", \"eventSourceARN\": \"arn:aws:dynamodb:eu-west-1:000000000000:table/OMITTED/stream/2023-08-02T17:50:37.362\", \"eventSource\": \"aws:dynamodb\", \"dynamodb\": {\"ApproximateCreationDateTime\": 1690998667.0, \"Keys\": {\"pk\": {\"S\": \"lavender\"}, \"sk\": {\"N\": \"3093338870456911667\"}}, \"NewImage\": {\"pk\": {\"S\": \"lavender\"}, \"sk\": {\"N\": \"3093338870456911667\"}, \"created_on\": {\"S\": \"2023-08-02T17:51:07.473Z\"}, \"request\": {\"B\": \"\\n\\tFantastic\\u0012\\u0004feed\\u001a\\bmagnetic\\\"\\blavender*\\u001330933388704569116672\\u0014Money Market Account:\\btransmitB\\u0003HDDJ\\u0007invoiceP\\u000fZ\\u0004limeb\\u0006matrixj:\\n\\u000bAmeliorated\\u0012\\nwithdrawal\\u001a\\u0006Sports\\\"\\u0003PCI*\\u0005Spurs2\\u000btransparentr\\nSeychellesx\\u0004\\ufffd\\u0001\\u0003\\ufffd\\u0001\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\u0001\\ufffd\\u0001\\nconnecting\\ufffd\\u0001\\ufffd\\ufffd\\u00bf\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\u0001\\ufffd\\u0001\\fBedfordshire\\ufffd\\u0001:\\t/\\ufffduB\\ufffd?\\ufffd\\u0001\\r\\b\\u0001\\u0010\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\u0001\\ufffd\\u0001\\u0003\\ufffd\\u0001\\f\\b\\u0001\\u0010\\u0002 \\u0001(\\u00040\\u0002@\\u0004\\ufffd\\u0001\\u0001\\ufffd\\u0001\\rrevolutionary\\ufffd\\u0001\\u0003RSS\\ufffd\\u0001\\u000b\\b\\u0002\\u0011LsnU;u\\ufffd?\\ufffd\\u0001\\ufffd\\u0001\\n\\u0011clicks-and-mortar\\u0012\\nsynthesize\\u001a%\\n\\u0003USB\\u0012\\u0017Fantastic Frozen Gloves\\u001a\\u0005Zloty\\u001a-\\n\\u0006Israel\\u0012\\u0017Gorgeous Concrete Shoes\\u001a\\ne-business\\u001a1\\n\\ftransmitting\\u0012\\u001aSlovakia (Slovak Republic)\\u001a\\u0005index\\}"

Below is a JSON item example:

{
    "pk": {
      "S": "testPK"
    },
    "sk": {
      "S": "testSK"
    },
    "request": {
      "B": "VGhpc0lzQVRlc3Q="
    }
}

The problem here is that "B": "VGhpc0lzQVRlc3Q=" is getting sent as a string JSON: \"\\n\\tFantastic\\u0012\\u0004feed\\u001a\\bmagnetic\\\"\\blavender*\\u0013309333887045...

[TRAD-Anthony-CKO]

Hey @giograno
I opened an issue as per your recommendation, original issue here for ref

[TRAD-Anthony-CKO]

Hey, any updates on this? @baermat @giograno

[giograno]

Hi @TRAD-Anthony-CKO, sorry for the late answer. I briefly tried to replicate the issue but without luck so far. Would you please:

• make sure it is still reproducible with the last version of LocalStack (3.0.1);
• give us a minimal script that exposes the problem for you.

Thank you very much.

[TRAD-Anthony-CKO]

hi @giograno,

sorry for the late reply, the issue is indeed still happening with the latest version of localstack (3.0.1). Any Byte property of DynamoDB is getting sent as a deserialized JSON instead of a base64 string.

Regarding the script, easily reproducable with aws cli by invoking any default lambda (plugged into DDB streams) with aws cli (invoke-function):

{
    "pk": {
      "S": "testPK"
    },
    "sk": {
      "S": "testSK"
    },
    "request": {
      "B": "VGhpc0lzQVRlc3Q="
    }
}

and the lambda will receive: "\n\tFantastic\u0012\u0004feed\u001a\bmagnetic\"\blavender*\u0013309333887045 instead of VGhpc0lzQVRlc3Q=.

[giograno]

Hi @TRAD-Anthony-CKO, we released a fix in the latest image that tackles encoding issues (and moves to B64 encoding). Please give it a try and let us know if you have any additional issues. Thanks 🙏