How to convert DynamoDBEvent images to DynamoDBv2.Document

[dscpinheiro]

Discussed in #1654

^{Originally posted by Dreamescaper January 16, 2024}
I have a lambda subscribed to dynamoDb events. I want to convert event's records to regular JSON, currently we use something like that:

public async Task FunctionHandler(DynamoDBEvent input, ILambdaContext context)
{
        foreach (var record in input.Records)
        {
                var image = record.Dynamodb.NewImage;
                var document = Document.FromAttributeMap(image);
                var json = document.ToJson();
                /*   ....    */
        }
}

However, it doesn't work after the #1648 is merged.
Could anyone suggest what is the easiest way to do same after the update?

[adam-knights]

We've just hit this as well. We can no longer do

var document = Document.FromAttributeMap(attributeValues);
return _dynamoDBContext.FromDocument<MyObject>(document);

I can understand wanting to break the dependency but its weird to have AttributeValue duplicated as different classes across the two libraries, is the plan to have it live in a shared model library?

[ashovlin]

We've released version 3.1.0 of Amazon.Lambda.DynamoDBEvents, which adds a ToJson and ToJsonPretty that can be used with DynamoDBEvent.

We still kept the event definition separate from the SDK definition in AWSSDK.DynamoDBv2, which we split in version 3.0.0 via #1648. This avoids some interaction with code relevant to the SDK but not to Lambda (request marshallers), and may reduce the package size for cases where one only needs to read the event without using the full DynamoDB SDK.

You can now use the ToJson method on either OldImage or NewImage to:

1. Convert to JSON
2. Convert it to the SDK's Document
3. Convert it to the SDK's object persistence classes.

Note that the JSON conversion has the same limitations as the SDK: the sets (SS, NS, BS) will be converted to JSON arrays, and binary (B) will be converted to Base64 strings

foreach (var record in dynamoEvent.Records)
{
    // Convert the event to a JSON string
    var json = record.Dynamodb.NewImage.ToJson();

    // Which you can convert to the mid-level document model
    var document = Document.FromJson(json);
    
    // And then to the high-level object model using an IDynamoDBContext
    var myClass = context.FromDocument<T>(document);
}

I think that will address both use cases reported above, but let us know if you're still seeing limitations after 3.1.0. Thanks.

[psdanielhosseini]

@ashovlin Thanks, everything worked great!

However, we have a scenario (testing) where we are building a DynamoDB stream event based of an object - see code

 public static DynamoDBEvent CreateDynamoDbEvent(OperationType type, object newObj, object prevObj)
    {
        return new DynamoDBEvent
        {
            Records = new List<DynamodbStreamRecord>
            {
                new()
                {
                    Dynamodb = new DynamoDBEvent.StreamRecord
                    {
                        NewImage = ToDynamoDbAttributes(newObj),
                        OldImage = ToDynamoDbAttributes(prevObj)
                    },
                    EventName = new OperationType(type)
                }
            }
        };
    }

private static Dictionary<string, DynamoDBEvent.AttributeValue> ToDynamoDbAttributes(object obj)
    {
        if (obj == null) return null;

        var attributes = Document.FromJson(obj.ToJson()).ToAttributeMap();
        return attributes;
    }

Now I want to return an Dictionary<string, DynamoDBEvent.AttributeValue, but not really sure how I would go about it now that ToAttributeMap() returns a different AttributeValue. What's the best approach here?

[adam-knights]

The only thing we spotted so far is that one of our unit tests shows An exception of type 'System.InvalidOperationException' occurred in System.Text.Json.dll but was not handled in user code: 'Cannot write a JSON property name following another property name. A JSON value is missing.'

I think this might be caused by the WriteJsonValue of

aws-lambda-dotnet/Libraries/src/Amazon.Lambda.DynamoDBEvents/ExtensionMethods.cs

Line 65 in 9569c1a

private static void WriteJson(Utf8JsonWriter writer, Dictionary<string, AttributeValue> item)

- namely there's no final else, so we get the error in the stack:

   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource, Int32 currentDepth, Int32 maxDepth, Byte token, JsonTokenType tokenType)
   at System.Text.Json.Utf8JsonWriter.WriteStringByOptionsPropertyName(ReadOnlySpan`1 propertyName)
   at System.Text.Json.Utf8JsonWriter.WritePropertyName(ReadOnlySpan`1 propertyName)
   at Amazon.Lambda.DynamoDBEvents.ExtensionMethods.WriteJson(Utf8JsonWriter writer, Dictionary`2 item)
   at Amazon.Lambda.DynamoDBEvents.ExtensionMethods.ToJson(Dictionary`2 item, Boolean prettyPrint)
   at Amazon.Lambda.DynamoDBEvents.ExtensionMethods.ToJson(Dictionary`2 item)

This was actually a 'bug' in our unit test where we were setting up an AttributeValue in an image Dictionary incorrectly

as new DynamoDBEvent.AttributeValue { N = null } or new DynamoDBEvent.AttributeValue()

So like:

public static void Main()
{
	var image = CreateBasicImage();
	var json = image.ToJson();
	Console.WriteLine(json);
}

private static Dictionary<string, DynamoDBEvent.AttributeValue> CreateBasicImage() =>
        new()
        {
            { "PublishThing", new DynamoDBEvent.AttributeValue { N = null } }, // Or new DynamoDBEvent.AttributeValue()
	    { "SomeDefaultThing", new DynamoDBEvent.AttributeValue { S = "foo" } }
        };

See https://dotnetfiddle.net/Fomy9m

So reporting as this was a change in behaviour for us versus FromAttributeMap, and also reporting incase you want to handle this in some way in your WriteJsonValue, rather than allowing STJ to fail. But we've fixed our test and are good.

[ashovlin]

1. @psdanielhosseini - ah, I see. So you're ultimately trying to go from a JSON string to Dictionary<string, DynamoDBEvent.AttributeValue>. But this is no longer possible since the FromJson -> Document -> ToAttributeValues path produces the SDK's AttributeValue, not the newly defined DynamoDBEvent.AttributeValue in the event package.

   I'll take this back to the team for prioritization, we were initially focused on ToJson since that seemed more relevant to production code, but I understand now how this is problematic for tests.

2. @adam-knights - that was inadvertent, thanks for the report. I'll take a look at better handling the null case.