[BUG] Unrecognized character escape issue for stream data in DynamoDB source

[daixba]

Describe the bug
If the source item contains string with ', the final document in OpenSearch will have empty body.

After debug, An exception has been throw when converting the Stream event to Json event using jackson library.

com.fasterxml.jackson.core.JsonParseException: Unrecognized character escape ''' (code 39)
 at [Source: (String)"{"Content":"I\'m sorry, but I don\'t have access to that.","UserId":"1234","StartTime":"2023-10-13T08:00:29.818246","SessionId":"123"}"; line: 1, column: 16]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2477)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:750)
	at com.fasterxml.jackson.core.base.ParserBase._handleUnrecognizedCharacterEscape(ParserBase.java:1353)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._decodeEscaped(ReaderBasedJsonParser.java:2713)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:2233)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:2206)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:323)
	at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializerNR.deserialize(UntypedObjectDeserializerNR.java:82)
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:623)
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:449)
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3772)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3755)
	at org.opensearch.dataprepper.plugins.source.dynamodb.converter.StreamRecordConverter.convertData(StreamRecordConverter.java:107)
	at org.opensearch.dataprepper.plugins.source.dynamodb.converter.StreamRecordConverter.writeToBuffer(StreamRecordConverter.java:74)
	at org.opensearch.dataprepper.plugins.source.dynamodb.stream.ShardConsumer.run(ShardConsumer.java:247)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

To Reproduce
Steps to reproduce the behavior:

For example, the content attribute contains '.

{
 "SessionId": "123",
 "UserId": "1234",
 "Content": "I'm sorry, but I don't have access to that.",
 "StartTime": "2023-10-13T08:00:29.818246"
}

And the doc in OpenSearch will be

{
        "_index": "test-complex",
        "_id": "123|1234",
        "_score": 1,
        "_source": {}
      }

Expected behavior
The doc in OpenSearch should match to DynamoDB source.

Screenshots
N/A

Additional context
No issue in export.

[daixba]

Seems to be an issue with Jackson Mapper, should we set up something like mapper.configure(JsonParser.Feature.ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER, true);

[dlvenable]

Standard JSON does not require escaping single quotes. But, it appears that something is escaping them in the example you've provided.

"Content":"I\'m sorry, but I don\'t have access to that."

Is there a list of characters that DynamoDB will escape? And can we configure Jackson to allow allow or expect those?

[dlvenable]

Relatedly, what is the purpose of this exception handling?

data-prepper/data-prepper-plugins/dynamodb-source/src/main/java/org/opensearch/dataprepper/plugins/source/dynamodb/converter/StreamRecordConverter.java

Lines 108 to 110 in 43a0c75

	} catch (JsonProcessingException e) {
	return null;
	}

This results in data being dropped. The dynamodb source accepts the data and acts as if it were handled. But, then sends it to OpenSearch empty.

[dlvenable]

@daixba , This is a bug in the AWS SDK and appears to have been fixed in main. aws/aws-sdk-java-v2#4156

I'm unsure if it has been deployed. It is currently labeled as "pending-release".

[dlvenable]

Updating the AWS SDK to 2.21.23 does fix this bug.

[daixba]

hat is the purpose of this exception handling?

I remembered that I put a TODO here about what should we do if we failed to parse the json, but not sure when it's gone.

Thanks for the help on this.