Amazon.Lambda.KafkaEvents.KafkaEvent cannot be deserialized

[gabriel-myob]

Description

The latest version of Amazon.Lambda.KafkaEvents.KafkaEvent cannot deserialize events when using SelfManagedKafka as source events.

Reproduction Steps

The hander function signature looks like

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
...
...
    public async Task FunctionHandler(KafkaEvent kafkaEvent, ILambdaContext context){...}

Logs

Error converting the Lambda event JSON payload to type Amazon.Lambda.KafkaEvents.KafkaEvent: The JSON value could not be converted to System.String. Path: $.records.TOPIC[0].partition | LineNumber: 0 | BytePositionInLine: 377.: JsonSerializerException
at Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer.Deserialize[T](Stream requestStream)
at lambda_method(Closure , Stream , Stream , LambdaContextInternal )

at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& readStack, Utf8JsonReader& reader, Exception ex)
at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)
at System.Text.Json.JsonSerializer.ReadCore(Type returnType, JsonSerializerOptions options, Utf8JsonReader& reader)
at System.Text.Json.JsonSerializer.ParseCore(ReadOnlySpan1 utf8Json, Type returnType, JsonSerializerOptions options) at System.Text.Json.JsonSerializer.Deserialize[TValue](ReadOnlySpan1 utf8Json, JsonSerializerOptions options)
at Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer.Deserialize[T](Stream requestStream)
Cannot get the value of a token type 'Number' as a string.: InvalidOperationException
at System.Text.Json.Utf8JsonReader.GetString()
at System.Text.Json.Serialization.Converters.JsonConverterString.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
at System.Text.Json.JsonPropertyInfoNotNullable`4.OnRead(ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.JsonPropertyInfo.Read(JsonTokenType tokenType, ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)

Environment

• Build Version: https://www.nuget.org/packages/Amazon.Lambda.KafkaEvents/1.0.0
• OS Info: MacOs BigSur 11.6.1
• Build Environment: cli/terminal
• Targeted .NET Platform: .NET Core 3.1 (C#/PowerShell)

Resolution

I noticed that there has been a change to the type Partition property - used to be int now is string. Any reason for that?
https://github.com/aws/aws-lambda-dotnet/blame/b49dfae373538aad2811340cd2a03b478cb59318/Libraries/src/Amazon.Lambda.KafkaEvents/KafkaEvent.cs#L46

---

[ashishdhingra]

Hi @gabriel-myob,

Good morning.

Thanks for reporting the issue. Using Lambda with self-managed Apache Kafka and Using Lambda with Amazon MSK provide some details on sample JSON for Kafka event, where it shows partition as of type string (from example). The Amazon.Lambda.KafkaEvents package was added as part of PR #984.

For testing, would you be able to serialize the incoming Kafka event object as JSON, log the same to CloudWatch and share it in this issue for analysis? Not sure if partition type is incorrectly shown as string in the above references (Apache Kafka reference https://kafka.apache.org/21/javadoc/org/apache/kafka/clients/consumer/ConsumerRecord.html lists partition of type int).

Thanks,
Ashish

[gabriel-myob]

Hi @ashishdhingra,

Thank you for looking into it.
Here is the JSON event.

{
  "eventSource": "SelfManagedKafka",
  "bootstrapServers": "xxx",
  "records": {
    "TOPIC-28": [
      {
        "topic": "TOPIC",
        "partition": 28,
        "offset": 0,
        "timestamp": 1643766098825,
        "timestampType": "CREATE_TIME",
        "key": "base64-key",
        "value": "base64-value"
      }
    ]
  }
}

Not sure if it is relevant, Kafka version is 2.6.2

[ganeshnj]

@gabriel-myob It seems there is inconsistency between the payload you are receiving (partition is int) and docs (partition is string).

I quickly checked Java code, and they appear to model partition as int as well.

This should be an easy fix but we should make sure it works with a real Kafka event.

[LintKicker]

Our team is also running into this issue. Is there any known workaround or a timeline for when this will be fixed?

[ashishdhingra]

P61053095

[ashishdhingra]

Got confirmation from service team that partition should be modeled as int. They will work to update the documentation (cannot share the ETA):

• Using Lambda with self-managed Apache Kafka
• Using Lambda with Amazon MSK

We would work in fixing this in Amazon.Lambda.KafkaEvents package.