Issue with unhandled exception message been multilined

[svoychik]

Describe the bug

As a result of unhandled exceptions, logs are written into Cloudwatch in multiline format.
#1165 - previous attempt to fix the issue didn't help so this probably has to be double-checked

For us, this issue causes quite a lot of garbage logs, which distracts attention from really important things

Expected Behavior

The exception message is stored as a single CloudWatch row. That's how it worked before on .NET 3.1

Current Behavior

Currently, exception logs are multilined, see screenshot

Reproduction Steps

You need to use lambda with .NET 6 AWS-supported runtime.
Then make that lambda fail - Throw any exception from lambda ExecuteAsync method and open CloudWatch logs to see the multilined exception logs

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

Amazon.Lambda.Core 2.1.0
Amazon.Lambda.Serialization.Json 2.0.0
Amazon.Lambda.APIGatewayEvents 2.4.1
Amazon.Lambda.Logging.AspNetCore 3.1.0
AWSSDK.CloudWatch 3.7.4.25
AWSSDK.DynamoDBv2 3.7.3.50
AWSSDK.StepFunctions 3.7.0.174

Targeted .NET Platform

.NET 6

Operating System and version

.NET 6 runtime

[normj]

Hi @svoychik

I pinged the Lambda team on the status of the deployment and they have been doing a lot of internal changes to how their runtime deployments. The side effect is the change made to the .NET runtime client hasn't been deployed yet to the managed runtime. Sorry for the delay.

[svoychik]

Hi @normj
Thanks for the response.
Is there a place where I can track the latest updates to the .NET runtime in terms of the version of packages to not spend a lot of time trying to understand whether the Lambda team released the fix or not?

Also, considering the speed of delivery (more than 3 months), it looks like a big reason to think about whether we really want to use custom runtime or the one provided by AWS 🤔.
I'd like to know the Lambda team's vision on the processes. Are such long delays still going to happen?

[DillonN]

Seeing the same issue on .NET 6 still. We did manage to find a workaround, but it's not that pretty. Basically implement a quick custom ILogger implementation that logs to Console.Out, but replaces all \n characters with \r. CloudWatch seems to pick up on the carriage return as an internal line break and formats the messages as expected.

E: Adding some context on the workaround since there was interest:

This really only applies if you're using the same Amazon.Lambda.Logging.AspNetCore mentioned in OP. It's a lightweight library from this repo that implements a Microsoft.Extensions.Logging provider using ILambdaLogger. Since ILambdaLogger stopped formatting multi-line messages properly with .NET 6, so too did this library.

The workaround is to swap out the logging library with one that logs to std out, e.g. Microsoft.Extensions.Logging.Console. Then modify this new library to replace all \n with \r in every log message. If you use the same ILogger[<T>] interface everywhere, this is a drop-in replacement.

But unfortunately you can't actually use Microsoft.Extensions.Logging.Console, since the relevant code is internal and sealed. What you can do is implement a custom logging provider that makes the \n->\r swap. This ends up being two classes to implement ILogger and ILoggerProvider. Inspiration on e.g. formatting can be taken from the Amazon.Lambda.Logging.AspNetCore implementation. Keep in mind logging can be a sensitive component, make sure everyone understands the risks of rolling a custom implementation. Also, you'll probably want the \n->\r thing turned off in local debugging.

This workaround came up years ago for a custom .NET Core 3.1 runtime, but after migrating to 3.1 official it wasn't necessary. After dusting it off to try on this problem in .NET 6, it works!

[BenjaRav]

@ashishdhingra I see this is marked as queued now. Is it possible to have an estimate for how long tasks like this usually resides in a queued state, before being started? I'm contemplating continuing to try to implement the fix Dillon suggested here, but i feel it will be wasted if a internal fix from AWS is around the corner.

[BenjaRav]

Any updates on this?

[ashishdhingra]

The changes are pushed, closing this issue. Feel free to open a new one in case of any issue.