Logger does not seem to be reliable long-term

[adamrkerr]

I'm attempting to use the AWS Cloudwatch .Net Core logger in a process that handles large amounts of data in a multi-threaded fashion. We're hosting this process in ECS, so Cloudwatch seemed natural. I've tried a few things, such as registering my LoggerFactory as a transient service:

serviceCollection.AddTransient<ILoggerFactory>((s) => new LoggerFactory()
            .WithFilter(new FilterLoggerSettings
                {
                    { "Microsoft", LogLevel.Warning },
                    { "System", LogLevel.Warning }
                })
                .AddAWSProvider(
                new AWS.Logger.AWSLoggerConfig
                {
                    Region = "xxxxxx",
                    LogGroup = "xxxxxxxx",
                    MaxQueuedMessages = Int32.MaxValue,
                    BatchPushInterval = new TimeSpan(0, 0, 0, 1)
                },
                LogLevel.Debug
                )
                .AddConsole(LogLevel.Error)
            );

And also getting the logger as a function in hopes of creating it anew:

serviceCollection.AddSingleton<Func<ILogger>>((IServiceProvider provider) => { return () => provider.GetRequiredService<ILogger<App>>(); });

But it seems that no matter what, the logger eventually dies silently and won't come back. The application continues to run, but no more messages appear in the log.

This is an example of the messages I see in aws-logger-errors.txt:

Log Entry :
2/12/2018 10:41:49 PM
:
:Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at Amazon.Runtime.Internal.Transform.JsonErrorResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\Transform\JsonErrorResponseUnmarshaller.cs:line 103
at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.PutLogEventsResponseUnmarshaller.UnmarshallException(JsonUnmarshallerContext context, Exception innerException, HttpStatusCode statusCode)
at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.UnmarshallException(UnmarshallerContext input, Exception innerException, HttpStatusCode statusCode) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\Transform\ResponseUnmarshallers.cs:line 198
at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\HttpErrorResponseExceptionHandler.cs:line 78
--- End of inner exception stack trace ---

[normj]

We are working on hardening up the Core package so that intermittent errors pushing logs do not cause the background thread to die a permanent death.

[costleya]

Ihave deployed a fix for this in AWS.Logger.Core 1.6/1.7. You can get it by either updating your chosen logging dependency to latest (AWS.Logger.AspNetCore), or downloading version 1.7 of core directly.

Let me know if this does not resolve the issue.

[seanvander]

I am still experiencing similar problems with AWS.Logger.AspNetCore 1.2.7 using AWS.Logger.Core 1.1.8. Within a day or two, the majority of my streams stop updating.

Unfortunately I could not find any related or useful entries in aws-logger-errors.txt to attach.

If I restart my services then new streams are created as expected.