Fix messages getting lost.

[monty241]

The logic of AddSingleMessage is unclear from the logic itself. In the previous version, the contents of the message would be replaced always be an error when the queue size is exceeded and the new text was being logged once every five minutes. In all other cases, no message was logged. In all cases, the original cases would be lost without displaying the amount lost.

The reason why messages are not being processed remains to be studied. It occurs despite sufficient sizing and several times a day in a large SAAS-environment.

Our local branch is too much off for easy patching, but the changes in AWSLoggerCore.cs are:

Header

        private static readonly DateTime MIN_TIMESTAMP_UTC_KIND = new DateTime(1, 1, 1, 0, 0, 0, DateTimeKind.Utc);

        /// <summary>
        /// Last time an error message was registered due to overflowing in-memory buffer.
        /// </summary>
        private DateTime _maxBufferTimeStamp = MIN_TIMESTAMP_UTC_KIND;

        /// <summary>
        /// Minimum interval in minutes between two error messages on in-memory buffer overflow.
        /// </summary>
        const double MAX_BUFFER_TIMEDIFF = 5;

AddSingleMessage

The deviating timestamp parameter can be removed, we use it to transport a deviating timestamp since the original event may have occurred minutes earlier due to internal queing of log messages.

        private void AddSingleMessage(string message, DateTime? timestamp)
        {
            DateTime timestampNn = timestamp ?? DateTime.Now;

            if (_pendingMessageQueue.Count > _config.MaxQueuedMessages)
            {
                if (_maxBufferTimeStamp.AddMinutes(MAX_BUFFER_TIMEDIFF) < DateTime.UtcNow)
                {
                    if (_maxBufferTimeStamp == MIN_TIMESTAMP_UTC_KIND)
                    {
                        string errorMessage = $"The AWS Logger in-memory buffer has reached maximum capacity of {_config.MaxQueuedMessages:N0} entries. Currently contains {_pendingMessageQueue.Count:N0} entries.";
                        LogLibraryServiceError(new System.InvalidOperationException(message));
                        _pendingMessageQueue.Enqueue(new InputLogEvent
                        {
                            Timestamp = timestampNn,
                            Message = errorMessage
                        });
                    }
                    _maxBufferTimeStamp = DateTime.UtcNow;
                }
            }

            _pendingMessageQueue.Enqueue(new InputLogEvent
            {
                Timestamp = timestampNn,
                Message = message,
            });
        }

[ashishdhingra]

Hi @monty241,

Good afternoon.

Could you please elaborate on the issue and what needs to be investigated, along with supporting code and detailed steps for reproduction?

Thanks,
Ashish

[github-actions]

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[monty241]

At least the call to _pendingMessageQueue.Enqueue is skipped when the queue is full. The rest remains unclear for reproducibility, but we have seen no more lost message since incorporating it into our fork.

[ashishdhingra]

At least the call to _pendingMessageQueue.Enqueue is skipped when the queue is full. The rest remains unclear for reproducibility, but we have seen no more lost message since incorporating it into our fork.

@monty241 Unfortunately, we would need consistent reproduction steps to investigate the issue further. And per your response, the issue is no longer reproducible at your end at this stage.

[github-actions]

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.