Events not sent to Segment servers: "Unexpected Status Code"

[mabead]

In order to get the fix for #156, I used version 3.7.0 of Analytics.Net. Unfortunately, I still get some errors that tell me that messages are lost.

First, here's what my Failed callback looks like:

private void Initialize()
{
    // ...
    client = new SegmentSdk.Client(...);
    client.Failed += LogFailure;
    // ...
}

private void LogFailure(SegmentSdk.Model.BaseAction action, Exception e)
{
    string details = null;
    if (e != null)
    {
        details = e.Message;
    }

    _logger.LogError(LoggingEvents.SegmentFailure, e, $"Failed to send analytic event {action.Type} for user {action.UserId} and message {action.MessageId} to segment.io server. Details: {details}");
}

With this, I got these error logs after a few hours:

Failed to send analytic event track for user 831ba960-a73c-49b7-be6b-771d2233ce2d and message 5c72ec76-0044-407b-8c69-9a8c3edb9aef to segment.io server. Details: Status Code: 0, Message: Has Backoff reached max: True with number of Attempts:8,
 Status Code: 0
, response message: 
Segment.Exception.APIException: Status Code: 0, Message: Has Backoff reached max: True with number of Attempts:8,
 Status Code: 0
, response message: 

Failed to send analytic event identify for user 831ba960-a73c-49b7-be6b-771d2233ce2d and message 2a2d3b52-2b01-4b21-9178-4876f1e725e8 to segment.io server. Details: Status Code: 0, Message: Has Backoff reached max: True with number of Attempts:8,
 Status Code: 0
, response message: 
Segment.Exception.APIException: Status Code: 0, Message: Has Backoff reached max: True with number of Attempts:8,
 Status Code: 0
, response message: 

Is there anything that I could do to help you troubleshoot this?

[mabead]

I find it weird that we see:

• number of Attemps = 8: this implies that _backo._attempt = 8
• Has Backed reached max = true : this implies that _currentAttemptTime >= _max which is the same as _currentAttemptTime >= 10000.
  I am not sure that I understand the code correctly but it lets me think that you will wait/retry for at most 10 seconds. This may be a little short if you eventually hit an http timeout.

[mabead]

In case it can help, I have attached somme logs coming from Analytics.Net before/after the error.

loggly_grid_2021-03-30 12_20_25.408029.log

[mabead]

The batch that failed to be sent is the one with id 44eb04cb-3947-4c35-b817-ba91f28ac983. I searched in my logs for this specific ID and I only found this:

It therefore looks to me like it was only retried once and not 8 times.

[mabead]

Once again, I suggest that you add logs here.

This is probably hidding something.

[juliette-ruchel]

@mabead May I ask if you have any custom settings for the configuration of the client or are you just using the default values of the configuration?

[mabead]

@juliette-ruchel Here's how we initialize the segment client:

using SegmentSdk = Segment;
// ...
private void Initialize()
{
    var segmentConfig = new SegmentSdk.Config()
        .SetAsync(_configuration.SendMessagesAsynchronously); // SendMessagesAsynchronously = true

    if (!string.IsNullOrEmpty(_configuration.DotNetBackEndSourceKey))
    {
        LandrMusicClient = new SegmentSdk.Client(_configuration.DotNetBackEndSourceKey, segmentConfig);
        LandrMusicClient.Failed += LogFailure;
    }

    SegmentSdk.Logger.Handlers += LogSegmentEvent;
}

private void LogFailure(SegmentSdk.Model.BaseAction action, Exception e)
{
    string details = null;
    if (e != null)
    {
        details = e.Message;
    }

    _logger.LogError(LoggingEvents.SegmentFailure, e, $"Failed to send analytic event {action.Type} for user {action.UserId} and message {action.MessageId} to segment.io server. Details: {details}");
}

private void LogSegmentEvent(SegmentSdk.Logger.Level level, string message, IDictionary<string, object> args)
{
    var fullMessage = message;

    if (args != null)
    {
        fullMessage += " " + string.Join(", ", args.Select(x => $"{x.Key}: {x.Value}"));
    }

    switch (level)
    {
        case SegmentSdk.Logger.Level.DEBUG:
            _logger.LogDebug(LoggingEvents.SegmentLog, fullMessage);
            break;
        case SegmentSdk.Logger.Level.INFO:
            _logger.LogInformation(LoggingEvents.SegmentLog, fullMessage);
            break;
        case SegmentSdk.Logger.Level.WARN:
            _logger.LogWarning(LoggingEvents.SegmentLog, fullMessage);
            break;
        case SegmentSdk.Logger.Level.ERROR:
            _logger.LogError(LoggingEvents.SegmentLog, fullMessage);
            break;
        default:
            _logger.LogError(LoggingEvents.SegmentLog, $"Unknown segment logging level {level} => {fullMessage}");
            break;
    }
}

We are therefore using the default values.

[juliette-ruchel]

Thanks for the information!
I believe the issue may be related to the response being null thus the status code is returned is 0.
This may mean that we are entering either the HttpRequestException o TaskCancelledException catch. We are working on adding a few more logs to determine the exact issue.

[pooyaj]

hey @mabead, would you mind doing a quick sanity check of this #164 to confirm it solves this issue?

[mabead]

@pooyaj I could give it a try but I don't see a new version of Analytics.net available on nuget. As soon as you have a new version, I will be able to try it out.