Dropped message because queue is too full

[mabead]

After many years of using Segment, I just realized that we sometimes call the "Identify" or "Track" method and events are lost because of this code (note that we are using version 3.3.1-alpha):

So no exceptions are raised. There is only a warning being triggered. Pretty annoying. For me, it's like doing an SQL INSERT where the database answered "success" when in fact it did "well you know, I was too busy. I just dropped your request... without letting you know. Oh yeah, I sent you a letter about it. If you're lucky you will receive it. But note that the letter will not let you know what I dropped exactly. It will just let you know that I was too busy."

What is your suggestion to make sure that no events are lost?

[mabead]

I see that version 3.4.1-alpha changed this behavior:

So this looks interesting but 3.4.1:

• is alpha
• has only 84 downloads

So my questions are:

• is 3.4.1-alpha ready to be used in production?
• do you intend to release non-alpha versions? The last non-alpha version was made in 2017, about 3 years ago.

[lubird]

Thanks for your questions. We did recently release 3.4.1-alpha and I was about to recommend you try it out to see if it has resolved your issues. Keep in mind that 3.3.1 has also always only been in alpha state, so the answer to "is it ready to be used in production?" should be the same across the two versions.

After we see more traffic through 3.4.1 and build more confidence in its reliability, we do intend to take it out of alpha in the future.

[mabead]

Since you offer no alternatives and no woraround, I will unfortunately have to do so.

That being said, I must say that the outcome is pretty dissaponting:

• you don't try to reproduce the issue
• you give me no confidence that this issue is resolved in the new code. A link with a PR that explains that the issue was solved would have been appreciated.
• you give me no confidence that the 3.4.0-alpha version is stable

It's even more dissapointing given the fact that the library is related to a product that we pay for.

[mabead]

@lubird I confirm that 3.4.0-alpha still drops messages. I don't have the log Dropped message because queue is too full anymore. The messages are just lost silently.

I did a burst of 4402 track + 4402 identify calls and only 360 of the track events made their way to amplitude.

I suggest that you look at the BlockingCollection class: https://docs.microsoft.com/en-us/dotnet/standard/collections/thread-safe/blockingcollection-overview

It usually yields simpler and safer code.

[mabead]

As a workaround, I tried to use the Segment library in synchronous mode:

var segmentConfig = new SegmentSdk.Config();
segmentConfig.SetAsync(false);

My code that sends burst of analytics is part of a nightly job that wants to send more than 5000x identify and 5000 x track events. So, now that I am in synchronous mode, it takes a very long time to send these 10000 messages to the Segment server. Furthermore, my code runs in an AWS lambda which is limited to 15 minutes. Given the high number of messages and the time constraints that I have, all messages can't be sent in 15 minutes.

So, using the synchronous mode is not an option for me.

[mabead]

I now have a successfull work around. Basically, the algorithm is the following:

Configure segment in async mode
WHILE (not done)
   send batch of 40 events
   flush segment queue
END WHILE

I created a burst of 1114 track + 1114 identify. With this workaround in place, 1113 identify made their way to Segment. So it's not perfect, but it's much better than before.

[lubird]

Thanks for continuing to follow up on this thread @mabead. We appreciate you reporting this issue as well as sharing your work-arounds.

I've noted the issue on our backend. We'll for sure take a look at this when we can.

In the meantime, I encourage you to submit a PR for this issue if you feel like you have a workable fix you can share with the community!

[mabead]

Sorry @lubird but since Segment is a product that we pay for, I am not very keen on submitting PRs. Furthermore, the fix is most likely not trivial and would require more than a few hours.

[cgalan-applaudostudios]

@mabead
I tried to replicate your scenario but I wasn't able, could you share your environment (framework version, OS, etc)? I have attached the results. I hope it could help you.
Segmen_ MaxQueueSize test.docx

[mabead]

I wrote this little utility to try to replicate the issue: https://github.com/mabead/Analytics.NET.Repro_issue_138

Here's what I get:

Analytics.NET Version	nb of events sent	nb of events seen in amplitude	nb of events lost
3.4.2-beta	12000	12000	0
3.4.1-alpha	12000	12000	0
3.3.1-alpha	12000	5361	6639

We therefore see that both 3.4.1-alpha and 3.4.2-beta deliver 100% of the events to Segment and then to Amplitude while 3.3.1-alpha is losing many events.

Note that I initially incorrectly reported that 3.4.1-alpha was also losing messages. I think that I now know where this error came from: after I did a burst of events, it took more than 30 minutes for all the events to be visible in Amplitude. Maybe I just didn't wait long enough.

Thanks for your help on this. We will definitely switch to 3.4.2-beta.