SegmentDestination is causing huge CPU spikes when blocked by LittleSnitch

[grzegorzkrukowski]

Describe the bug
When using LittleSnitch to block traffic to Segement.io from our application, we are getting huge CPU Spikes really often, and when entering foreground/background - basically every time the flush is called.

I think there are several problems in SegmentDestination:

1. Storage can grow without limits - old events that are not sent are never removed.
2. Storage is keeping files infinitely - which is causing a snowball effect for every sync.
3. Flush logic is not well prepared for having a lot of data to work with.
   First of all, it read all the data from the filesystem every time it's called.
   Then for each of the files, it creates a network request straight away iterating over all of them - which in my case ends up being over 1500 requests created and failing in milliseconds. This is when CPU spikes are kicking in.
   The whole process is repeated every time the flush() is called and things are getting worse and worse over time when more un-send events are piling up in the storage.

To Reproduce
Steps to reproduce the behavior:

1. Install LittleSnitch
2. Block segment.io traffic in LittleSnitch
3. Create around 100.000 tracked events in Segment
4. Observe the CPUSpikes when flush logic is called.
5. Log an enormous amount of network connections created during the flush() call.

Expected behavior

1. If the user decides to block segment.io coming from our application using external software there should be no performance drop or CPU spikes coming out of this library.

2. Storage should not grow infinitely - at some point, older events should be removed when newer events are added - there should be the maximum amount of files that are allowed to be cached for sending.

3. flush() method should not go so aggressive with creating network events for all un-send files - instead it should either go one by one and read the next files only when the previous one succeeded or it should pre-check if segment batch API is reachable before trying to create a massive amount of requests.

Platform (please complete the following information):

• Library Version in use: 1.2.2
• Platform being tested: macOS
• Integrations in use: Amplitude

Workaround
As for now, I am using an ugly workaround from this fork that simply deleted all failed events if the error is URLError.networkConnectionLost - which is the case for LittleSnitch.

[bsneed]

Looking at this today.

[bsneed]

@grzegorzkrukowski could you have a look at or maybe try the PR above and see if it resolves the issue for you? Thanks!

[grzegorzkrukowski]

hey @bsneed I am afraid it won't really help in our case at least. Spikes will still happen every time application goes from background to foreground when flush is triggering independently from the timer itself.

The main problem I think is inside a flush execution itself. It opens all existing cached files and creates a network request for every single one. Those requests are then canceled straight away by Little Snitch.
If I keep going from background -> foreground that triggers flush() - it's a never-ending growing problem.

I think the logic should be rebuilt to handle a huge amount of events that will be never send properly - that's our case.
And currently, it leads to growing infinitely growing Storage and the flush method freezing the application when it executes. I am afraid it has nothing to do with a timer itself, sorry.

I think what will fix spikes is as suggested:

• take one event only, try to send it, and then if that worked, send another batch - instead of reading all existing cached batches and creating thousands of network requests at once
• on top of that to resolve a problem of infinite storage - I think there should be a maximum amount of files allowed only and in case of cache grow over it when creating new files some of the old ones should be discarded.

[bsneed]

Thanks for the quick response @grzegorzkrukowski, I'll revisit this and see how to make that happen. It's likely though that we'll separate out the storage thing into something else. We have SLAs around data loss and capping it would result in that so will continue thinking on how to best solve that problem separately.

[grzegorzkrukowski]

Ok, I understand your perspective. At the moment the biggest issue at least for us is flush method logic that is really heavy if you try to resend every cached batch on each flush call - creating all those network calls.
From my debugging process, it seems that it's more about network calls themselves being recreated so often and in such a huge amount when all of 1500 requests created fail straight away.
On top of that, each background/foreground switch triggers it again.

So for me, an acceptable solution would be to just avoid that and rewrite logic that will handle a huge amount of cached batches better than the current one - using fewer resources.

You can easily reproduce the problem by installing any network blocking software, blocking segment.io traffic, and then tracking a huge amount of events. Once you have that setup, calling flush and observing the activity monitor will show the mentioned problem. On top of that, we are also getting tons of network requests killed because they are created in a for-each loop that is for some of our users over a few thousand requests each time :/

My perspective is that we will never get any tracking from those users, but we don't want to have any performance issues because of their decision to block traffic. Even if we will lose some tracking by opt-in for some "limit" settings - it would be fine for me.

[bsneed]

Hi @grzegorzkrukowski! We looked into how we'd like to address this a little more and we think a stackable flush policy is probably the way to go, along with some access to the storage made public. I don't currently have an ETA on when this will be complete yet. It does look like you have a temporary workaround going at the moment though. We've made a ticket internally that links back to here and will continue moving forward.

[bsneed]

Happy to report this is fixed now.

Please see Analytics-Swift 1.4.4; Specifically there's an example flush policy implementation here: https://github.com/segmentio/analytics-swift/blob/main/Examples/tasks/NetBlockerFlushPolicy.swift

You can see it in use in the MacExample application.