[Question] Best practices for throttling outgoing data?

[mmeluch]

Description/Screenshot
Recently, we've run into an issue where a very small percentage of users produced an excessive amount of logs due to a code error on our side - a 3rd party dependency was triggering a logged (trace) callback thousands of times per second.

Obviously, we should fix the issue on our side first, however, it made us think about avoiding such cases in the future. The most straightforward solution seems to be throttling of the outgoing user data, however, I am not sure how to approach this. Ideally, we want to keep the data complete for all non-abusive users, so no sampling (a critical message might get sampled out and we wouldn't be able to track down a technical issue on the customer's end).

My setup:

"@microsoft/applicationinsights-common": "^2.7.3",
"@microsoft/applicationinsights-react-js": "^3.2.3",
"@microsoft/applicationinsights-web": "^2.7.3"
"react": "^17.0.2",
"react-dom": "^17.0.2"

config: {
  instrumentationKey,
  eventsLimitInMem: 2000,
  isStorageUseDisabled: true,
  enableSessionStorageBuffer: false,
  maxBatchInterval: 10000,
  extensions: [this.reactPlugin],
  extensionConfig: {[this.reactPlugin.identifier]: { history: browserHistory }},
  enableUnhandledPromiseRejectionTracking: true
}

Additional Context
I am currently aware of the following settings related to data throttling:

• eventsLimitInMem: The number of events that can be kept in memory before the SDK starts to drop events. By default, this is 10,000.
• maxBatchInterval: How long to batch telemetry for before sending (milliseconds)
• Throttle Manager: I don't fully understand the settings here, but I might be able to achieve what I want with a reasonable combination of maxSendNumber (e.g. 1000) and dayInterval (e.g. 1/24 - hourly telemetry dispatches)?

I was thinking about lowering eventsLimitInMem, however, it still doesn't prevent our users from dispatching dozens/hundreds of individual tracking requests per second. I had no luck with maxBatchInterval either - it seems to act only as a guard against accumulating excessive data on the user's end (e.g. if the network is slow) and does not trigger between individual tracking calls if the user is online.

What seems to be missing here is something like a lodash throttle - a time-based interval which would allow users to only dispatch a limited amount of data once per a defined time interval (e.g. hourly or every minute or so).

• Can something like this be achieved with the currently available settings?
• Is the Throttle Manager component compatible with the other settings (eventsLimitInMem, maxBatchInterval)?
• Does the Throttle Manager configuration support intervals (dayInterval) shorter than 1 day? (e.g. 1/24 - an hourly rate)

[mmeluch]

It is possible that the duplicates are caused by #2205 or #2195 instead of our code. My original question is still relevant I think, though - as a defense mechanism against such cases in the future:

[MSNev]

It is possible that the duplicates are caused by #2205 or #2195 instead of our code. My original question is still relevant I think, though - as a defense mechanism against such cases in the future:

Since your using "@microsoft/applicationinsights-web": "^2.7.3" No, these issues where introduced in the v3.x code bases.

We currently don't provide any "out of the box" throttling from the client side as you point out that generally throttling on the client side may cause something critical to get dropped, and if your entire set of client users drop the event you could be missing something critical -- and as this would probably depend on your application the throttling rules could get complex. And of course once the page reloads all previous knowledge is lost (unless persisted in Session / Local storage / IndexedDb)

We do have a "limit" on the total number of "dependency" requests that would be sent for a single page load maxAjaxCallsPerView which just blocks all events after reaching this limit.

You can implement your own "throttling" logic via a Telemetry Initializer, while it's primary use-case is to allow decorating / modifying the event before it gets batched, if you return false (explicit not just something falsely like 0 or undefined) then the event will be dropped and never batched / sent.

eventsLimitInMem - This is generally used to limit the amount of memory consumed within the SDK, so it's not really designed as a throttling mechanism to stop the SDK from crashing the browser if for some reason there are excessive events or it can't send the events out the door.

maxBatchInterval - This is just the amount of time it "waits" after batching the first event before it sends out the batched events.

And the Throttle Manager is really about allowing the throttling of specific generated messages (currently by the SDK) so rather than sending out an event for every user / every page load it's configurable to allow a lower percentage of events (so only send out a event once per month and only on a monday -- this is one of the configurable throttling) -- but this doesn't apply to "normal" automatic events. If your firing custom events, then yes you could configure and use this yourself -- but I'm guessing that's not really what your asking.

Can something like this be achieved with the currently available settings?

Not by default, but you could use a telemetry initializer fairly simply to duplicate this behavor.

Is the Throttle Manager component compatible with the other settings (eventsLimitInMem, maxBatchInterval)?

There are different, in that you would manually "ask" the throttle manager whether it can send an event and if you do it's batched with the other configs.

Does the Throttle Manager configuration support intervals (dayInterval) shorter than 1 day? (e.g. 1/24 - an hourly rate)

Tagging @Karlie-777, this sounds like a possible enhancement that we could consider, can you please create a GitHub enhancement issue (tagging this one) so that we can keep it on our backlog, rather than just tagging this one as this issue will probably have more discussion than just providing the ability for limiting hourly / minute -- or we may create a new specific Client side event throttler.

[Karlie-777]

[like] Karlie Li reacted to your message:

…

________________________________ From: Nev ***@***.***> Sent: Monday, April 8, 2024 4:22:29 PM To: microsoft/ApplicationInsights-JS ***@***.***> Cc: Mention ***@***.***> Subject: Re: [microsoft/ApplicationInsights-JS] [Question] Best practices for throttling outgoing data? (Issue #2322) It is possible that the duplicates are caused by #2205<#2205> or #2195<#2195> instead of our code. My original question is still relevant I think, though - as a defense mechanism against such cases in the future: Since your using ***@***.***/applicationinsights-web": "^2.7.3" No, these issues where introduced in the v3.x code bases. We currently don't provide any "out of the box" throttling from the client side as you point out that generally throttling on the client side may cause something critical to get dropped, and if your entire set of client users drop the event you could be missing something critical -- and as this would probably depend on your application the throttling rules could get complex. And of course once the page reloads all previous knowledge is lost (unless persisted in Session / Local storage / IndexedDb) We do have a "limit" on the total number of "dependency" requests that would be sent for a single page load maxAjaxCallsPerView which just blocks all events after reaching this limit. You can implement your own "throttling" logic via a Telemetry Initializer<https://github.com/Microsoft/ApplicationInsights-JS?tab=readme-ov-file#telemetry-initializers>, while it's primary use-case is to allow decorating / modifying the event before it gets batched, if you return false (explicit not just something falsely like 0 or undefined) then the event will be dropped and never batched / sent. eventsLimitInMem - This is generally used to limit the amount of memory consumed within the SDK, so it's not really designed as a throttling mechanism to stop the SDK from crashing the browser if for some reason there are excessive events or it can't send the events out the door. maxBatchInterval - This is just the amount of time it "waits" after batching the first event before it sends out the batched events. And the Throttle Manager<https://github.com/microsoft/ApplicationInsights-JS/blob/main/docs/ThrottleMgr.md> is really about allowing the throttling of specific generated messages (currently by the SDK) so rather than sending out an event for every user / every page load it's configurable to allow a lower percentage of events (so only send out a event once per month and only on a monday -- this is one of the configurable throttling) -- but this doesn't apply to "normal" automatic events. If your firing custom events, then yes you could configure and use this yourself -- but I'm guessing that's not really what your asking. Can something like this be achieved with the currently available settings? Not by default, but you could use a telemetry initializer fairly simply to duplicate this behavor. Is the Throttle Manager component compatible with the other settings (eventsLimitInMem, maxBatchInterval)? There are different, in that you would manually "ask" the throttle manager whether it can send an event and if you do it's batched with the other configs. Does the Throttle Manager configuration support intervals (dayInterval) shorter than 1 day? (e.g. 1/24 - an hourly rate) Tagging @Karlie-777<https://github.com/Karlie-777>, this sounds like a possible enhancement that we could consider, can you please create a GitHub enhancement issue (tagging this one) so that we can keep it on our backlog, rather than just tagging this one as this issue will probably have more discussion than just providing the ability for limiting hourly / minute -- or we may create a new specific Client side event throttler. — Reply to this email directly, view it on GitHub<#2322 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AS7LF2VRWJCWAHQU4VC43U3Y4K74LBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVAZTGMZXGMZDSMUCUR2HS4DFUVUXG43VMWSXMYLMOVS2UMRSGMYDMMZUGM3DJJ3UOJUWOZ3FOKTGG4TFMF2GK>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[mmeluch]

@MSNev thanks for the elaborate answer, it clarified many things which I initially misunderstood from the docs.

I think I should be able to implement my own client-side throttling now or at least utilize some kind of network-level brute-force protection by routing the the telemetry via our own servers first (endpointUrl).