IOptionsSnapshot Results in High CPU 100% at 1500 RPS with 15 Node Cluster (8core per vm 32GB Memory)

[justConfused]

If you believe you have an issue that affects the security of the platform please do NOT create an issue and instead email your issue details to secure@microsoft.com. Your report may be eligible for our bug bounty but ONLY if it is reported through email.

Describe the bug

IOptionsSnapshot Results in High CPU 100% at 1500 RPS with 15 Node Cluster (8core per vm 32GB Memory). HTTP.Sys Asp Net core 2.1

It looks like Wait on GC is blocking for some mutex as captured traces from azure profile show. Image below.

To Reproduce

Steps to reproduce the behavior:
1.) Start Azure Vm StandardDv2_v8
2.) 100 RPS
3.) HTTP.SYS(AspNetCore): Add IOptionsSnapshot as scoped depend so it created on ever request
4.) CPU should 100

Expected behavior

High CPU

Screenshots

[davidfowl]

IOptionsSnapshot<T> might be a performance trap indeed since binding isn't currently cached and runs per request:

cc @HaoK @Pilchie for 3.x
cc @ericstj @maryamariyan

This is something we might want to address in 5.0 or 6.0 and backport to 3.1.

[HaoK]

We don't use options snapshot anywhere by default right? But yeah config binding every request is going to be extremely slow

[ghost]

Tagging subscribers to this area: @maryamariyan
See info in area-owners.md if you want to be subscribed.

[BrennanConroy]

This is something we might want to address in 5.0 or 6.0

6.0 😄

[davidfowl]

@maryamariyan we should close as dupe of #36130