AzureBlobStorageClient: unsynchronized Dictionary causes infinite loop and 100% CPU under concurrent requests

[david-axelson]

Description

Impact

On Sunday morning, July 12, 2026, our production web server went to 100% CPU and the site became slow and unreliable.

The problem began around 4:00 AM local, right after the nightly application restart, and CPU climbed steadily until it saturated near 6:30 AM. The site did not recover on its own. It took a manual application pool recycle (around 7:50 AM) to bring CPU back to normal, so someone had to be pulled in to intervene early on a Sunday. Tellingly, the requests that hung during the incident were image requests for the site's livestream page, which is exactly the page online worshippers rely on for Sunday services.

The most serious part is that this is a race condition that can trigger on any application restart under load. This time it saturated during the pre-service hours and was cleared before the main services, but the same bug could just as easily peg the server in the middle of a live Sunday service. That is why we are reporting it even though a recycle clears the immediate symptom.

What we found

Rock.Storage.Common.AzureBlobStorageClient is a process-wide singleton that caches BlobContainerClient instances in a plain, non thread-safe Dictionary<int, BlobContainerClient>. GetBlobClient(...) reads and writes that dictionary with no synchronization. Under concurrent requests the dictionary's internal state can be corrupted, after which Dictionary.FindEntry enters an infinite loop and every subsequent call spins a CPU core indefinitely. That infinite loop is the 100% CPU.

Any site using the Azure Blob Storage provider for one or more BinaryFileType records (for example, images served through GetImage.ashx) is exposed. The window is largest immediately after an application restart, when the cache dictionary is empty and multiple threads race to populate it.

Affected component

• File: Rock/Storage/Common/AzureBlobStorageClient.cs
• Method: GetBlobClient( string accountName, string accountKey, string customDomain, string containerName, string blobName )

Root cause

The singleton shares mutable state without synchronization:

private static readonly AzureBlobStorageClient _instance = new AzureBlobStorageClient();   // singleton (line 36)

private Dictionary<int, BlobContainerClient> _containerClients
    = new Dictionary<int, BlobContainerClient>();                                          // shared, not thread-safe (line 46)

public BlobClient GetBlobClient( string accountName, string accountKey, string customDomain, string containerName, string blobName )
{
    _httpClient = _httpClient ?? new HttpClient();                                         // secondary race (line 59)

    var hashKey = ( accountName + accountKey + customDomain + containerName ).GetHashCode();
    if ( !_containerClients.ContainsKey( hashKey ) )                                       // read via FindEntry (line 62)
    {
        // ... build connection string and options ...
        _containerClients.Add( hashKey, containerClient );                                 // write (line 77)
    }

    return _containerClients[hashKey].GetBlobClient( blobName );                           // read (line 80)
}

Dictionary<TKey, TValue> does not support concurrent read and write. When one thread executes Add while others execute ContainsKey or the indexer, the internal bucket chain can be corrupted into a cycle. From that point on FindEntry never terminates and the calling thread spins at 100% CPU. Because the class is a singleton, the corrupted dictionary lives for the life of the process, so only an application pool recycle clears it.

Evidence

Captured from a production incident. Of 141 request threads aborted at the ASP.NET execution timeout, 140 had this identical stack. The remaining 1 was in Dictionary.Insert, that is, the thread performing the corrupting write:

System.Threading.ThreadAbortException
   at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   at Rock.Storage.Common.AzureBlobStorageClient.GetBlobClient(String accountName, String accountKey, String customDomain, String containerName, String blobName) in \Rock\Storage\Common\AzureBlobStorageClient.cs:line 62
   at Rock.Storage.Provider.<Azure blob storage provider content fetch>
   ... (BinaryFile content requested through GetImage.ashx)

Proposed fix

Replace the unsynchronized Dictionary with a ConcurrentDictionary and use GetOrAdd. Initialize the shared HttpClient once at construction to remove the secondary lazy-initialization race. The class is internal sealed, so there is no public API change.

private readonly HttpClient _httpClient = new HttpClient();

private readonly ConcurrentDictionary<int, BlobContainerClient> _containerClients
    = new ConcurrentDictionary<int, BlobContainerClient>();

public BlobClient GetBlobClient( string accountName, string accountKey, string customDomain, string containerName, string blobName )
{
    var hashKey = ( accountName + accountKey + customDomain + containerName ).GetHashCode();

    var containerClient = _containerClients.GetOrAdd( hashKey, _ =>
    {
        var connectionString = $"DefaultEndpointsProtocol=https;AccountName={accountName};AccountKey={accountKey}";
        if ( !string.IsNullOrWhiteSpace( customDomain ) )
        {
            connectionString = $"{connectionString};BlobEndpoint={customDomain}";
        }

        var clientOptions = new BlobClientOptions
        {
            Transport = new Azure.Core.Pipeline.HttpClientTransport( _httpClient )
        };

        return new BlobContainerClient( connectionString, containerName, clientOptions );
    } );

    return containerClient.GetBlobClient( blobName );
}

Under contention GetOrAdd may run the value factory more than once and build a throwaway BlobContainerClient. That is harmless and idempotent, and it can never corrupt the collection.

(Screenshot: attach the web server CPU chart showing the ramp to 100% and the instant drop on recycle.)

Actual Behavior

Under concurrent requests for Azure Blob backed binary files, the shared Dictionary in AzureBlobStorageClient becomes corrupted and threads spin inside Dictionary.FindEntry at 100% CPU.

Observed incident profile:

• Web server CPU ramped from normal to 100% over roughly 2.5 hours following a routine restart, then held at 100%.
• Database CPU and available memory stayed normal throughout. An infinite loop over an in-memory dictionary does no database I/O and allocates no memory.
• Only requests that fetch blob-backed content stalled. They hit the roughly 110 second execution timeout and threw ThreadAbortException (see the stack in the Description). Requests that did not touch blob storage were unaffected.
• CPU returned to normal instantly on application pool recycle and did not recur. The corruption is a probabilistic race against the cold, empty dictionary, so it does not happen on every restart.

Expected Behavior

Concurrent calls to GetBlobClient should safely share the cached BlobContainerClient instances without corrupting shared state or spinning the CPU. A burst of concurrent blob-backed image requests, including right after an application restart when the cache is empty, should never be able to drive the web server to a sustained 100% CPU that requires a recycle to clear.

Steps to Reproduce

• Configure a BinaryFileType to use the Azure Blob Storage provider and store several images with it.
• Restart the application so the AzureBlobStorageClient container cache starts empty.
• Immediately issue many concurrent requests for uncached blob-backed images, so multiple threads enter GetBlobClient and race to populate the dictionary at the same time.
• Intermittently the shared dictionary corrupts. Web server CPU jumps to 100%, threads are stuck in Dictionary.FindEntry, and requests for blob-backed content hang until the execution timeout.
• Only an application pool recycle restores normal CPU.

Note: this is a race condition, so reproduction is probabilistic. Higher concurrency and a cold cache increase the likelihood. The incident here was diagnosed from production thread stacks (see Description) rather than a scripted repro.

Issue Confirmation

• Perform a search on the Github Issues to see if your bug is already reported.
• Reproduced the problem on a fresh install or on the demo site.

Rock Version

19.2.0

Client Culture Setting

en-US

[nairdo]

Thanks for the detailed writeup, that made it easier to track down.

We wrote a test that reproduced the hang first, confirmed it hit the race in GetBlobClient, then applied a fix and confirmed the test passes on the patched code.

[karenn-ccbc]

We just had this happen to us today. Thanks for reporting David.