Blob Storage requests slow down significantly when there are multiple concurrent requests to blob storage. #78
Description
Occasionally, requests to blob storage will take a long time. This happens when the number of requests to blob storage suddenly increases. Blob storage metrics show a small latency increase for the blob storage server but a large spike for E2E - implying the slowdown is due to the Azure Web app.
For us, this seems to only affect a small number of users (we have other caching in place) but the impact for those users is significant.
This is perhaps related to #77 (in terms of thinking about cutting down on the number of requests).
After a bit of investigation at Bump, here's what we think is happening:
Umbraco is not using ImageSharp's built-in file provider:
Instead an IFileProvider is provided
And ImageSharp then uses the static file provider:
One problem with this approach is that the IFileProvider is synchronous and means that image requests are blocking threads when they talk to blob storage. It could be this that's causing performance problems when multiple images are requested simultaneously
We're going to swap to the AzureBlobStorageImageProvider and see if that makes a meaningful difference to performance, though It'll only make a difference to images processed with ImageSharp.
For requests that don't need image processing, serving directly from blob storage might be a helpful workaround too, as it saves Umbraco from wasting resources proxying/streaming the file at all.
We think that it would be better to use something other than IFileProvider generally though. I can see why it's there, it simplifies APIs, but being synchronous it's only going to be performant for local filesystems with fast SSDs.
Obviously, after the first request for an image, using something like Cloudflare or Front Door to cache the request improves performance generally. But out of the box this is approach is still inefficient and causes noticeable performance degradation for users that encounter a cache miss.
There's also a problem for 404s, which are uncacheable. Requests for an image that aren't there will always result in a call to blob storage.
Thinking out loud...
What if there was an IMediaUrlService counterpart to the IDocumentUrlService?
So we had a routing table of media that we know exists. We could eagerly 404 without touching Blob Storage, or disk, or whatever.