Multiple open read requests in Azure

[RachBreeze]

Hello

We are seeing multiple read requests when accessing blob storage in Azure a sample trace for a PDF is shown below

Azure does throttle blob storage requests, if the number of requests can be reduced this would improve performance and also help reduce the speed at which throttling occurs.

Thanks in advance

Rachel

Versions: Umbraco 13.6.0
Umbraco.StorageProviders.AzureBlob 13.1.0
Umbraco.StorageProviders.AzureBlob.ImageSharp 13.1.0

[RachBreeze]

I have run a load test on a different site and am seeing this kind of latency. It would be ideal if blob storage would cache the requests OOB

[glombek]

Hi @RachBreeze I've been experiencing similar issues with a site. However, the number of requests being made is not enough to hit any of Blob Storage's thresholds for throttling.

In your blob storage metrics, is the Success Server Latency statistic following the same pattern as Success E2E Latency? If not, the Microsoft troubleshooting guide suggests the problem might be the Umbraco server acknowledging the response from Blob Storage.

This could mean its an issue with thread starvation. Which making multiple requests to blob storage will not be helping! But these functions not being async will also be having an impact.

[RachBreeze]

Hi @glombek yes the exact same pattern.

The code is something like <img src="/media/xxx/nn.jpg" alt="alt text" />

The code for the latency sample uses:

• Umbraco.CMS. 13.7.2
• Umbraco.StorageProviders. AzureBlob 13.1.0
• Umbraco.StorageProviders.AzureBlob.ImageSharp 13.1.0

And thank you :-)

[glombek]

Sorry, I made a mistake in my previous message. I've corrected it from "if so" to "if not".

[JasonElkin]

Notice the two 206 response codes is because Blob Storage is chunking the response - probably because of the size of the image.

I'd be interested to know if we can lower the number of preliminary requests before download though.

[JasonElkin]

@RachBreeze are you seeing this for ImageSharp requests (i.e. /media/abcdefgh/image.jpg?width=200) or requests to media directly, or both?

[ronaldbarendse]

Hi @RachBreeze and @JasonElkin, thanks for the detailed report and investigations. The multiple blob requests are all expected:

• The first HEAD request is done in the Umbraco file system implementation to check whether the file exists and get details like last modified date and length:

  Umbraco.StorageProviders/src/Umbraco.StorageProviders.AzureBlob/AzureBlobFileProvider.cs

  Lines 61 to 77 in 291eee7

  	public IFileInfo GetFileInfo(string subpath)
  	{
  	var path = GetFullPath(subpath);
  	BlobClient blobClient = _containerClient.GetBlobClient(path);
  	BlobProperties properties;
  	try
  	{
  	properties = blobClient.GetProperties().Value;
  	}
  	catch (RequestFailedException ex) when (ex.Status == (int)HttpStatusCode.NotFound)
  	{
  	return new NotFoundFileInfo(AzureBlobItemInfo.ParseName(path));
  	}
  	return new AzureBlobItemInfo(blobClient, properties);
  	}

• The second HEAD request is done by the Azure.Storage.Blobs SDK when calling OpenRead(), followed by the GET request when reading the stream:

  • Umbraco.StorageProviders/src/Umbraco.StorageProviders.AzureBlob/AzureBlobItemInfo.cs

    Line 78 in 291eee7

    public Stream CreateReadStream() => _blobClient.OpenRead();

  • https://github.com/Azure/azure-sdk-for-net/blob/c3fb5ffbd8370c1ece98ee2fe1351348d53fe0aa/sdk/storage/Azure.Storage.Blobs/src/BlobBaseClient.cs#L3278-L3282
• As Jason mentioned, the subsequent GET request is due to the Azure.Storage.Blobs SDK 'chunking' larger files into multiple requests (which seems to default to 4MB).

For caching the requests to the media files, I'd recommend configuring a CDN/proxy using AddCdnMediaUrlProvider() provided by the Umbraco.StorageProviders package.

Lets continue the discussion about the latency/slowdown of requests in #78.

[umbracocommunity]

This issue has been mentioned on Umbraco community forum. There might be relevant details there:

https://forum.umbraco.com/t/usage-of-cloud-storage-in-umbraco-v16/6675/1