"500 Operation could not be completed within the specified time" when downloading huge file

[schindlerd]

I'm trying to download a big exported disk snapshot (1.6 TB) but azcopy fails upon downloading with error 500.

Which version of the AzCopy was used?

10.21.2

Note: The version is visible when running AzCopy without any argument

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

azcopy.exe copy "https://md-gdlkcldgsmbx.z1.blob.storage.azure.net/dk4rgsx2pkrj/abcd?sv=2018-03-28&sr=b&si=xxx&sig=xxx" "D:\download\datadisk.vhd"

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

What problem was encountered?

2023/11/08 13:50:12 PERF: primary performance constraint is Unknown. States: C:  0, R:  0, W:  0, F:  0, H:  0, B:  0, S:  0, P:  0, Q:  0, D:  0, E:  0, T:  0, GRs: 192
2023/11/08 13:50:12 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2023/11/08 13:50:12 INFO: [P#0-T#0] Page blob throughput tuner: Target Mbps 4066
2023/11/08 13:50:12 ==> REQUEST/RESPONSE (Try=1/30.1644245s[SLOW >3s], OpTime=30.2698049s) -- RESPONSE STATUS CODE ERROR
   GET https://md-gdlkcldgsmbx.z1.blob.storage.azure.net/dk4rgsx2pkrj/abcd?comp=pagelist&si=xxx&sig=-REDACTED-&sr=b&sv=2018-03-28
   Accept: application/xml
   User-Agent: AzCopy/10.21.2 azsdk-go-azblob/v1.1.0 (go1.19.12; Windows_NT)
   X-Ms-Client-Request-Id: xxx
   x-ms-version: 
   --------------------------------------------------------------------------------
   RESPONSE Status: 500 Operation could not be completed within the specified time.
   Connection: Keep-Alive
   Content-Length: 246
   Content-Type: application/xml
   Date: Wed, 08 Nov 2023 13:50:12 GMT
   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
   X-Ms-Client-Request-Id: xxx
   X-Ms-Error-Code: OperationTimedOut
   X-Ms-Request-Id: xxx
   X-Ms-Version: 2020-10-02
Response Details: <?xml version="1.0" encoding="utf-8"?> <Error><Code>OperationTimedOut</Code><Message>Operation could not be completed within the specified time. </Message>

How can we reproduce the problem in the simplest way?

Try downloading a huge VHD/Azure VM Disk?

Have you found a mitigation/solution?

no

[schindlerd]

See also: https://feedback.azure.com/d365community/idea/6a479865-d383-ee11-a81c-000d3ae5ae95

Looks like older version 10.18.1 hasn't got the problem.

[pid1]

I can reliably reproduce this. 10.22.1 on macOS (Apple Silicon) gets stuck at Pending with the occasional 500 Operation could not be completed within the specified time.

10.18.1 works as expected.

Interestingly, 10.22.1 works correctly with a ~136GB VHD file download, but not one ~200GB. Is this potentially caused by some pre-download source file size checking that ends up out of bounds?

[gapra-msft]

Hi @schindlerd and @pid1 could you share the full AzCopy log as well as the console output of a recent run? This will help our investigation. Also, please keep the request ID intact in the log, so we can look up service logs if necessary.

[JohnLBevan]

Thank-you for fixing this.
FYI: We were having this issue with Azure Storage Explorer. Including insider builds and running update didn't get us the fixed version of AZCopy (at time of writing).
For anyone else wanting this fix before Azure Storage Explorer is updated, I found I could do this update by:

• Download the latest version of AZCopy as appropriate for your OS
• Ensure Azure Storage Explorer is not running on your machine.
• Navigate to Azure Storage Explorer's main directory, then go to the child path: \resources\app\node_modules\@azure-tools\azcopy-win64\dist\bin (on my machine that translates to C:\Program Files\Microsoft Azure Storage Explorer\resources\app\node_modules\@azure-tools\azcopy-win64\dist\bin).
• Extract azcopy.exe from the download into this directory (filename may vary based on your OS)
• Rename or delete the old azcopy_windows_amd64.exe.old file (again, filename may vary)
• Rename azcopy.exe to azcopy_windows_amd64.exe (amending filenames as appropriate for your OS.
• Rerun Azure Storage Explorer, click Help, About. You should now see the version of azcopy has updated, and you're able to download large files without hitting the timeout issue.