Unable to upload large size files to Azure Page Blob

[athatte]

Hi, I am unable to upload large sized files (> 5 GB) to the Page Blob service. We are using the
put_page_blog_from_path method to do this, and consistently getting the following error:

ERR 104: Connection reset by peer

On one occasion, the connection was reset 68 times.

I want to know if there's anything we're doing incorrectly or is it an API issue. The same file got uploaded using the Windows Azure Powershell tool.

[huguesv]

Are you using the latest from the master branch? If not, you should try that. If yes, how many connections are using? Do you get more/less/same number of errors as you lower the number of connections?

[athatte]

Hi Hugues, I tried the upload with the latest master branch. The image of size 2 GB got uploaded in ~20 minutes with 5 connections.
The connection reset error we faced was with the older version of the SDK.
Let me run multiple uploads to validate the new branch, and will get back to you.

Thanks

[vmksuw]

Hi , I also tried to upload with the latest master branch. 5 GB got uploaded in ~35 minutes with 8 connections.
We faced issues of connection reset with old sdk.

Thanks

[huguesv]

So it sounds like it's working for both of you, but that's kind of slow. Is it faster with powershell or azcopy? Of course I don't know how fast of an internet connection you have between your machine and your storage account...

Last week, when I tried from a Linux Azure VM to a storage account in the same data center (which should be super fast), I got around 60-70MB/sec upload, 50MB/sec download.

The latest in master branch has retries, so it's possible you are still getting connection reset, but since it retries, it ends up succeeding. If that's the case, the errors will slow down your upload (it sleeps 1 sec between retries, you can control that via parameter).

You could modify the code to make it print when there are retries, so you'll see if that's the case.
https://github.com/Azure/azure-sdk-for-python/blob/master/azure/storage/__init__.py#L1035