The request body is too large and exceeds the maximum permissible limit

[jmlero]

Using the python blobxfer, many times and with many large files (but always smaller than 190G), I receive the error "The request body is too large and exceeds the maximum permissible limit"
I can upload some large files, but is not possible for others files, doesn't matter how many retries or the number of workers.

Also, I have this problem when the local resource to upload is a folder (a folder containing one single file), but if I try to upload the same file specifying the file, the error doesn't appear.

I am using the following python packages:

pip freeze
azure==1.0.2
azure-common==1.0.0
azure-mgmt==0.20.1
azure-mgmt-common==0.20.0
azure-mgmt-compute==0.20.0
azure-mgmt-network==0.20.1
azure-mgmt-nspkg==1.0.0
azure-mgmt-resource==0.20.1
azure-mgmt-storage==0.20.0
azure-nspkg==1.0.0
azure-servicebus==0.20.1
azure-servicemanagement-legacy==0.20.1
azure-storage==0.20.2
elasticsearch==2.2.0
futures==3.0.3
python-dateutil==2.4.2
requests==2.9.1
six==1.10.0
urllib3==1.14
wheel==0.26.0

And blobxfer.py v0.9.9.5

As an example:

azure blobxfer parameters [v0.9.9.5]
subscription id: None
management cert: None
transfer direction: local->Azure
local resource: archive/
remote resource: None
max num of workers: 4
timeout: None
storage account: ---
use SAS: False
upload as page blob: False
auto vhd->page blob: False
container: ---
blob container URI: https://---.blob.core.windows.net/---
compute file MD5: True
skip on MD5 match: True
chunk size (bytes): 4194304
create container: True
keep mismatched MD5: False
recursive if dir: True
keep root dir on up: False
collate to: disabled

script start time: 2016-01-29 10:00:09

g--.tar.gz md5: lhD55kDeLW9uh4PXtJ7LhQ==
detected 0 empty files to upload
performing 25600 put blocks/blobs and 1 put block lists
xfer progress: [ ] 0.00% 0.00 blocks/min The request body is too large and exceeds the maximum permissible limit.

RequestBodyTooLargeThe request body is too large and exceeds the maximum permissible limit.

RequestId:a6af7e74-0001-00f6-0474-5ae0d5000000
Time:2016-01-29T09:06:25.4964043Z100000

ls -lah
-rw-r--r-- 1 root root 100G Jan 26 15:46 g--.tar.gz

[alfpark]

Can you upgrade to 0.9.9.10 and reproduce the issue?

[jmlero]

I tried with the version 0.9.9.9 (the one available at this moment).

The structure of the folder I would like to upload is the following:

/data/groups/folder/archive/file1
/data/groups/folder/archive/file2
...
/data/groups/folder/archive/fileN

100 GB each one.

And the layout I would like on the storage is the following in this case:
file1
file2
...
fileN

Using --strip-components=3

azure blobxfer parameters [v0.9.9.9]

platform: Linux-2.6.32-431.20.3.el6.x86_64-x86_64-with-redhat-6.5-Carbon
python interpreter: CPython 2.7.11
package versions: az.common=1.0.0 az.sml=0.20.1 az.stor=0.20.2 req=2.9.1
subscription id: None
management cert: None
transfer direction: local->Azure
local resource: /data/groups/folder/archive/
include pattern: None
remote resource: None
max num of workers: 48
timeout: None
storage account: XXX
use SAS: False
upload as page blob: False
auto vhd->page blob: False
container: folder
blob container URI: https://XXX.blob.core.windows.net/folder
compute file MD5: True
skip on MD5 match: True
chunk size (bytes): 4194304
create container: True
keep mismatched MD5: False
recursive if dir: True
component strip on up: 3
remote delete: False
collate to: disabled
local overwrite: True
encryption mode: disabled
RSA key file: disabled
RSA key type: disabled

script start time: 2016-02-01 10:03:40
computing file md5 on: /data/groups/folder/archive/file.tar.gz.split

md5: lhD55kDeLW9uh4PQ==
detected 0 empty files to upload
performing 25600 put blocks/blobs and 1 put block lists
spawning 48 worker threads

xfer progress: [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 100.00% 793.60 blocks/min

102400.0 MiB transfered, elapsed 1935.4741931 sec. Throughput = 423.255449709 Mbit/sec

Is working fine at least for 1 file, but the structure on the storage is:
archive/file1

Using --strip-components=4 I obtain the following:

azure blobxfer parameters [v0.9.9.9]

platform: Linux-2.6.32-431.20.3.el6.x86_64-x86_64-with-redhat-6.5-Carbon
python interpreter: CPython 2.7.11
package versions: az.common=1.0.0 az.sml=0.20.1 az.stor=0.20.2 req=2.9.1
subscription id: None
management cert: None
transfer direction: local->Azure
local resource: /data/groups/folder/archive/
include pattern: None
remote resource: None
max num of workers: 48
timeout: None
storage account: XXX
use SAS: False
upload as page blob: False
auto vhd->page blob: False
container: folder
blob container URI: https://XXX.blob.core.windows.net/folder
compute file MD5: True
skip on MD5 match: True
chunk size (bytes): 4194304
create container: True
keep mismatched MD5: False
recursive if dir: True
component strip on up: 4
remote delete: False
collate to: disabled
local overwrite: True
encryption mode: disabled
RSA key file: disabled
RSA key type: disabled

script start time: 2016-02-01 09:58:24
computing file md5 on: /data/groups/folder/archive/file.tar.gz.split
md5: lhD55kDeLW9uh4PXtJ7LhQ==
detected 0 empty files to upload
performing 25600 put blocks/blobs and 1 put block lists
spawning 48 worker threads
xfer progress: [ ] 0.00% 0.00 blocks/min Traceback (most recent call last):
File "blobxfer_0.9.9.9.py", line 880, in run
offset, bytestoxfer, encparam, flock, filedesc)
File "blobxfer_0.9.9.9.py", line 1002, in putblobdata
content_md5=contentmd5, timeout=self.timeout)
File "blobxfer_0.9.9.9.py", line 1309, in azure_request
return req(_args, *_kwargs)
File "/cm/local/apps/python/2.7.11_azure/lib/python2.7/site-packages/azure/storage/blob/blobservice.py", line 2366, in put_block
self._perform_request(request)
File "/cm/local/apps/python/2.7.11_azure/lib/python2.7/site-packages/azure/storage/storageclient.py", line 178, in _perform_request
_storage_error_handler(ex)
File "/cm/local/apps/python/2.7.11_azure/lib/python2.7/site-packages/azure/storage/_serialization.py", line 25, in _storage_error_handler
return _general_error_handler(http_error)
File "/cm/local/apps/python/2.7.11_azure/lib/python2.7/site-packages/azure/storage/_common_error.py", line 82, in _general_error_handler
raise AzureHttpError(message, http_error.status)
AzureHttpError: The request body is too large and exceeds the maximum permissible limit.

RequestBodyTooLargeThe request body is too large and exceeds the maximum permissible limit.

RequestId:c7662108-0001-0111-2ccf-5cb68d000000
Time:2016-02-01T09:02:45.8204723Z100000

Thanks and regards

[alfpark]

Unfortunately, I cannot reproduce this error.

Can you place this statement: print(offset, bytestoxfer, len(data)) prior to the azure_request function call on line 999? Please re-run your scenario and paste your output.

[jmlero]

The output:

detected 0 empty files to upload
performing 25600 put blocks/blobs and 1 put block lists
spawning 48 worker threads
xfer progress: [ ] 0.00% 0.00 blocks/min 0 4194304 4194304
4194304 4194304 4194304
8388608 4194304 4194304
12582912 4194304 4194304
16777216 4194304 4194304
20971520 4194304 4194304
25165824 4194304 4194304
29360128 4194304 4194304
33554432 4194304 4194304
37748736 4194304 4194304
41943040 4194304 4194304
46137344 4194304 4194304
50331648 4194304 4194304
54525952 4194304 4194304
58720256 4194304 4194304
62914560 4194304 4194304
67108864 4194304 4194304
71303168 4194304 4194304
75497472 4194304 4194304
79691776 4194304 4194304
83886080 4194304 4194304
88080384 4194304 4194304
92274688 4194304 4194304
96468992 4194304 4194304
100663296 4194304 4194304
104857600 4194304 4194304
109051904 4194304 4194304
113246208 4194304 4194304
117440512 4194304 4194304
121634816 4194304 4194304
125829120 4194304 4194304
130023424 4194304 4194304
134217728 4194304 4194304
138412032 4194304 4194304
142606336 4194304 4194304
146800640 4194304 4194304
150994944 4194304 4194304
155189248 4194304 4194304
159383552 4194304 4194304
163577856 4194304 4194304
167772160 4194304 4194304
171966464 4194304 4194304
176160768 4194304 4194304
180355072 4194304 4194304
184549376 4194304 4194304
188743680 4194304 4194304
192937984 4194304 4194304
197132288 4194304 4194304
Traceback (most recent call last):
File "/data/groups/adm_informatics/prod/archive2azure/blobxfer_0.9.9.9.py", line 880, in run
offset, bytestoxfer, encparam, flock, filedesc)
File "/data/groups/adm_informatics/prod/archive2azure/blobxfer_0.9.9.9.py", line 1003, in putblobdata
content_md5=contentmd5, timeout=self.timeout)
File "/data/groups/adm_informatics/prod/archive2azure/blobxfer_0.9.9.9.py", line 1310, in azure_request
return req(_args, *_kwargs)
File "/cm/local/apps/python/2.7.11_azure/lib/python2.7/site-packages/azure/storage/blob/blobservice.py", line 2366, in put_block
self._perform_request(request)
File "/cm/local/apps/python/2.7.11_azure/lib/python2.7/site-packages/azure/storage/storageclient.py", line 178, in _perform_request
_storage_error_handler(ex)
File "/cm/local/apps/python/2.7.11_azure/lib/python2.7/site-packages/azure/storage/_serialization.py", line 25, in _storage_error_handler
return _general_error_handler(http_error)
File "/cm/local/apps/python/2.7.11_azure/lib/python2.7/site-packages/azure/storage/_common_error.py", line 82, in _general_error_handler
raise AzureHttpError(message, http_error.status)
AzureHttpError: The request body is too large and exceeds the maximum permissible limit.

RequestBodyTooLargeThe request body is too large and exceeds the maximum permissible limit.

RequestId:3309da30-0001-0000-78be-5dc7c3000000
Time:2016-02-02T13:33:05.8171503Z100000

[alfpark]

Thanks for running the script with the modification. According to the new debug lines, the data being sent to the Azure Python Storage SDK is consistent with the maximum allowable block size of 4MB.

I have two suggestions:

1. Retry the upload using a SAS key so the request is a direct REST call via requests
2. Pass --chunksizebytes 4194296 as a parameter

[jmlero]

Using the second option, pass --chunksizebytes 4194296 as a parameter works fine. I will try also using a SAS key, but I think that for me the best solution is change the value _MAX_BLOB_CHUNK_SIZE_BYTES to 4194296 as you suggested

If this error is related with the SDK, also I will try the next release of the SDK as soon as it will be available.

Thanks and regards

[alfpark]

Thanks for working through the issue. If you want, you can directly raise this issue to the Azure Python Storage SDK github repo. I will close this issue.