Out of memory when uploading large files exceeding 1GB using --data-binary @

[timyuer]

I did this

When I upload a large file to HDFS, curl reports a memory overflow，This file has a size of 1.1G.

curl -sS -L -w '%{http_code}' -X PUT --data-binary '@/root/hbase.tar.gz' -H 'Content-Type: application/octet-stream' 'http://192.168.8.101:50070/webhdfs/v1/hdp/apps/3.3.2.0-008/hbase/hbase.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'

When I modify the following code

curl/src/tool_paramhlp.c

Line 87 in c7aef0a

#define MAX_FILE2MEMORY (1024*1024*1024) /* big enough ? */

to

#define MAX_FILE2MEMORY (1024*1024*1024*2) /* big enough ? */

Run successfully after recompilation.

I expected the following

I hope to upload files larger than 1GB.

curl/libcurl version

curl-7.79.1，According to my testing, this issue has been present since version 7.73.0

operating system

openEuler-22.03-LTS-SP4

[bagder]

Or you upload it in a streaming fashion instead of loading the entire thing into memory before sending and then you can upload however large file you want:

curl -sS -L -w '%{http_code}' -T '/root/hbase.tar.gz' -H 'Content-Type: application/octet-stream' 'http://192.168.8.101:50070/webhdfs/v1/hdp/apps/3.3.2.0-008/hbase/hbase.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'