Repository navigation
Out of memory when uploading large files exceeding 1GB using --data-binary @ #14521
Description
Activity
Or you upload it in a streaming fashion instead of loading the entire thing into memory before sending and then you can upload however large file you want:
curl -sS -L -w '%{http_code}' -T '/root/hbase.tar.gz' -H 'Content-Type: application/octet-stream' 'http://192.168.8.101:50070/webhdfs/v1/hdp/apps/3.3.2.0-008/hbase/hbase.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'- added a commit that references this issue
on Aug 13, 2024 - added a commit that references this issue
on Aug 14, 2024
I did this
When I upload a large file to HDFS, curl reports a memory overflow，This file has a size of 1.1G.
When I modify the following code
curl/src/tool_paramhlp.c
Line 87 in c7aef0a
to
Run successfully after recompilation.
I expected the following
I hope to upload files larger than 1GB.
curl/libcurl version
curl-7.79.1，According to my testing, this issue has been present since version 7.73.0
operating system
openEuler-22.03-LTS-SP4