Large files are truncated when uploaded to s3

[aharonh]

When uploading large files to aws s3, large files - for example an 120Gi file is truncated to 22.3 Gi.

Expected behaviour - upload the files in full.

[aharonh]

to reproduce the issue:

generate large file:

truncate -s 120g /var/lib/cassandra/test.file

run job examples/in-cluster/job.yaml

logs will be as follows:

test-skbn-kqhrt test-skbn 2024/06/11 19:26:44 [1/1] copy: k8s://invu-sre/cassandra-primary-r1-sts-0/cassandra/var/lib/cassandra/test.file -> s3://nuvo-dev-dbbackup/test.file
test-skbn-kqhrt test-skbn 2024/06/11 19:42:05 [1/1] done: k8s://invu-sre/cassandra-primary-r1-sts-0/cassandra/var/lib/cassandra/test.file -> s3://nuvo-dev-dbbackup/test.file

The issue was reproduced. the created file was 22.3 Gi

After adding verbose log

2024/06/12 01:33:51 [1/1] copy: k8s://invu-sre/cassandra-primary-r1-sts-0/cassandra/var/lib/cassandra/test.file -> s3://nuvo-dev-dbbackup/test.file
2024/06/12 01:33:51 Attempt 1 to upload file to s3://nuvo-dev-dbbackup/test.file
2024/06/12 01:33:51 Attempt 1 to download file from invu-sre/cassandra-primary-r1-sts-0/cassandra:/var/lib/cassandra/test.file
2024/06/12 01:40:22 Uploaded file to s3://nuvo-dev-dbbackup/test.file
2024/06/12 01:40:22 Error: MultipartUpload: upload multipart failed
upload id: Y2TKM3Bw2B0dTojHAb2aRzxan580b4_WsQ_IrlJpLbt1iseehxxgnHEMXrGOPwG_et.jOKkEc_nJkQcWDGiYI14eu9d3v2bcPic0OfEVpx547ZU5QiaMrf_m0wjuw3.HVsig5HQTGQoei35ylV7Tj7sTOW4JfMx.CpN_D90_DQM-
caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit

So, at least, need to allow for configuration of maxUploadParts and partSize for s3 uploads.