Corrupt upload using AWS Java SDK

[adamlamar]

Describe the bug
Client upload using AWS SDK 1.11.745 fails with data integrity error:

Unable to verify integrity of data upload. Client calculated content hash (contentMD5: DyYDJ6odjkGLkn0zZ7ojjA== in base 64) didn't match hash (etag: 3df389e4041f77e0a9557d486d6e9fcf in hex) calculated by Amazon S3.  You may need to delete the data stored in Amazon S3. (metadata.contentMD5: null, md5DigestStream: com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream@341308f5, bucketName: foo, key: path/to/key.jar)

A partial file is uploaded. Upon download the file is obviously corrupt:

$ unzip -l file.jar
Archive:  file.jar
error: End-of-centdir-64 signature not where expected (prepended bytes?)
  (attempting to process anyway)
warning [file.jar]:  230757 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [file.jar]:  start of central directory not found;
  zipfile corrupt.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)

Example Java SDK usage:

    if (!amazonS3Client.doesObjectExist(s3Bucket, keyName)) {
      try {
        Resource resource = new UrlResource("path/to/resource");
        ObjectMetadata metadata = new ObjectMetadata();
        metadata.setContentLength(resource.contentLength());
        metadata.setContentType(MediaType.APPLICATION_OCTET_STREAM_VALUE);
        PutObjectRequest request = new PutObjectRequest(s3Bucket, keyName, resource.getInputStream(), metadata);
        amazonS3Client.putObject(request);

System Setup

• weed master -volumeSizeLimitMB=1000
• weed volume -dir=/data -mserver=seaweedfs-master:9333 -max=1000 -port=8080
• weed filer -s3 -master=seaweedfs-master:9333 port=8888
• Docker container chrislusf/seaweedfs:2.48
• output of weed version

# weed version
version 30GB 2.48 45a76222 linux amd64

• if using filer, show the content of filer.toml
  None - using defaults from docker image

Expected behavior
The full file should be uploaded.

Additional context
Uploads using the aws CLI command and the AWS golang API do not have the same issue.

[chrislusf]

could you please share a java example project to reproduce this?

[adamlamar]

Sure, here you go.
aws-java-test.zip

Run with:

mvn install && java -jar target/aws-java-test-1.0-SNAPSHOT-jar-with-dependencies.jar

Which shows

uploading to http://localhost:8333
Exception in thread "main" com.amazonaws.SdkClientException: Unable to verify integrity of data upload. Client calculated content hash (contentMD5: gA3l92CXzNJ7dje2FfJdzw== in base 64) didn't match hash (etag: 1a3dda4eb371be6d4f92518b8be2df08 in hex) calculated by Amazon S3.  You may need to delete the data stored in Amazon S3. (metadata.contentMD5: null, md5DigestStream: com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream@5d0a1059, bucketName: foo, key: bar)
	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1834)
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)
	at com.example.Main.main(Main.java:41)

[chrislusf]

Please run this in weed shell and try again:

> s3.configure -access_key=any -secret_key=any -buckets=foo -user=me -actions=Read,Write,List -apply
{
  "identities": [
    {
      "name": "me",
      "credentials": [
        {
          "accessKey": "any",
          "secretKey": "any"
        }
      ],
      "actions": [
        "Read:foo",
        "Write:foo",
        "List:foo"
      ]
    }
  ]
}

[adamlamar]

That worked! Is there a bug that needs to be addressed for the case when authentication is not setup? Or will the AWS java SDK only work when authentication is enabled due to some protocol reason?

[chrislusf]

Thanks! I added some code to fail the signed requests if no authentication is setup.