Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cosbench 0.4.2.c4 gave data integrity error at the end of the read tests #320

Closed
jianghaitao opened this Issue Jun 15, 2016 · 9 comments

Comments

Projects
None yet
7 participants
@jianghaitao
Copy link

jianghaitao commented Jun 15, 2016

Cosbench v0.4.2.c4 against Ceph (Hammer) / radosgw / HAproxy's HTTP end point. All seems fine except that at the end of the read test, it failed the task and gave following errors

com.amazonaws.AmazonClientException: Unable to verify integrity of data download.  Client calculated content hash didn't match hash calculated by Amazon S3.  The data may be corrupt.
        at com.amazonaws.services.s3.internal.DigestValidationInputStream.validateMD5Digest(DigestValidationInputStream.java:79)
        at com.amazonaws.services.s3.internal.DigestValidationInputStream.read(DigestValidationInputStream.java:61)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
        at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155)
        at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:147)
        at com.intel.cosbench.driver.operator.Reader.copyLarge(Reader.java:120)
        at com.intel.cosbench.driver.operator.Reader.doRead(Reader.java:92)
        at com.intel.cosbench.driver.operator.Reader.operate(Reader.java:69)
        at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)
        at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)
        at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)
        at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
@ywang19

This comment has been minimized.

Copy link
Contributor

ywang19 commented Jun 16, 2016

could you paste the workload config file?

@ywang19

This comment has been minimized.

Copy link
Contributor

ywang19 commented Jun 16, 2016

here is a thread about the exception from AWS:
https://github.com/awslabs/aws-sdk-android-samples/issues/62

@ywang19

This comment has been minimized.

Copy link
Contributor

ywang19 commented Jun 16, 2016

about setting the property:

  1. stop all cosbench processes (controller, drivers)
  2. edit cosbench-start.sh, locate the java launching line, and add "-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true"
  3. restart cosbench processes.
@jianghaitao

This comment has been minimized.

Copy link
Author

jianghaitao commented Jun 16, 2016

@ywang19 Thank you very much! -D option helped. Tests completed successfully.

@ywang19 ywang19 closed this Jun 17, 2016

@srujun

This comment has been minimized.

Copy link

srujun commented Jul 12, 2016

I was able to reproduce this issue using this conf file using version 0.4.2.c4: ceph-s3-config.txt

If I set it to just one worker running the read operation, it succeeds with 100% success rate. But any additional workers results in a less than 5% success rate and the eventual failure as mentioned above. However, upon setting hashCheck=true in the read operation, the workload does finish successfully. And all of this is without setting the disableGetObjectMD5Validation=true property.

My question is, if I set the javac property and also keep the hashCheck=true setting for the any operations, will it still check for data integrity or not?

@ericychoi

This comment has been minimized.

Copy link

ericychoi commented Sep 13, 2016

Just wanted to chime in that I am seeing the exact same issue; @srujun any fix you found for this? I was wondering if this is an issue in RadosGW / Ceph or cosbench..

@jamie-liu

This comment has been minimized.

Copy link

jamie-liu commented Feb 17, 2017

@ericychoi I think it's an issue in aws-java-sdk and it has been fixed in 1.5.0 release.
https://forums.aws.amazon.com/thread.jspa?threadID=126007

@ywang19 What's the aws-java-sdk version used in Cosbench 0.4.2.c4?

@hrchu

This comment has been minimized.

Copy link
Contributor

hrchu commented Feb 17, 2017

1.10.76 in master branch, but seems that it is not yet released.

d7a4d7c

@CMADHANKUMAR

This comment has been minimized.

Copy link

CMADHANKUMAR commented Oct 9, 2018

about setting the property:

  1. stop all cosbench processes (controller, drivers)
  2. edit cosbench-start.sh, locate the java launching line, and add "-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true"
  3. restart cosbench processes.

Right one
thankyou

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.