Performance Measurements with Google CS

Some performance results

Tested from Germany on March 30th 2017 from a Lenovo X240 Thinkpad. Internet connection was ADSL with 16 MBps downstream and 1 MBps upstream. The services (buckets) tested were located in the US.

Google Cloud Storage service

These options need to be modified inside the file ~/.s3cfg in order to interact with Google Cloud Storage.

access_key = insert here your access key
secret_key = insert here your secret access key
host_base = storage.googleapis.com
host_bucket = %(bucket).storage.googleapis.com

5 Files, 1 Byte (2^0 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 1 -p
...
Required time to create the bucket:                 1.291s
Required time to upload the files:                  2.096s
Required time to download the files:                2.245s
Required time to erase the objects:                 3.931s
Required time to erase the bucket:                  1.092s
Required time to perform all S3-related operations: 10.655s

5 Files, 2 Byte (2^1 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 2 -p
...
Required time to create the bucket:                 1.702s
Required time to upload the files:                  2.037s
Required time to download the files:                2.198s
Required time to erase the objects:                 3.670s
Required time to erase the bucket:                  0.966s
Required time to perform all S3-related operations: 10.573s

5 Files, 4 Byte (2^2 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 4 -p
...
Required time to create the bucket:                 1.479s
Required time to upload the files:                  1.897s
Required time to download the files:                1.901s
Required time to erase the objects:                 3.766s
Required time to erase the bucket:                  1.033s
Required time to perform all S3-related operations: 10.076s

5 Files, 8 Byte (2^3 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 8 -p
...
Required time to create the bucket:                 1.623s
Required time to upload the files:                  2.295s
Required time to download the files:                1.854s
Required time to erase the objects:                 3.751s
Required time to erase the bucket:                  0.980s
Required time to perform all S3-related operations: 10.503s

5 Files, 16 Byte (2^4 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 16 -p
...
Required time to create the bucket:                 1.581s
Required time to upload the files:                  2.240s
Required time to download the files:                1.724s
Required time to erase the objects:                 3.908s
Required time to erase the bucket:                  0.924s
Required time to perform all S3-related operations: 10.377s

5 Files, 32 Byte (2^5 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 32 -p
...
Required time to create the bucket:                 1.399s
Required time to upload the files:                  2.283s
Required time to download the files:                1.945s
Required time to erase the objects:                 3.966s
Required time to erase the bucket:                  1.139s
Required time to perform all S3-related operations: 10.732s

5 Files, 64 Byte (2^6 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 64 -p
...
Required time to create the bucket:                 1.589s
Required time to upload the files:                  2.354s
Required time to download the files:                1.839s
Required time to erase the objects:                 3.973s
Required time to erase the bucket:                  2.351s
Required time to perform all S3-related operations: 12.106s

5 Files, 128 Byte (2^7 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 128 -p
...
Required time to create the bucket:                 1.372s
Required time to upload the files:                  2.396s
Required time to download the files:                1.873s
Required time to erase the objects:                 5.442s
Required time to erase the bucket:                  0.948s
Required time to perform all S3-related operations: 12.031s

5 Files, 256 Byte (2^8 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 256 -p
...
Required time to create the bucket:                 1.789s
Required time to upload the files:                  2.221s
Required time to download the files:                1.693s
Required time to erase the objects:                 4.064s
Required time to erase the bucket:                  0.914s
Required time to perform all S3-related operations: 10.681s

5 Files, 512 Byte (2^9 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 512 -p
...
Required time to create the bucket:                 1.507s
Required time to upload the files:                  2.236s
Required time to download the files:                2.100s
Required time to erase the objects:                 4.487s
Required time to erase the bucket:                  1.226s
Required time to perform all S3-related operations: 11.556s

5 Files, 1 kb (2^10 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 1024 -p
...
Required time to create the bucket:                 1.415s
Required time to upload the files:                  2.096s
Required time to download the files:                2.088s
Required time to erase the objects:                 3.893s
Required time to erase the bucket:                  0.920s
Required time to perform all S3-related operations: 10.412s

5 Files, 2 kb (2^11 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 2048 -p
...
Required time to create the bucket:                 1.575s
Required time to upload the files:                  2.440s
Required time to download the files:                2.076s
Required time to erase the objects:                 4.069s
Required time to erase the bucket:                  1.142s
Required time to perform all S3-related operations: 11.302s

5 Files, 4 kb (2^12 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 4096 -p
...
Required time to create the bucket:                 1.497s
Required time to upload the files:                  2.072s
Required time to download the files:                1.739s
Required time to erase the objects:                 4.112s
Required time to erase the bucket:                  1.275s
Required time to perform all S3-related operations: 10.695s

5 Files, 8 kb (2^13 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 8192 -p
...
Required time to create the bucket:                 1.593s
Required time to upload the files:                  2.269s
Required time to download the files:                2.047s
Required time to erase the objects:                 4.205s
Required time to erase the bucket:                  1.171s
Required time to perform all S3-related operations: 11.285s

5 Files, 16 kb (2^14 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 16384 -p
...
Required time to create the bucket:                 1.558s
Required time to upload the files:                  2.514s
Required time to download the files:                2.042s
Required time to erase the objects:                 4.017s
Required time to erase the bucket:                  1.094s
Required time to perform all S3-related operations: 11.225s

5 Files, 32 kb (2^15 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 32768 -p
...
Required time to create the bucket:                 1.540s
Required time to upload the files:                  2.492s
Required time to download the files:                2.287s
Required time to erase the objects:                 4.212s
Required time to erase the bucket:                  1.137s
Required time to perform all S3-related operations: 11.668s

5 Files, 64 kb (2^16 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 65536 -p
...
Required time to create the bucket:                 1.669s
Required time to upload the files:                  2.843s
Required time to download the files:                3.323s
Required time to erase the objects:                 4.483s
Required time to erase the bucket:                  1.121s
Required time to perform all S3-related operations: 13.439s

5 Files, 128 kb (2^17 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 131072 -p
...
Required time to create the bucket:                 1.525s
Required time to upload the files:                  3.701s
Required time to download the files:                2.373s
Required time to erase the objects:                 4.153s
Required time to erase the bucket:                  1.034s
Required time to perform all S3-related operations: 12.786s

5 Files, 256 kb (2^18 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 262144 -p 
...
Required time to create the bucket:                 1.484s
Required time to upload the files:                  4.552s
Required time to download the files:                2.303s
Required time to erase the objects:                 4.189s
Required time to erase the bucket:                  1.226s
Required time to perform all S3-related operations: 13.754s

5 Files, 512 kb (2^19 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 524288 -p
...
Required time to create the bucket:                 1.605s
Required time to upload the files:                  6.153s
Required time to download the files:                2.696s
Required time to erase the objects:                 4.128s
Required time to erase the bucket:                  1.130s
Required time to perform all S3-related operations: 15.712s   

5 Files, 1 MB (2^20 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 1048576 -p  
...
Required time to create the bucket:                 1.592s
Required time to upload the files:                  13.479s
Required time to download the files:                3.278s
Required time to erase the objects:                 3.950s
Required time to erase the bucket:                  1.119s
Required time to perform all S3-related operations: 23.418s

5 Files, 2 MB (2^21 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 2097152 -p  
...
Required time to create the bucket:                 1.592s
Required time to upload the files:                  13.479s
Required time to download the files:                3.278s
Required time to erase the objects:                 3.950s
Required time to erase the bucket:                  1.119s
Required time to perform all S3-related operations: 23.418s

5 Files, 4 MB (2^22 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 4194304 -p  
...
Required time to create the bucket:                 1.445s
Required time to upload the files:                  19.873s
Required time to download the files:                10.074s
Required time to erase the objects:                 4.046s
Required time to erase the bucket:                  0.949s
Required time to perform all S3-related operations: 36.387s

5 Files, 8 MB (2^23 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 8388608 -p  
...
Required time to create the bucket:                 1.469s
Required time to upload the files:                  75.071s
Required time to download the files:                16.321s
Required time to erase the objects:                 3.841s
Required time to erase the bucket:                  1.223s
Required time to perform all S3-related operations: 97.925s

5 Files, 16 MB (2^24 Byte) each, parallel transfer

$ ./s3perf.sh -n 5 -s 16777216 -p  
...
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.