Google Cloud Storage | CRC32c of large files.

[pguedes17]

Hello,

I'm developing a service to upload files between 1GB and 2GB to Google Cloud Storage.

There's a way to calculate the CRC32hash while i'm writing the stream in the WriteChannel ?

The problem is that always i need to calculate the hash of the whole file before create the blobInfo with the calculated hash.

Here is my current code.

Thanks everyone!

[fazunenko]

I think, the way you upload files is the right way.
Because instances of Blob and BlobInfo classes are immutable objects, you cannot modify Crc32 value of an existing object. So, to enable data validation you need to provide CRC32hash before you open a WriteChannel. The hash calculated after WriteChannel is opened can't be used for validation.

To speed up crc32 hash calculating for your local files I would recommend you to increase the buffer size. 1k seems to be very small size. On my local machine resizing from 1k to 128k reduced hash calculation time from 50 seconds to 4. With the buffer of 1M size calculation took 3 seconds. Experimenting with the size will allow you to figure out the most optimal value.

I hope my answer will help.
Thanks

[fazunenko]

I also noticed a bug in your code of CRC calculation:

Instead of:
hasher = hasher.putBytes(ByteBuffer.wrap(buffer, 0, limit-1));

should be:
hasher = hasher.putBytes(ByteBuffer.wrap(buffer, 0, limit));

Otherwise, the last byte in each chunk will be ignored.

[dmitry-fa]

@Tiozi I hope the question is answered, if you need more details please provide more info.