Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggested Optimization #190

Closed
seanbrakefield opened this issue Jun 2, 2021 · 2 comments
Closed

Suggested Optimization #190

seanbrakefield opened this issue Jun 2, 2021 · 2 comments

Comments

@seanbrakefield
Copy link

I've been reviewing several potential low-level zip libraries and found this compact and very readable for my purposes. Good work! I'm testing this on Android devices, and I found the performance was ~3x slower than Java's ZipInputStream class. This was surprising so I benchmarked the current zip_extract method and found that the bulk of the delay was with the current mz_crc32 method. The file I am testing on has ~600 files in a 350mb uncompressed zip file. The current library was taking around 4.5 seconds to unzip an uncompressed zip file. Java was taking 1.8 seconds to do the same task. To fix this, I tested replacing the current mz_crc32 method with a method from FastCRC32 by Stephen Brumme https://github.com/stbrumme/crc32 That library has several extreme optimizations for calculating a crc32, and the sweet spot from my testing was using the crc32_16bytes_prefetch method with the 256 look ahead as noted in the comments. This reduced the 4.5 second loading to 1.4 seconds. Hope this feedback helps! :)

@kuba--
Copy link
Owner

kuba-- commented Jun 5, 2021

Good stuff, Thanks!
I'm not sure if we can use directly https://github.com/stbrumme/crc32 library here (because of licensing), but definitively we can try extend API, so everyone can pass own implementation of CRC32.

@kuba--
Copy link
Owner

kuba-- commented Jun 14, 2021

@seanbrakefield PTAL: https://github.com/kuba--/zip/pull/191

you can set your own CRC32 function:

unsigned long my_crc32(unsigned long crc, const void *buf, size_t bufsize) {	
  uint32_t c = crc32_16bytes_prefetch(buf, bufsize, (uint32_t)crc);	return (unsigned long)c;
}

//...
zip_crc32_func(my_crc32);
zip_extract("foo.zip", "/tmp", NULL, NULL);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants