perf: Prefer GIL-releasing isal over deflate - #126
Conversation
|
This is petty cool I’ll give this a shot!
…On Sun, Jun 21, 2026 at 6:57 AM Nico Kemnitz ***@***.***> wrote:
@nkemnitz <https://github.com/nkemnitz> requested your review on:
seung-lab/cloud-files#126
<#126> perf: Prefer
GIL-releasing isal over deflate.
—
Reply to this email directly, view it on GitHub
<#126?email_source=notifications&email_token=AATGQSNAMXCUZAZ3DGQQQ4L5A65RZA5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTENZQGA3TKNZUGUYTNJTSMVQXG33OWBZGK5TJMV3V64TFOF2WK43UMVSKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-27007574516>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AATGQSLLOMYAEWGEE5W2GZL5A65RZAVCNFSNUABFKJSXA33TNF2G64TZHMZDMNZRGE4DQNJYHNEXG43VMU5TINZRGAYTOMRRGA22C5QC>
.
You are receiving this because your review was requested.Message ID:
***@***.***>
|
|
I ran my own little benchmark on decompressing a small dataset with cloudfiles. I saw the per hit for isal was 10x deflate, though the total time amounted to only slightly more. Can you share your benchmark? This might be file size dependent. |
|
Oh you know what it probably is? deflate is SIMD and I ran it on segmentation. I suspect you ran it on images. The regularity of segmentation gives a large benefit to SIMD, asm will probably do better on images. |
|
Checked and can replicate the slowdown on arm64. (Not quite as extreme, but at least 2x). This, and I saw that at fastest compression settings, libdeflate beats igzip in both compression speed, decompression speed and compression ratio for our test data. |
2 participants
Single-threaded isal itself already gives a significant speed-boost for compression (3.3x !?) and decompression (1.2x) on my local i7-12700 over
deflate, But more importantly, it releases the GIL, no longer blocks the other worker-threads in multi-threaded Python (e.g. CloudVolume).Wheels are available for all major platforms and CPython versions, but I hid it behind a guard for graceful fallback to
deflate, just in case.