Failed to malloc ckbuf in hash_search2

[phiresky]

This was fixed according to the changelog, but I'm getting it in version 0.621 with a 12 GB directory on Linux 4.2.1-1-ARCH x86_64

[phiresky]

As far as I can tell I get this with all files larger than ~4 GB

[ckolivas]

Is this with lrztar or lrzip directly? If it's with lrztar can you try with lrzip directly on one of the files and see if it only happens with lrztar?

[phiresky]

Both. The original was with lrztar, but I just had this problem with lrzip 2015-09-24-raspbian-jessie.img which is 4.1G

Do you need any other info?

[phiresky]

Wait, it just tried it again with the same file and it worked. So this does not happen always for me.

[ckolivas]

Is it possible you have another application using a lot of ram at the time this happens?

[phiresky]

Yes. I'm trying to reproduce it right now.

[phiresky]

I just tested this for the same 4125 MB file above, filling memory by creating files in /tmp. When free -m reported 5695 MB available before starting lrzip -l file it did not work. With 5914 MB it worked fine.

The error always occurs at 99%. For this file, the failed output is 1.3G (correct output is 1.5G)

Edit: I tried it 3 more times for both 5600MB and 5900M free, the results are consistent.

[ckolivas]

Well then the problem is one of there not being enough free ram to run that level of compression. It's tricky to know how to handle this but I guess seeing how much it can successfully allocate at the beginning is a better way of tackling this than failing during the compression. I will eventually get around to addressing this, thanks.

This is quite different to the original bug report which failed inappropriately, although at the same location.

[Piqlet]

Report : 32 bit, Q9650, 8 GB RAM, lrzip 0.621

$ lrzip -L1 win10.iso
Output filename is: win10.iso.lrz
Failed to malloc ckbuf in hash_search2
Fatal error - exiting

$ lrzip -L1 -p3 win10.iso
Output filename is: win10.iso.lrz
Unable to malloc buffer of size 257196032 in flush_buffer
Fatal error - exiting

$ lrzip -L1 -p6 win10.iso
Output filename is: win10.iso.lrz
win10.iso - Compression Ratio: 1.271. Average Compression Speed: 17.874MB/s.
Total time: 00:04:29.03

$ lrzip -t win10.iso.lrz
Decompressing...
100% 4808.12 / 4808.12 MB
Average DeCompression Speed: 43.315MB/s
[OK] - 5041676288 bytes
Total time: 00:01:51.31

64 bit : OK

[ostrokach]

I am getting the same error compressing a 1.3 GB text file. Using -L1 -p6 did not solve the problem in my case.

Linux Mint 64 bit, 16 GB RAM, lrzip 0.621

[kcavagnolo]

Since this looks like a very specific case you may tackle at some point, I generated this same error while running a memory and cpu intensive process by calling lrztar -z file.h5 on a 756M file. Memory state at the time: 0.92G free of 11.8G

Linux ubuntu 4.2.0-23-generic #28-Ubuntu SMP x86_64 GNU/Linux

[ctirad]

Hello. I face the "Failed to malloc ckbuf in hash_search2" problem as well. I have a custom backup system that i wrote for our office. For a historical reasons it is still a 32bit setup that runs as a live system from initramfs and thus it has no swap. Recently I switched to lrzip for its exellent compression ratio and speed, but it segfaults quite often. I tried the last couple of released versions including the last one and also tried to compile it with different versions of GCC and against older and newer glibc, but the results was allways the same.

So I tried to replace 32bit kernel with 64bit (both standard X86/X86_64 PC) and after that I have the Failed to malloc ckbuf in hash_search2 error instead of segfaults. The rootfs is still exactly the same 32bit. Currently I use -m10 as a workardoud and had no problems so far.

I noticed you have statical builds of lrzip here on page, but the last 32bit version is 0.600. Could you please to make the 32bit build of the latest version, ideally with some debug enabled, for me? I want ot be 100% sure the problem is not in my build envirnoment. Thanks.

[ckolivas]

Fixed.

[awnz]

What version was this fixed in? I'm getting it while trying to compress a 73GB .vdi (VBox image file) on a 2x-quad core Xeon server with 24GB RAM and doing almost no other work. (using -U option) on v0.616 which is the default that comes in the EPEL repo for EL6.

Trying without -U now, will edit this comment with the result.

[ckolivas]

0.616 is ancient and it was fixed long after that as you see on Jun 9 2016 and version 0.616 was released in 2013. I suggest your distribution's repository is considered ancient by any standards.

[awnz]

Ok thanks. I'm a little confused as this bug is dated after that too and above it's reported in v0.621 anyway.

I'll see if there's an EPEL maintainer for it.

[ckolivas]

Yes 0.621 predates the fix as well.

[awnz]

Thanks. Now testing with locally built master, will know in a couple of hours. I was also experiencing lockup issues with this server esp when trying to ZPAQ big stuff, will be interesting to see if those go away too.

[awnz]

My crash (actually lockup) problem appears to be local so won't be lrzip. Above malloc problem solved with master build.

Thanks for the help :)