error when decompressing big files

[tansy]

I stumbled upon this error trying to use different test files (#9). It turned out, it is caused by a size of input. Mozilla from Silesia corpus is enough to cause this error even though it uses ~133MB of RAM with the file.

$ g++ -o rygrans-exam main.cpp -g -O3 -Wno-literal-suffix -lm -lrt
$ ln -fs mozilla book1
$ gdb --args rygrans-exam mozilla

(gdb) r

rANS encode:
1184805127 clocks, 23.1 clocks/symbol ( 175.4MiB/s)
(...)
rANS: 39840007 bytes
1061714280 clocks, 20.7 clocks/symbol ( 184.1MiB/s)
(...)
ERROR: bad decoder!

interleaved rANS encode:
927790930 clocks, 18.1 clocks/symbol ( 196.3MiB/s)
(...)
interleaved rANS: 39840009 bytes
1018967059 clocks, 19.9 clocks/symbol ( 187.7MB/s)
(...)
ERROR: bad decoder!
*** glibc detected *** ~/ryg_rans/rygrans-exam: free(): invalid pointer: 0xb2b7f008 ***
======= Backtrace: =========
/lib/libc.so.6(+0x78027)[0xb7cef027]
/usr/lib/libstdc++.so.6(_ZdlPv+0x18)[0xb7eb5878]
/usr/lib/libstdc++.so.6(_ZdaPv+0x18)[0xb7eb58d8]
~/ryg_rans/rygrans-exam[0x804960b]
/lib/libc.so.6(__libc_start_main+0xf5)[0xb7c905a5]
~/ryg_rans/rygrans-exam[0x8049805]
======= Memory map: ========
08048000-0804b000 r-xp 00000000 07:4b 63951      ~/ryg_rans/rygrans-exam
0804b000-0804c000 rw-p 00002000 07:4b 63951      ~/ryg_rans/rygrans-exam
0804c000-08071000 rw-p 00000000 00:00 0          [heap]
afaa5000-b7c5d000 rw-p 00000000 00:00 0 
(...)

Program received signal SIGABRT, Aborted.
0xb7ca6af5 in raise () from /lib/libc.so.6
(gdb) bt
#0  0xb7ca6af5 in raise () from /lib/libc.so.6
#1  0xb7ca8453 in abort () from /lib/libc.so.6
#2  0xb7ce74b5 in ?? () from /lib/libc.so.6
#3  0xb7cef027 in ?? () from /lib/libc.so.6
#4  0xb7eb5878 in operator delete(void*) () from /usr/lib/libstdc++.so.6
#5  0xb7eb58d8 in operator delete[](void*) () from /usr/lib/libstdc++.so.6
#6  0x0804960b in main () at main.cpp:293
(gdb) l main.cpp:293
288         if (memcmp(in_bytes, dec_bytes, in_size) == 0)
289             printf("decode ok!\n");
290         else
291             printf("ERROR: bad decoder!\n");
292
293         delete[] out_buf;
294         delete[] dec_bytes;
295         delete[] in_bytes;
296         return 0;
297     }

[rygorous]

Yes. This code does not support files that encode to more than 32MB.

This code is an illustration of the algorithm, not a data compression utility. Think of it as executable pseudocode.