Could not read frame info: ERROR_frameHeader_incomplete

[filimonov]

Sample case:

use Compress::LZ4Frame;
my $sample_data = pack( 'H*', '04224d186040822901000014000100ff0b01080cc120a6f701000000015e7b08a6d5ffffffff000107f6580100ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffc5505858585858130000001f580100ffffffffffffffff0750585858585800000000');
warn length(Compress::LZ4Frame::decompress($sample_data));

Outputs:

Could not read frame info: ERROR_frameHeader_incomplete at test.pl line 4.
Use of uninitialized value in length at test.pl line 4.
0 at test.pl line 4.

At the same time i can decompress the same data with commandline tool alse based on lz4frame lib:

my $sample_data = pack( 'H*', '04224d186040822901000014000100ff0b01080cc120a6f701000000015e7b08a6d5ffffffff000107f6580100ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffc5505858585858130000001f580100ffffffffffffffff0750585858585800000000');
open(my $out, '>:raw', 'sample.md4') or die "Unable to open: $!";
print $out $sample_data;
close($out);

#➜ lz4 -d sample.md4 md4_test.out
#Successfully decoded 67608 bytes   

[Drako]

Thanks for bringing that up.
I'll investigate it later today.

[LittleFox94]

@Drako works locally, did you forget pushing changes? ^^

[Drako]

@LittleFox94 that would not make any sense.
If I forgot to push anything, he still would have the same version you have and would get the same results.

[LittleFox94]

@Drako tsm says that bug is already known. Calling that sample multiple times results in that error on our machines.

use Compress::LZ4Frame;
for(1..10000){
    my $sample_data = pack( 'H*', '04224d186040822901000014000100ff0b01080cc120a6f701000000015e7b08a6d5ffffffff000107f6580100ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffc5505858585858130000001f580100ffffffffffffffff0750585858585800000000');
    my $x =  length(Compress::LZ4Frame::decompress($sample_data));
}

[Drako]

@LittleFox94 nice to learn that it was known already, lol
anyways, unless you guys fixed it by then I will have a look at it later.

[Drako]

also does the error appear on the second iteration or sometime later?
and did you check if it happened at a consistent point or does it happen at different times?

[pivo42]

Hi,

the is problem is only recently know and I should have reported it earlier. My bad, sorry.
But the I think "our" problem is a different one. In our case the header was broken.
And after thousand of iterations looks_like_lz4frame was returning true for a broken header.
After we found out that a broken header was the problem, we just fix the header and forgot to post the problem on github. Out of sight, out of mind.

BTT...
This code:

use Compress::LZ4Frame;
for(1..10000){
    my $sample_data = pack( 'H*', '04224d186040822901000014000100ff0b01080cc120a6f701000000015e7b08a6d5ffffffff000107f6580100ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffc5505858585858130000001f580100ffffffffffffffff0750585858585800000000');
    my $x =  length(Compress::LZ4Frame::decompress($sample_data));
}

, just ran into an out of memory on a very small machine.
Also sorry, for this miss leading hint.

[filimonov]

On my environment in fails even w/o any iterations, with plenty of free memory.
Centos 6.5, perl 5.10.1

[pivo42]

Ok, new insights.
The "faulty" files we had were packed by the apache kafka java lib (coincidence?).
To enforce the error we had to uncompress those files over and over again.

If you uncompress the faulty file/content and (re)compress it by Compress::LZ4Frame error never appears again.

As far I can see there are a bunch of fixes in the lz4 core lib. May an upgrade of the lz4.c will do the trick.

[Drako]

As far I can see there are a bunch of fixes in the lz4 core lib. May an upgrade of the lz4.c will do the trick.

This is what I intended to try first anyways.

[filimonov]

BTW: kafka used bad implementation of LZ4Frame in first versions.
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-57+-+Interoperable+LZ4+Framing

But that data came from new 'fixed' version of Kafka. And cli tool lz4 (quite old actually) uncompress that data w/o any problems, so i doubt that kafka is the reason.

➜ lz4 -v
*** LZ4 command line interface 64-bits r128, by Yann Collet (Jul  6 2015) ***

P.S. May be I should try to uncompress that with new version of lz4 and it will also fail? :D

PPS. Checked - newest version of lz4 also decompress that w/o problems.

[Drako]

I seem to have a bug in the handling of frames with no content size header.

[Drako]

yep, found the issue, the loop condition was bad :D

[Drako]

okay, 0.012001 release is uploaded and should be available in a few hours.

[filimonov]

Thank you guys, you're awesome. Now everything work as expected 👍