Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decompression failure #15

Closed
alecmocatta opened this issue Jul 20, 2020 · 1 comment
Closed

Decompression failure #15

alecmocatta opened this issue Jul 20, 2020 · 1 comment

Comments

@alecmocatta
Copy link

Really like this crate, nice work @main--!

I bumped into a decompression that fails. It's truncated below but here's a link to the full code.

fn main() {
    let input = [0x80, 0x01, 0x04, 0x90, 0x4E, 0xD5, 0x8D, 0xA5, 0xDF, 0xA8, 0xE3, 0xDA, 0xE9, 0x3B, 0xE9, 0x8E, 0xF9, 0xA0, 0xDE, 0x81, 0xC2, 0x84, 0xF9, 0x01, 0x40, 0x40, 0x40, 0x40, 0x87, 0x8C, 0x65, 0x58, 0xFB, 0xF2, 0xB9, 0x23, 0xDF, 0xCB, 0xE2, 0x07, 0x34, 0x2A, 0x92, 0x15, 0x3C, 0xA1, 0x8E, 0x76, 0x1F, 0xB2, 0xED, 0x03, 0xA0, 0xEB, 0x04, 0xA2, 0x25, 0x64, 0x78, 0x2A, 0x64, 0xBF, 0x2E, 0x65, 0x9F, 0xB3, 0x7A, 0x2D, 0x17, 0xC7, 0xBC, 0x67, 0x29, 0x3E, 0x9F, 0x76, 0xA0, 0xAE, 0xAC, 0xB0, 0x4B, 0x5C, 0x30, 0xC9, 0xD0, 0x41, 0x3C, 0x8C, 0x54, 0x24, 0x38, 0x11, 0xEE, 0x19, 0xDF, 0xB5, 0x72, 0xAA, 0x8A, 0x58, 0xF1, 0x2E, 0xCB, 0x29, 0x97, 0x8D, 0xBA, 0xEC, 0xAA, 0x63, 0x1D, 0x29, 0xCB, 0xDB, 0x69, 0x06, 0x90, 0xF5, 0xA0, 0x75, 0xE5, 0x6B, 0x2F, 0xBD, 0xE7, 0xDD, 0xB8, 0x19, 0xDB, 0x1A, 0xB9, 0xF3, 0x0F, 0x37, 0xF3, 0x02, 0xA9, 0x7D, 0x07, 0x16, 0x90, 0x48, 0x1A, 0x2B, 0xB4, 0x3E, 0x7E, 0xDD, 0x3E, 0xFE, 0x15, 0xA1, 0xDB, 0x56, 0x20, 0xDF, 0xB2, 0xD7, 0xA7, 0x75, 0xCF, 0xEC, 0xAD, 0x97, 0x38, 0xE3, 0x6E, 0x1D, 0x1C, 0x51, 0xE9, 0x5A, 0x40, 0x7D, 0xD7, 0xCC, 0x1D, 0x2A, 0x25, 0x31, 0xD4, 0xEC, 0x1C, 0x8D, 0x2D, 0x43, 0x45, 0x11, 0xC5, 0x02, 0xE2, 0xAE, 0xF1, 0xDC, 0x4E, 0xD9, 0x87, 0x0D, 0xB6, 0xB7, 0x3A, 0x29, 0x78, 0xE3, 0x53, 0xE8, 0xDD, 0x71, 0xC8, 0x2E, 0x4A, 0x4C, 0x9A, 0x26, 0x4A, 0xE9, 0x36, 0x08, 0x17, 0x68, 0x18, 0x3F, 0x13, 0x04, 0xCA, 0x61, 0x45, 0x66, 0x9A, 0xC1, 0x09, 0xEB, 0xCA, 0x8F, 0x50, 0xA3, 0xFA, 0x0E, 0x28, 0x37, 0xD3, 0xCA, 0xC4, 0x38, 0xD5, 0x6C, 0x79, 0x9B, 0x1F, 0x8F, 0xA8, 0x8E, 0x91, 0x87, 0x09, 0x31, 0xFB, 0x75, 0xCA, 0xDA, 0xC9, 0x1C, 0x3D, 0xD6, 0xF3, 0x79, 0x87, 0xF9, 0xEE, 0x85, 0x19, 0x9C, 0x6A, 0xC8, 0xA9, 0xA4, 0x76, 0x61, 0x8A, 0xD8, 0x51, 0x3C, 0x70, 0x4E, 0x79, 0x19, 0x58, 0xD5, 0x66, 0x77, 0xC2, 0x71, 0x4D, 0xE3, 0xDB, 0xB2, 0x3E, 0xB4, 0x05, 0x43, 0x62, 0xEB, 0x01, 0xD2, 0x74, 0xA9, 0xD4, 0x7A, 0xCC, 0xB8, 0x69, 0x08, 0xE8, 0x99, 0x28, 0x2C, 0xE9, 0xFC, 0x58, 0x69, 0x68, 0x4B, 0x48, 0xCC, 0x76, 0xFA, 0x83, 0x04, 0x78, 0xA7, 0xF6, 0x20, 0xF2, 0x59, 0x65, 0x23, 0x49, 0xD0, 0x54, 0x77, 0x33, 0xC8, 0xD8, 0xE5, 0x20, 0xB3, 0xB2, 0x76, 0x3C, 0x5E, 0x55, 0x87, 0xFB, 0xF7, 0x0B, 0x89, 0xD7, 0xF7, 0x2B, 0xD1, 0xA5, 0x66, 0x4F, 0x84, 0x94, 0x44, 0xBD, 0x65, 0x5D, 0x15, 0x27, 0x10, 0xC3, 0x21, 0xC2, 0xB4, 0xC7, 0x90, 0x95, 0x02, 0x7D, 0x28, 0x6E, 0xD1, 0xF4, 0xE1, 0x5D, 0x83, 0x79, 0xF5, ...];
    let mut compressed = vec![];
    lz_fear::framed::CompressionSettings::default().independent_blocks(false).block_size(64*1024).compress(&input[..], &mut compressed).unwrap();

    let mut decompressed = vec![];
    lz_fear::framed::LZ4FrameReader::new(&*compressed).unwrap().into_read().read_to_end(&mut decompressed).unwrap();
    assert_eq!(&input[..], &*decompressed);
}
"underlying IO error: the raw LZ4 decompression failed (data corruption?)"

The compressed bytes are identical to the output of the lz4 crate with default config.

main-- added a commit that referenced this issue Sep 12, 2020
Seems like all prior testing (and even fuzzing) missed the case where a block is encoded as incompressible and then the following block has backreferences into the previous block.
We were handling this case incorrectly by ignoring incompressible/literal blocks entirely w.r.t. updating the block rollover window.

Fixed by moving the window update code so that it runs for both compressed and literal blocks.
@main--
Copy link
Owner

main-- commented Sep 12, 2020

Thank you!

@main-- main-- closed this as completed Sep 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants