Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bzip2: fix bug with duplicate read of CRC32 when it overlaps end of buffer and 'bzip2-done is thrown. #12

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

vseloved
Copy link

@vseloved vseloved commented Jul 26, 2017

This bug is quite rare and usually manifests only on large bzip2 files (several 100s of Megabytes in size). For instance, it can be reproduced while processing this part of wikipedia dump with the following simple script:

(with-open-file (f "enwiki-20170720-pages-meta-current12.xml-p3926863p5040436.bz2"
                 :element-type 'unsigned-byte)
   (let* ((in (chipz:make-decompressing-stream :bzip2 f))
          (in (flex:make-flexi-stream in :external-format :utf-8)))
      (loop :for line := (read-line in nil) :while line)))

At block 383 the invalid-bzip2-data error will be signalled.

We also had a conversation about the same bug some time ago in the issues (which are not accessible now, unfortunately).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant