Parsing of archive structure failed #27
Comments
|
Thanks for this report. According to the exception you quote, it should be a problem with |
|
For the full debugging data, you can take a look here: typelead/eta#184 The most important part I'll reproduce here:
This is 98 bytes. An empty zip file will be 22 bytes.
createArchive p (return ())
withArchive p $ do
forM_ files $ \(path, contents) -> do
entrySel <- mkEntrySelector path
addEntry compress contents entrySelLet me know if it is at all possible for the code above to generate the output above for suitable values of the free variables. EDIT: compress = Deflate in the case above. |
|
Sorry, I don't understand. You claim that I haven't read entrie thread you link to, but is it possible that the file is incomplete? You say it should be much bigger. |
|
I have my doubts though that |
|
I used Please clone that repo, and run I'm starting to have a feeling it's from my side, but it seems strange that it only happens on certain platforms. I'm waiting on more information from the guy to trace the bug to a certain execution point. |
|
Looks like it fails to detect ECD because it's placed on offset that is not |
|
@rahulmutt Any progress with your investigation? |
|
@mrkkrp Some very interesting stuff turned up. First, I changed the code I had above from to and it ended up creating almost the correct jar with a bit of junk. In particular, there's a file called One interesting observation: In the previous info I gave you, an empty jar is 22 bytes and an empty corrupted jar was 98 bytes. Meaning Do you have any idea why those extra bytes may be generated on some systems and not others? Let me know if you need anymore info from my side. |
|
This is interesting. I'm not sure why this could happen, the code that generates central directory records is pure Haskell, there should be nothing platform-dependent. I need to take a closer look though. |
|
Can we try to run |
|
Yeah of course. You can even add debugging logs inside of the |
|
Great, can you then ask the people who are facing the issue try to run the test suite on their machines? |
|
Hi, i reported the original issue with eta and running the zip test suite i've got this output: https://gist.github.com/jneira/2d2a1c64895b480919710d10f353efcf |
|
OK, I consider it a bug now. The next thing I'll try is to reproduce it on a Windows machine I have access to. Thanks for reporting this. |
|
Actually I'll probably add Appveyor testing, it should catch this. |
|
I was able to reproduce the issue. The fix is coming. |
|
FTR, what is happening is that central directory for some reason (not sure why yet) gets too long to be stored without Zip64 extension (longer than So we write Zip64 end of central directory and Zip64 end of central directory locator before writing the vanilla ECD. With these though, the algorithm for ECD detection chokes (not sure exactly why). I have modified the Travis script to make a build when we unconditionally use Zip64 (because it's kinda difficult to catch this with normal generative tests, we need something big), this is tested along with normal tests now. It fails as it should: https://travis-ci.org/mrkkrp/zip/builds/199397711 the set of failures is similar to https://gist.github.com/jneira/2d2a1c64895b480919710d10f353efcf Now I need to answer the questions:
|
|
@jneira In the meantime, could you add |
|
@rahulmutt I believe I have fixed at least part of the issue in 39ffb9a. The 76 bytes is combined size of zip 64 end of central directory and zip 64 ECD locator. For some reason the lib decides to use zip64 extension and there is indeed a bug preventing it from reading empty zip64 archives (this is also the cause of test most test failures @jneira showed us). I fixed this bug and it allowed me to load https://github.com/mrkkrp/zip/blob/master/Codec/Archive/Zip/Internal.hs#L484-L486 we need either number of files greater than |
|
@rahulmutt Please provide the first file from https://gist.github.com/rahulmutt/1ef23bb08552348cdb3c7e8492a4d65d as a normal file. I tried to restore it from the dump, but I have no time or intention to fight this tool, I just need the file, desirably in its “proper” form and in its “corrupted” form. |
|
Going to release version 0.1.6 with the fix, if the problem persists, feel free to reopen. |
|
@jneira FTR, I can't reproduce your failures on my Windows machine, but they should be fixed in version 0.1.6. It's still very interesting why In any case, without further input, I can't really help much, one bug I've found and fixed, but there are still strange things I don't understand. |
|
@mrkkrp hi, i've rerun the test suite after pulling (but without printing cdSize). The result is in https://gist.github.com/jneira/2d2a1c64895b480919710d10f353efcf (11 failures) writeCD h comment m = do
let cd = runPut (putCD m)
cdOffset <- hTell h
B.hPut h cd -- write central directory
let totalCount = M.size m
cdSize = B.length cd
needZip64 =
#ifdef USE_ZIP64_ECD
True
#else
totalCount >= 0xffff
|| cdSize >= 0xffffffff
|| cdOffset >= 0xffffffff
#endif
print $ "cdSize:" ++ show cdSize
when needZip64 $ doi'll post the test suite output with the print line asap |
|
@jneira Also, give the following a try? We can see which condition is causing the Zip64 record to be used. import Debug.Trace(traceShow)
...
writeCD h comment m = do
let cd = runPut (putCD m)
cdOffset <- hTell h
B.hPut h cd -- write central directory
let totalCount = M.size m
cdSize = B.length cd
needZip64 =
#ifdef USE_ZIP64_ECD
True
#else
totalCount >= 0xffff
|| cdSize >= 0xffffffff
|| cdOffset >= 0xffffffff
#endif
print $ "cdSize:" ++ show cdSize
when (traceShow (totalCount, cdSize, cdOffset) needZip64) $ do |
|
I've tried to run the test suite with the trace in win xp 32 bits but the test target fails: |
|
I had a different issue: snoyberg/bzlib-conduit#3, but also about unknown symbols. Not sure what's going on, Haskell on Windows seems to be still fragile :-| |
|
@jneira What you see (in printout of test suite) proves that even without the new |
|
@mrkkrp Even on my Windows machine I was unable to reproduce what @jneira and a couple others have reported. It's not a windows-specific issue, because it was reproduced on a 32-bit LinuxMint system. |
|
Good, still it's quite strange. @jneira is this a 32 bit machine? Maybe it has something to do with that? |
|
I forked this repo, added some trace messages, and created a new branch on the Local working installation: The Appyveyor build: The interesting observation here is that: returns true on the failing build even if |
|
OK, it means that |
|
This is easy to fix though, I'll probably replace all the literals with a constant of type I'll push the fix very soon and release 0.1.7. Thanks a lot for reporting this and helping to investigate, @rahulmutt, @jneira. |
|
@rahulmutt I believe we won't hear about this issue again. Make sure Eta requires version 0.1.7 of |
|
Just tested ran the Appveyor build on new branch w/ the updated dependency and it's working. Will update the dependency on master now, thanks! |
|
great! thanks for the support |
This error is coming up for Eta users and moreover it's not very consistent - it happens for only a handful of people. Initially, it was happening only on Windows, but we recently got a repro case on Linux.
The last time I filed an issue, I changed the algorithm to select the largest jar file among a set of jar files to merge as the base and merged the rest into that one. After this change, I noticed that the Windows build failure for Eta (due to the error above) progressed to one more step (the step it was able to pass was the case of merging a single zip file). This implies usage of the
zipAPI is somehow causing corrupt zip files on some specific platform configurations. Because it's inconsistent, I'm wondering whether it is a problem with some of the transitive C dependencies thatzipuses. What do you think may be causing it?In the meantime, I'll try to make a minimal test case of the
ziplibrary where I create an empty zip file and add a single entry and have the people with the affected systems try compiling that program and opening up the zip file.The text was updated successfully, but these errors were encountered: