Command ended with error IOError Custom kind InvalidData

[omac777]

I ran the mlar command. It seemed have behaved correctly and created .mla file.

./target/release/mlar create -p key.pub -o my_archive.mla clear-33590-live-desktop.iso pop-os_20.10_amd64_intel_1.iso archlinux-2020.11.01-x86_64.iso

But when I went to list the .mla contents it gave and error:

./target/release/mlar list -k key -i my_archive.mla
[!] Command ended with error: IOError(Custom { kind: InvalidData, error: "Invalid Data" })

As a result of this, using the mlar tool still doesn't inspire a level of confidence to use it for anything in production just yet.

[commial]

Hello,

Thanks for reporting this. I've been able to reproduce it, using the same files (based on filenames).

As a side note, mlar is considered production ready, but clearly bugs remains (despite tests & fuzzing strategies). Obviously, some tests are missings.

I'll come back to you after investigating it.

[commial]

#68 should fixes it. Already created archive are still valid.

Running a test similar to yours:

$ ./target/release/mlar list -i my_archive.mla -k key
archlinux-2020.11.01-x86_64.iso
clear-33590-live-desktop.iso
pop-os_20.10_amd64_intel_1.iso
$ ./target/release/mlar cat -i my_archive.mla -k key archlinux-2020.11.01-x86_64.iso > arch.test.iso
$ sha512sum archlinux-2020.11.01-x86_64.iso arch.test.iso
a357d437c31a2c61758591ddfb0a4bf73b77537e56b9cde8db2e2ffa76a0813f8f9d20e80cdae3a8d433b81349d644fc94b05d4414506d1b60609af08d15f9fd  archlinux-2020.11.01-x86_64.iso
a357d437c31a2c61758591ddfb0a4bf73b77537e56b9cde8db2e2ffa76a0813f8f9d20e80cdae3a8d433b81349d644fc94b05d4414506d1b60609af08d15f9fd  arch.test.iso

I also add a test (working on more than 32bits files) to avoid a regression on this situation later.

[omac777]

The regression test should be checking the extraction of every file and the integrity of every file otherwise the risk of one corrupt file may occur and the error will not surface.

$ time ./target/release/mlar create -p key.pub -o my_archive.mla clear-33590-live-desktop.iso pop-os_20.10_amd64_intel_1.iso archlinux-2020.11.01-x86_64.iso
clear-33590-live-desktop.iso
pop-os_20.10_amd64_intel_1.iso
archlinux-2020.11.01-x86_64.iso

real	3m40.195s
user	3m30.647s
sys	0m9.343s

$ ./target/release/mlar list -i my_archive.mla -k key
archlinux-2020.11.01-x86_64.iso
clear-33590-live-desktop.iso
pop-os_20.10_amd64_intel_1.iso

$ ./target/release/mlar cat -i my_archive.mla -k key archlinux-2020.11.01-x86_64.iso > arch.test.iso
$ sha512sum archlinux-2020.11.01-x86_64.iso arch.test.iso
a357d437c31a2c61758591ddfb0a4bf73b77537e56b9cde8db2e2ffa76a0813f8f9d20e80cdae3a8d433b81349d644fc94b05d4414506d1b60609af08d15f9fd  archlinux-2020.11.01-x86_64.iso
a357d437c31a2c61758591ddfb0a4bf73b77537e56b9cde8db2e2ffa76a0813f8f9d20e80cdae3a8d433b81349d644fc94b05d4414506d1b60609af08d15f9fd  arch.test.iso
$ rm arch.test.iso

$ ./target/release/mlar cat -i my_archive.mla -k key pop-os_20.10_amd64_intel_1.iso > popos.test.iso
$ sha512sum pop-os_20.10_amd64_intel_1.iso popos.test.iso
621875f183e38d5dc50d98298438ff68c0912f6ed2343e755425deb8e4341a7fa29d5fbb08a55a9beed0121c92ffbb1b54fea3740b14e1ffc7edfa3f02ed890a  pop-os_20.10_amd64_intel_1.iso
621875f183e38d5dc50d98298438ff68c0912f6ed2343e755425deb8e4341a7fa29d5fbb08a55a9beed0121c92ffbb1b54fea3740b14e1ffc7edfa3f02ed890a  popos.test.iso
$ rm popos.test.iso 

$ ./target/release/mlar cat -i my_archive.mla -k key clear-33590-live-desktop.iso > clear.test.iso
$ sha512sum clear-33590-live-desktop.iso clear.test.iso
658eabae6dad5b425e8997d27ff303004f73ce1434893b9a3780fe374def4efc849c58e033840b7fc80e8ad21d82fa586399f0e457971a5b841428c896568be6  clear-33590-live-desktop.iso
658eabae6dad5b425e8997d27ff303004f73ce1434893b9a3780fe374def4efc849c58e033840b7fc80e8ad21d82fa586399f0e457971a5b841428c896568be6  clear.test.iso
$ rm clear.test.iso 

$ rustc --version
rustc 1.50.0-nightly (2225ee1b6 2020-12-11)

$ cargo --version
cargo 1.50.0-nightly (d274fcf86 2020-12-07)

$ ./target/release/mlar --version
mlar 1.1.0

Please use b3sum rather than sha512sum as it is much faster and especially on larger files.

$ time sha512sum clear-33590-live-desktop.iso
658eabae6dad5b425e8997d27ff303004f73ce1434893b9a3780fe374def4efc849c58e033840b7fc80e8ad21d82fa586399f0e457971a5b841428c896568be6  clear-33590-live-desktop.iso

real	0m9.855s
user	0m9.295s
sys	0m0.560s

$ time b3sum clear-33590-live-desktop.iso
e7fd5329de134e4a600c61d9e224dc6d361f5901558acfedf5191e5e5ef6aefe  clear-33590-live-desktop.iso

real	0m2.939s
user	0m1.933s
sys	0m0.483s

[commial]

This is actually what the regression test is doing (

MLA/mla/src/lib.rs

Line 2069 in 911ed7d

fn more_than_u32_file() {

).

The sha512sum excerpt was only there to illustrate the fix.