A few test failures on s390x (big-endian)

[musicinmybrain]

Describe the bug

When running cargo test on s390x (a big-endian architecture), several failures occur:

failures:

---- write::test::fuzz_crash_2024_07_17 stdout ----
Error: InvalidArchive("Could not find EOCD")

---- write::test::fuzz_crash_2024_07_19 stdout ----
Error: InvalidArchive("Could not find EOCD")

---- write::test::test_deep_copy stdout ----

thread 'write::test::test_deep_copy' panicked at src/write.rs:2282:53:
called `Result::unwrap()` on an `Err` value: InvalidArchive("Could not find EOCD")

---- write::test::test_fuzz_crash_2024_06_17a stdout ----
Error: InvalidArchive("Could not find EOCD")

---- write::test::test_fuzz_failure_2024_06_08 stdout ----
Error: InvalidArchive("Could not find EOCD")

---- write::test::test_fuzz_crash_2024_06_18b stdout ----
Error: FileNotFound


failures:
    write::test::fuzz_crash_2024_07_17
    write::test::fuzz_crash_2024_07_19
    write::test::test_deep_copy
    write::test::test_fuzz_crash_2024_06_17a
    write::test::test_fuzz_crash_2024_06_18b
    write::test::test_fuzz_failure_2024_06_08

test result: FAILED$<2>. 111 passed; 6 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.31s

To Reproduce
Steps to reproduce the behavior:

1. Work on real s390x hardware, in a virtual machine under qemu-system emulation, or in a chroot (e.g. a mock chroot) under qemu-user-static emulation.
2. git clone https://github.com/zip-rs/zip2.git
3. cd zip2
4. cargo test

Expected behavior
All tests pass.

Screenshots
N/A

Desktop (please complete the following information):

• OS: Fedora Linux (Rawhide)
• Browser N/A
• Version zip2 2.2.3, c6ba201

Additional context
These failures were first observed on real hardware while working on a proposed update to the rust-zip package in Fedora, then were reproduced in emulation. I have non-interactive access to test proposed patches on real hardware (via RPM package test-builds), and I can easily do interactive experiments in an emulated chroot. Please let me know if there’s anything else I can provide that would make this easier to understand and fix.

[Pr0methean]

Could you please provide your build with the tests? I tried to produce one with

rustup +stable target add s390x-unknown-linux-gnu
cargo clean
cargo +stable build --tests --target s390x-unknown-linux-gnu

so that I'd be able to reproduce the issue in qemu-s390x-static, but I got

error occurred in cc-rs: failed to find tool "s390x-linux-gnu-gcc": No such file or directory (os error 2)

Also, could you please run the failing tests with RUST_BACKTRACE=full and provide the entire output?

[musicinmybrain]

Also, could you please run the failing tests with RUST_BACKTRACE=full and provide the entire output?

Certainly.

running 121 tests
test aes_ctr::tests::crypt_aes_256_0_byte ... ok
test aes_ctr::tests::crypt_aes_128_40_byte ... ok
test aes_ctr::tests::crypt_aes_192_40_byte ... ok
test aes_ctr::tests::crypt_aes_192_5_byte ... ok
test aes_ctr::tests::crypt_aes_128_5_byte ... ok
test aes_ctr::tests::crypt_aes_256_40_byte ... ok
test aes_ctr::tests::crypt_aes_256_5_byte ... ok
test compression::test::to_eq_from ... ok
test cp437::test::ascii ... ok
test cp437::test::example_slice ... ok
test compression::test::from_eq_to ... ok
test compression::test::to_display_fmt ... ok
test crc32::test::test_byte_by_byte ... ok
test crc32::test::test_zero_read ... ok
test crc32::test::test_empty_reader ... ok
test read::stream::test::invalid_cde_number_of_files_allocation_greater_offset ... ok
test cp437::test::example_vec ... ok
test read::stream::test::invalid_offset2 ... ok
test read::stream::test::invalid_cde_number_of_files_allocation_smaller_offset ... ok
test cp437::test::to_char_valid ... ok
test read::test::deflate64_not_enough_space ... ok
test read::test::invalid_cde_number_of_files_allocation_greater_offset ... ok
test read::stream::test::zip_read_streaming ... ok
test read::test::invalid_cde_number_of_files_allocation_smaller_offset ... ok
test read::stream::test::file_and_dir_predicates ... ok
test read::test::invalid_offset2 ... ok
test read::test::invalid_offset ... ok
test read::stream::test::invalid_offset ... ok
test read::test::file_and_dir_predicates ... ok
test read::test::test_can_create_destination ... ok
test read::test::test_is_symlink ... ok
test read::stream::test::test_can_create_destination ... ok
test read::test::test_utf8 ... ok
test read::test::test_utf8_2 ... ok
test read::test::test_utf8_extra_field ... ok
test read::test::deflate64_index_out_of_bounds ... ok
test read::test::zip64_magic_in_filenames ... ok
test read::test::test_read_with_data_descriptor ... ok
test read::test::zip_contents ... ok
test spec::test::block_serde ... ok
test read::test::zip_clone ... ok
test types::test::datetime_default ... ok
test read::test::zip64_with_leading_junk ... ok
test types::test::datetime_equality ... ok
test types::test::datetime_bounds ... ok
test read::test::zip_read_streaming ... ok
test types::test::datetime_order ... ok
test types::test::datetime_max ... ok
test types::test::offset_datetime_try_from_bounds ... ok
test types::test::datetime_display ... ok
test types::test::datetime_try_from_offset_datetime ... ok
test types::test::datetime_try_from_bounds ... ok
test types::test::system ... ok
test types::test::offset_datetime_try_from_datetime ... ok
test types::test::time_at_january ... ok
test read::stream::test::test_cannot_symlink_outside_destination ... ok
test types::test::time_out_of_bounds ... ok
test types::test::sanitize ... ok
test read::test::test_cannot_symlink_outside_destination ... ok
test types::test::time_conversion ... ok
test write::test::crash_with_no_features ... ok
test write::test::duplicate_filenames ... ok
test write::test::path_to_string ... ok
test write::test::fuzz_crash_2024_07_21 ... ok
test write::test::fuzz_crash_2024_06_21 ... ok
test write::test::fuzz_crash_2024_07_19 ... FAILED
test write::test::fuzz_crash_2024_07_17 ... FAILED
test write::test::fuzz_crash_2024_07_20 ... ok
test write::test::remove_encrypted_aligned_symlink ... ok
test write::test::test_alignment ... ok
test write::test::test_crash_short_read ... ok
test write::test::test_filename_looks_like_zip64_locator_2 ... ok
test write::test::test_filename_looks_like_zip64_locator ... ok
test write::test::remove_encrypted_file ... ok
test write::test::test_filename_looks_like_zip64_locator_2a ... ok
test write::test::test_alignment_2 ... ok
test write::test::test_filename_looks_like_zip64_locator_3 ... ok
test write::test::test_fuzz_crash_2024_06_13a ... ok
test write::test::test_fuzz_crash_2024_06_13b ... ok
test write::test::test_fuzz_crash_2024_06_14 ... ok
test write::test::test_fuzz_crash_2024_06_14a ... ok
test write::test::remove_shallow_copy_keeps_original ... ok
test write::test::test_fuzz_crash_2024_06_14b ... ok
test write::test::test_fuzz_crash_2024_06_14c ... ok
test write::test::test_fuzz_crash_2024_06_14d ... ok
test write::test::test_fuzz_crash_2024_06_14e ... ok
test write::test::test_filename_looks_like_zip64_locator_5 ... ok
test write::test::test_filename_looks_like_zip64_locator_4 ... ok
test write::test::test_fuzz_crash_2024_06_18 ... ok
test write::test::test_fuzz_crash_2024_06_17a ... FAILED
test write::test::test_fuzz_crash_2024_06_18a ... ok
test write::test::test_fuzz_crash_2024_06_17b ... ok
test write::test::test_fuzz_crash_2024_06_19 ... ok
test write::test::test_invalid_extra_data ... ok
test write::test::test_invalid_extra_data_unreserved ... ok
test write::test::test_fuzz_failure_2024_06_08 ... FAILED
test write::test::test_fuzz_crash_2024_06_17 ... ok
test write::test::test_short_extra_data ... ok
test write::test::unix_permissions_bitmask ... ok
test write::test::write_empty_zip ... ok
test write::test::write_mimetype_zip ... ok
test write::test::write_non_utf8 ... ok
test write::test::test_path_normalization ... ok
test write::test::write_symlink_simple ... ok
test write::test::write_symlink_wonky_paths ... ok
test write::test::write_zip_dir ... ok
test write::test::test_shallow_copy ... ok
test write::test::fuzz_crash_2024_07_19a ... ok
test aes::tests::crypt_aes_128_5_byte ... ok
test aes::tests::crypt_aes_128_40_byte ... ok
test write::test::test_fuzz_failure_2024_05_08 ... ok
test aes_ctr::tests::new_with_wrong_key_size - should panic ... ok
test write::test::test_deep_copy ... FAILED
test aes::tests::crypt_aes_192_5_byte ... ok
test aes::tests::crypt_aes_192_40_byte ... ok
test write::test::zopfli_empty_write ... ok
test write::test::test_fuzz_crash_2024_06_18b ... FAILED
test aes::tests::crypt_aes_256_40_byte ... ok
test aes::tests::crypt_aes_256_5_byte ... ok
test aes::tests::crypt_aes_256_0_byte ... ok
test read::test::test_64k_files ... ok

failures:

---- write::test::fuzz_crash_2024_07_19 stdout ----
Error: InvalidArchive("Could not find EOCD")

---- write::test::fuzz_crash_2024_07_17 stdout ----
Error: InvalidArchive("Could not find EOCD")

---- write::test::test_fuzz_crash_2024_06_17a stdout ----
Error: InvalidArchive("Could not find EOCD")

---- write::test::test_fuzz_failure_2024_06_08 stdout ----
Error: InvalidArchive("Could not find EOCD")

---- write::test::test_deep_copy stdout ----

thread 'write::test::test_deep_copy' panicked at src/write.rs:2296:53:
called `Result::unwrap()` on an `Err` value: InvalidArchive("Could not find EOCD")
stack backtrace:
   0:      0x2aa0044c460 - std::backtrace_rs::backtrace::libunwind::trace::h5850a37c4a5391d6
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:      0x2aa0044c460 - std::backtrace_rs::backtrace::trace_unsynchronized::h1a7c1b60516a7935
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:      0x2aa0044c460 - std::sys::backtrace::_print_fmt::h015a9f911184916f
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/sys/backtrace.rs:66:9
   3:      0x2aa0044c460 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hf7a6991e9e6f1085
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/sys/backtrace.rs:39:26
   4:      0x2aa00487df4 - core::fmt::rt::Argument::fmt::hb71cf6ba91ec4818
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/fmt/rt.rs:177:76
   5:      0x2aa00487df4 - core::fmt::write::hb4fe9157fd272558
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/fmt/mod.rs:1440:21
   6:      0x2aa004463ce - std::io::Write::write_fmt::hf0f6aa3296a4f2b2
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/io/mod.rs:1887:15
   7:      0x2aa0044c160 - std::sys::backtrace::BacktraceLock::print::hd4c662042c8020e8
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/sys/backtrace.rs:42:9
   8:      0x2aa0044e046 - std::panicking::default_hook::{{closure}}::he43b9aa92465c75a
   9:      0x2aa0044de02 - std::panicking::default_hook::h494878c3e8b78a01
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panicking.rs:319:9
  10:      0x2aa001fd6ca - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h46d297b01d180423
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/alloc/src/boxed.rs:2007:9
  11:      0x2aa001fd6ca - test::test_main::{{closure}}::hdb340f5eb7e13ecd
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/test/src/lib.rs:134:21
  12:      0x2aa0044ea50 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h0c1550c6989b98e4
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/alloc/src/boxed.rs:2007:9
  13:      0x2aa0044ea50 - std::panicking::rust_panic_with_hook::h7c4a31facc6cde7b
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panicking.rs:836:13
  14:      0x2aa0044e5f2 - std::panicking::begin_panic_handler::{{closure}}::h59ff935700ac6e64
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panicking.rs:701:13
  15:      0x2aa0044cbb4 - std::sys::backtrace::__rust_end_short_backtrace::h71b7863d4e760613
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/sys/backtrace.rs:168:18
  16:      0x2aa0044e1ca - rust_begin_unwind
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panicking.rs:692:5
  17:      0x2aa0003ed2a - core::panicking::panic_fmt::h1b7c3dcef0cfd1c9
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/panicking.rs:75:14
  18:      0x2aa0003f27e - core::result::unwrap_failed::hb92ce167806aaa77
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/result.rs:1704:5
  19:      0x2aa000b2762 - core::result::Result<T,E>::unwrap::hfa4eed808bc40cc5
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/result.rs:1109:23
  20:      0x2aa000b2762 - zip::write::test::test_deep_copy::hd35cdbcf8fff57b2
                               at /builddir/zip2/src/write.rs:2296:26
  21:      0x2aa0005ddac - zip::write::test::test_deep_copy::{{closure}}::hd4a85286b49eb52e
                               at /builddir/zip2/src/write.rs:2276:24
  22:      0x2aa000e5674 - core::ops::function::FnOnce::call_once::h653fb090892f6af9
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/ops/function.rs:250:5
  23:      0x2aa00203c54 - core::ops::function::FnOnce::call_once::h62950594154ed602
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/ops/function.rs:250:5
  24:      0x2aa00203c54 - test::__rust_begin_short_backtrace::h6fb9c83803926c00
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/test/src/lib.rs:632:18
  25:      0x2aa00202822 - test::types::RunnableTest::run::h77d70bef566472e8
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/test/src/types.rs:145:40
  26:      0x2aa00202822 - test::run_test_in_process::{{closure}}::h7b36f0a7255abb9e
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/test/src/lib.rs:655:60
  27:      0x2aa00202822 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h62c86c10f4c70ebd
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/panic/unwind_safe.rs:272:9
  28:      0x2aa00202822 - std::panicking::try::do_call::h2de310c89d1be6ce
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panicking.rs:584:40
  29:      0x2aa00202822 - std::panicking::try::hff8e2db48c8da56a
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panicking.rs:547:19
  30:      0x2aa00202822 - std::panic::catch_unwind::hb64da738ba779ec9
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panic.rs:358:14
  31:      0x2aa00202822 - test::run_test_in_process::ha24b693add01d9f4
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/test/src/lib.rs:655:27
  32:      0x2aa00202822 - test::run_test::{{closure}}::he958199d53a057c5
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/test/src/lib.rs:576:43
  33:      0x2aa001af022 - test::run_test::{{closure}}::h336a52a30c658d41
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/test/src/lib.rs:606:41
  34:      0x2aa001af022 - std::sys::backtrace::__rust_begin_short_backtrace::h0dd33c4a0fdcf53d
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/sys/backtrace.rs:152:18
  35:      0x2aa001b3fd8 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h5ce4fdd22094ae42
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/thread/mod.rs:564:17
  36:      0x2aa001b3fd8 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hf9327272e867e8a9
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/panic/unwind_safe.rs:272:9
  37:      0x2aa001b3fd8 - std::panicking::try::do_call::h499e8d75f021b53a
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panicking.rs:584:40
  38:      0x2aa001b3fd8 - std::panicking::try::hea55799f4a339b7d
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panicking.rs:547:19
  39:      0x2aa001b3fd8 - std::panic::catch_unwind::hbe76eaca26cac092
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/panic.rs:358:14
  40:      0x2aa001b3fd8 - std::thread::Builder::spawn_unchecked_::{{closure}}::h708bebb6e4dcb1a9
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/thread/mod.rs:562:30
  41:      0x2aa001b3fd8 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc1f5e612b1b452a3
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/core/src/ops/function.rs:250:5
  42:      0x2aa00454d3e - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hba166fd935953c16
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/alloc/src/boxed.rs:1993:9
  43:      0x2aa00454d3e - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h20f1b302f67e0830
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/alloc/src/boxed.rs:1993:9
  44:      0x2aa00454d3e - std::sys::pal::unix::thread::Thread::new::thread_start::hb95b536084f8b345
                               at /builddir/build/BUILD/rust-1.85.0-build/rustc-1.85.0-src/library/std/src/sys/pal/unix/thread.rs:106:17
  45:     0x7f8c5bec0f1c - start_thread
  46:     0x7f8c5bf3e8b8 - <unknown>
  47:                0x0 - <unknown>

---- write::test::test_fuzz_crash_2024_06_18b stdout ----
Error: FileNotFound


failures:
    write::test::fuzz_crash_2024_07_17
    write::test::fuzz_crash_2024_07_19
    write::test::test_deep_copy
    write::test::test_fuzz_crash_2024_06_17a
    write::test::test_fuzz_crash_2024_06_18b
    write::test::test_fuzz_failure_2024_06_08

test result: FAILED. 115 passed; 6 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.97s

error: test failed, to rerun pass `--lib`

[musicinmybrain]

Could you please provide your build with the tests? […]

Could you please clarify exactly which files you’re asking for?

(If it helps, I’m working on Fedora 41 on an x86_64 host; I have the qemu-user-static package installed, I have installed mock and added my user to the mock group, and I’m working in a fedora-rawhide-s390x chroot with rust, cargo, git, and vim-enhanced installed.)

[Pr0methean]

Oh okay, I thought you were cross-compiling from outside the VM and producing an executable. This may take me a while to reproduce, and longer still to cover in CI, since I use Ubuntu and not (as mock clearly assumes) an RPM-based distro.

[musicinmybrain]

Oh okay, I thought you were cross-compiling from outside the VM and producing an executable. This may take me a while to reproduce, and longer still to cover in CI, since I use Ubuntu and not (as mock clearly assumes) an RPM-based distro.

I don’t have an Ubuntu system handy, but you should be able to use virt-manager in combination with qemu-system-s390x to boot an s390x version of something like Fedora Server. (There is also an Ubuntu Server for s390x, but they seem to be trying to trying to charge a lot of money for it, and I don’t quite understand whether a free version is available or not.)

I just tried this on Fedora and it worked: I downloaded Fedora-Server-KVM-41-1.4.s390x.qcow2, copied it into /var/lib/libvirt/images, started to create a new VM in Virtual Machine Manger, chose Import existing disk image, chose s390x under Architecture options, and was able to boot and configure an emulated VM, sudo dnf install git cargo, and reproduce this report. I believe this should work on an Ubuntu host as well. The emulation isn’t fast by any means, but it gets the job done eventually, and it’s straightforward to set up.

[decathorpe]

Depending on which standard library functionality (or other libc calls) are used during test, running them on the miri interpreter might work too, without having to resort to virtualization at all.

[Pr0methean]

@musicinmybrain I've found a bug in deep_copy_file that may explain all the failures, and confirmed that ed78171 fixes all the failures, except that fuzz_crash_2024_07_17 doesn't run in Miri because of a native bzip2 library. Could you please confirm that this test also passes with the change?

[musicinmybrain]

@musicinmybrain I've found a bug in deep_copy_file that may explain all the failures, and confirmed that ed78171 fixes all the failures, except that fuzz_crash_2024_07_17 doesn't run in Miri because of a native bzip2 library. Could you please confirm that this test also passes with the change?

I finally had time to revisit this. As a quick test, I tried backporting ed78171 as a patch for version 2.2.3 as currently packaged in Fedora. I can confirm that all tests now pass on s390x. (Next I’ll work on updating the package to the latest version.) Thanks for the quick analysis and for the fix!