Some cht tests failed for 32-bit MIPS and ARMv5TE targets

[tatsuya6502]

For v0.7.2, we copied some source files of the concurrent hash table (cht) from moka-cht crate and removed moka-cht from the dependencies. When we enabled tests for the cht in Moka, we saw some tests failed for 32-bit MIPS and ARMv5TE targets. We disabled these failing tests and released v0.7.2. #86 (comment)

Since then, I have done some investigation, and came to the following conclusion:

• Those failures were likely due to some incomplete parallelism supports in QEMU user space emulator.
  • This is confirmed by running the same tests on QEMU system emulator. (It was able to run the tests with no failures)
• Therefore, we can safely ignore these failing tests.

Here is a summary of my investigation:

The issue

Some cht tests failed for 32-bit MIPS and ARMv5TE targets. These platforms are used by the some users of aliyundrive-webdav because SoC chips with these architectures are still common in home Wi-Fi routers and NAS models.

Failing tests

mips-unknown-linux-musl and mipsel-unknown-linux-musl targets

1. concurrent_growth
2. concurrent_growth_and_removal
3. concurrent_insert_with_or_modify
4. concurrent_insertion
5. concurrent_insertion_and_removal
6. concurrent_overlapped_insertion
7. concurrent_overlapped_removal
8. concurrent_removal

Error

memory allocation of 1052 bytes failed. signal: 6, SIGABRT: process abort signal.

armv5te-unknown-linux-musleabi target

1. concurrent_overlapped_growth

Error

signal: 4, SIGILL: illegal instruction

Some backgrounds

• Moka uses cross-rs/cross for testing with cross compilation.
• For these platforms with failing tests, cross uses QEMU's user space emulator.
• Unlike QEMU system emulator (virtual machines), the user space emulator is not very mature and has some limitations and issues:
  • From cross's README:
    • "Testing support (cross test) is more complicated. It relies on QEMU emulation, so testing may fail due to QEMU bugs rather than bugs in your crate."
  • From the "User space emulator" chapter of QEMU manual:
    • "Threading: ... Note that not all targets currently emulate atomic operations correctly."

Investigation: Running the tests on a VM instead of the user space emulator

To check whether or not those failures are caused by the issues with the user space emulator, I ran the same tests using the system emulator with a MIPS-based emulated CPU (qemu-system-mips -M malta).

OpenWrt project distributes firmware (embedded Linux) files that can run on QEMU system emulators. One of these firmware files is for MIPS architecture: "OpenWrt in QEMU MIPS"

Steps

1. Ran tests using cross => It failed as expected.
2. Started qemu-system-mips (a virtual machine) using the OpenWrt firmware.
3. On the VM, copied the testing binary built at step 1.
4. Ran the binary on the VM.

Result (MIPS)

I verified that all cht tests succeeded on the MIPS system emulator. I concluded the failures were due to some issues in the user space emulator and we can safely ignore them.

root@OpenWrt:~# uname -a
Linux OpenWrt 5.4.154 #0 SMP Sun Oct 24 09:01:35 2021 mips GNU/Linux

root@OpenWrt:~# ./moka-tests --test-threads 1 cht

running 18 tests
test cht::segment::map::tests::concurrent_growth ... ok
test cht::segment::map::tests::concurrent_growth_and_removal ... ok
test cht::segment::map::tests::concurrent_insert_with_or_modify ... ok
test cht::segment::map::tests::concurrent_insertion ... ok
test cht::segment::map::tests::concurrent_insertion_and_removal ... ok
test cht::segment::map::tests::concurrent_overlapped_growth ... ok
test cht::segment::map::tests::concurrent_overlapped_insertion ... ok
test cht::segment::map::tests::concurrent_overlapped_removal ... ok
test cht::segment::map::tests::concurrent_removal ... ok
test cht::segment::map::tests::drop_many_values ... ok
test cht::segment::map::tests::drop_many_values_concurrent ... ok
test cht::segment::map::tests::drop_value ... ok
test cht::segment::map::tests::growth ... ok
test cht::segment::map::tests::insert_with_or_modify ... ok
test cht::segment::map::tests::insertion ... ok
test cht::segment::map::tests::removal ... ok
test cht::segment::map::tests::remove_if ... ok
test cht::segment::map::tests::single_segment ... ok

test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 43 filtered out; finished in 2145.54s

A note on ARMv5TE architecture

I could not do the same test for ARMv5TE.

• OpenWrt distributes firmware file for ARM architecture on QEMU.
  • However it seems the firmware is for ARMv7 architecture.
• I also could not find a way to run the system emulator with ARMv5TE-based CPU.
  • It seems Versatile boards (versatileab, versatilepb) emulate ARMv5TE-base SoCs:
    • https://qemu.readthedocs.io/en/latest/system/arm/versatile.html
    • https://www.elinux.org/ARM_Versatile
  • However, I could not run it on my QEMU installation.
    • Maybe because I use a wrong OpenWrt firmware? I guess I have to build the firmware by myself.

[tatsuya6502]

Terminal log from the testing (MIPS)

On the host

$ cross test --target mips-unknown-linux-musl --no-default-features
    Finished test [unoptimized + debuginfo] target(s) in 0.34s
     Running unittests (/target/mips-unknown-linux-musl/debug/deps/moka-a8b7658fdc5b39d9)

running 61 tests
test cht::segment::map::tests::concurrent_growth ... memory allocation of 1052 bytes failed
error: test failed, to rerun pass '--lib'

Caused by:
  process didn't exit successfully: `qemu-mips /target/mips-unknown-linux-musl/debug/deps/moka-a8b7658fdc5b39d9` (signal: 6, SIGABRT: process abort signal)
^C

$ file ./target/mips-unknown-linux-musl/debug/deps/moka-a8b7658fdc5b39d9
./target/mips-unknown-linux-musl/debug/deps/moka-a8b7658fdc5b39d9:
  ELF 32-bit MSB shared object, MIPS, MIPS32 rel2 version 1,
  dynamically linked (uses shared libs), with unknown capability 0x41000000 = 0xf676e75,
  with unknown capability 0x10000 = 0x70403, not stripped

In the VM (MIPS)

$ qemu-system-mips -version
QEMU emulator version 6.1.0
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

$ qemu-system-mips -M malta \
    -hda openwrt-21.02.1-malta-be-rootfs-ext4.img \
    -kernel openwrt-21.02.1-malta-be-vmlinux.elf \
    -nographic -append "root=/dev/sda console ttyS0"

 ...
 OpenWrt 21.02.1, r16325-88151b8303

root@OpenWrt:~# uname -a
Linux OpenWrt 5.4.154 #0 SMP Sun Oct 24 09:01:35 2021 mips GNU/Linux

root@OpenWrt:~# scp -p $REMOTE_HOST:moka/target/mips-unknown-linux-musl/debug/deps/moka-a8b7658fdc5b39d9 moka-tests

root@OpenWrt:~# ls -l
-rwxr-xr-x    1 root     root      68371460 Feb  7 00:53 moka-tests

root@OpenWrt:~# ./moka-tests --test-threads 1 cht

running 18 tests
test cht::segment::map::tests::concurrent_growth ... ok
test cht::segment::map::tests::concurrent_growth_and_removal ... ok
test cht::segment::map::tests::concurrent_insert_with_or_modify ... ok
test cht::segment::map::tests::concurrent_insertion ... ok
test cht::segment::map::tests::concurrent_insertion_and_removal ... ok
test cht::segment::map::tests::concurrent_overlapped_growth ... ok
test cht::segment::map::tests::concurrent_overlapped_insertion ... ok
test cht::segment::map::tests::concurrent_overlapped_removal ... ok
test cht::segment::map::tests::concurrent_removal ... ok
test cht::segment::map::tests::drop_many_values ... ok
test cht::segment::map::tests::drop_many_values_concurrent ... ok
test cht::segment::map::tests::drop_value ... ok
test cht::segment::map::tests::growth ... ok
test cht::segment::map::tests::insert_with_or_modify ... ok
test cht::segment::map::tests::insertion ... ok
test cht::segment::map::tests::removal ... ok
test cht::segment::map::tests::remove_if ... ok
test cht::segment::map::tests::single_segment ... ok

test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 43 filtered out; finished in 2145.54s

root@OpenWrt:~# poweroff

[tatsuya6502]

Closing this issue for now without confirming the one failing test on ARMv5TE. I will reopen if necessary.