arm32v6 always (and arm32v7 sometimes(?)) fails unit test "low TTL item should be gone" / t/restart.t

[grooverdan]

Describe the bug

Unit test failures on https://doi-janky.infosiftr.net/job/multiarch/job/arm32v6/job/memcached/ are constant for the emcached:1.6.9-alpine build preventing the creation of the docker image.

Arm32v7 can fail the same way
https://doi-janky.infosiftr.net/job/multiarch/job/arm32v7/job/memcached/432/execution/node/78/log/

./testapp
....

�[91m# Load a couple chunked items
�[0mt/metaget.t ................. ok
�[91m# Data that should expire while stopped.
�[0m�[91m# killed, waiting
�[0m�[91mGracefully stopping
�[0m�[91m# reconnected
�[0mt/lru-maintainer.t .......... ok
�[91m# low TTL item should be gone

#   Failed test 'low2 == 'mo''
#   at /usr/src/memcached/t/lib/MemcachedTest.pm line 97.
#          got: 'END
# '
#     expected: 'VALUE low2 0 2
# mo
# END
# '
�[0m�[91m# Looks like you failed 1 test of 93.
�[0mt/restart.t ................. 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/93 subtests 
t/issue_68.t ................ ok
�[91m# Some chunked item tests
�[0mt/chunked-extstore.t ........ ok
t/binary.t .................. ok
t/getset.t .................. ok
t/chunked-items.t ........... ok
t/lru-crawler.t ............. ok
t/binary-extstore.t ......... ok

Test Summary Report
-------------------
t/restart.t               (Wstat: 256 Tests: 93 Failed: 1)
  Failed test:  54
  Non-zero exit status: 1
Files=80, Tests=175266, 91 wallclock secs (36.82 usr  1.90 sys + 95.62 cusr 28.70 csys = 163.04 CPU)
Result: FAIL

To Reproduce

Per build logs linked.

System Information

• OS/Distro: Linux/alpine
• Version of OS/distro: 3.14
• Version of memcached: 1.6.9
• Hardware detail: arm32v6, arm32v7

Detail (please include!)

I hope you see something in the logs in the links. I don't have the hardware to test or extract more detail from this test failure unfortunately.

[dormando]

Hmmmm... I'll take a look, but this could be difficult without the hardware. Hopefully it's an obvious race. I'll throw up a help wanted tag in case anyone else wants to look.

[tianon]

The good news is that I can reproduce on QEMU's user-mode emulation, so it's not something specific to the hardware those builds are happening on.

(I'm using https://packages.debian.org/bullseye/qemu-user-static + docker build --platform linux/arm/v6 alpine to test, although the same would work on Docker for Mac or Docker for Windows.)

@dormando you've got an RPi1, right? That's v6, and I wonder whether you've seen this there (I'm guessing you're on Raspbian?) or if it's specific to musl.

[tianon]

Ok, this must be related to arm32 + musl (or something else Alpine-specific like a quirk in a dependent package) because I updated debian/Dockerfile to be FROM https://hub.docker.com/r/tianon/raspbian instead of debian and it passes just fine.

[dormando]

My RPi1 is pretty dead. Only functional ones I have are a 3, some 4's, and nano's I guess.

Haven't had a chance to look at this yet, sorry.

[yosifkit]

Just a quick addition of some additional failures on Memcached 1.6.22 on both Debian and Alpine based Docker images on arm32v7. We aren't currently building memcached Docker images for arm32v6, so I don't know if it also exhibits there.

Use of uninitialized value $res in print at t/watcher_connid.t line 42, <GEN2> line 1.

#   Failed test 'Saw a miss with the connection id 27'
#   at t/watcher_connid.t line 43.
#                   undef
#     doesn't match '(?^:ts=\d+\.\d+\ gid=\d+ type=item_get key=foo status=not_found clsid=\d+ cfd=27)'
Use of uninitialized value $res in print at t/watcher_connid.t line 46, <GEN2> line 1.

#   Failed test 'Saw a set with the connection id 27'
#   at t/watcher_connid.t line 47.
#                   undef
#     doesn't match '(?^:ts=\d+\.\d+\ gid=\d+ type=item_store key=foo status=stored cmd=set ttl=\d+ clsid=\d+ cfd=27)'
Can't use an undefined value as a symbol reference at t/watcher_connid.t line 52, <GEN2> line 1.
# Looks like your test exited with 111 just after 3.
t/watcher_connid.t .......... 
Dubious, test returned 111 (wstat 28416, 0x6f00)
Failed 3/4 subtests