Many lib-runtime-events tests fail on armel

[glondu]

Many tests of OCaml 5.2.0 fail on Debian sid, armel.

Quoting the build log:

[...]
List of failed tests:
    tests/lib-runtime-events/'test.ml' with default (bytecode) Process 4069629 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_caml.ml' with default (bytecode) Process 4069634 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_caml_counters.ml' with default (bytecode) Process 4069639 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_caml_exception.ml' with default (bytecode) Process 4069644 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_caml_parallel.ml' with default (bytecode) Process 4069649 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_caml_reentry.ml' with default (bytecode) Process 4069661 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_caml_runparams.ml' with default (bytecode) Process 4069666 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_caml_slot_reuse.ml' with default (bytecode) Process 4069671 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_caml_stubs_gc.ml' with default (bytecode) Process 4070193 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_compact.ml' with default (bytecode) Process 4070198 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_dropped_events.ml' with line 9 (bytecode) Process 4070203 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_env_start.ml' with default (bytecode) Process 4070211 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_fork.ml' with line 6 (bytecode) Process 4070228 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_user_event.ml' with default (bytecode) Process 4070238 got signal 11(Segmentation fault), no core dumped
    tests/lib-runtime-events/'test_user_event_unknown.ml' with line 7 (bytecode) Process 4070251 got signal 11(Segmentation fault), no core dumped

Summary:
  2480 tests passed
   659 tests skipped
    15 tests failed
   602 tests not started (parent test skipped or failed)
     0 unexpected errors
  3756 tests considered
#### Something failed. Exiting with error status.

It can be reproduced with qemu-user-static and an armel chroot.

I can't reproduce this on armel on a non-Linux platform, so this is likely something specific to Linux (or Debian, or the specific gcc+glibc version you're using).

Is there a place I can download a Debian/armel image suitable for use with qemu in order to investigate further?

[glondu]

I don't have a Debian/armel image at hand, but from Debian/amd64 you can do, after having installed pbuilder, qemu-user-static (and maybe other packages):

sudo pbuilder create --basetgz /var/cache/pbuilder/sid-armel.tgz --distribution sid --debootstrapopts --variant=minbase --architecture armel
sudo pbuilder login --basetgz /var/cache/pbuilder/sid-armel.tgz

I don't have a Debian/armel image at hand, but from Debian/amd64 you can do, after having installed pbuilder, qemu-user-static (and maybe other packages):

sudo pbuilder create --basetgz /var/cache/pbuilder/sid-armel.tgz --distribution sid --debootstrapopts --variant=minbase --architecture armel
sudo pbuilder login --basetgz /var/cache/pbuilder/sid-armel.tgz

Unfortunately, I've not been able to make this work (so far) from an ubuntu system (using --mirror http://www-ftp.lip6.fr/pub/linux/distributions/debian/ as extra argument), pbuild complains about being unable to make the /{bin,lib,sbin} symlinks to /usr/{bin,lib,sbin}. Probably some ubuntu-specific change to pbuilder getting in the way. I'll try to get access to a debian setup soonish and retry.

[lthls]

I can reproduce the problem with the setup above, on trunk.
However gdb doesn't work in this setup so I'm a bit stuck.
I did manage to find out that for testsuite/tests/lib-runtime-events/test.ml it segfaults somewhere inside get_event_counts, but that's about as far as I could get.

[lthls]

With some additional printf debugging I managed to pinpoint to this line:

ocaml/otherlibs/runtime_events/runtime_events_consumer.c

Line 395 in 1d7d915

ring_head = atomic_load_acquire(&runtime_events_buffer_header->ring_head);

[glondu]

An alignment issue?

[glondu]

According to the NetBSD manpage, atomic_load_acquire on 64-bit values might not work on 32-bit architectures when __HAVE_ATOMIC64_LOADSTORE is not defined, which is the case here.

[glondu]

My observations:

• in the given context (line 395 of runtime_events_consumer), the pointer given to atomic_load_acquire is 64-bit-aligned (always 0x424da040 in my setup)
• the segfault happens in atomic_load_acquire
• the segfault stays if I remove the call to atomic_load_acquire, i.e. if I use directly runtime_events_buffer_header->ring_head (looking at the assembly, an atomic load is still done)
• if I copy sizeof(struct runtime_events_buffer_header) bytes from runtime_events_buffer_header to a freshly malloc-ed foo (always 0x467e50 in my setup), and use foo->ring_head, there is no segfault

It looks like atomic loads are impossible in the memory nearby 0x424da040, but possible nearby 0x467e50. Is that even possible?

[lthls]

A wild guess: is it possible that 0x424da040 comes from mmaped memory, and the processor can't perform atomic accesses on such memory ?

[glondu]

Right! I could come up with the following reduced repro case:

$ dd if=/dev/zero of=test.bin bs=1 count=64
$ cat test.c
#include <stdio.h>
#include <stdint.h>
#include <stdatomic.h>
#include <unistd.h>
#include <sys/mman.h>
#include <fcntl.h>

#define atomic_load_acquire(p)                    \
  atomic_load_explicit((p), memory_order_acquire)

int main() {
  int fd = open("test.bin", O_RDONLY, 0);
  uint64_t *a = mmap(NULL, 64, PROT_READ, MAP_SHARED, fd, 0);
  uint64_t b = atomic_load_acquire(a);
  printf("%lld\n", b);
  return 0;
}
$ gcc test.c -latomic
$ ./a.out
Segmentation fault (core dumped)

[gasche]

Thanks for the detective chase. I think that we have two choices:

1. try to adapt runtime-events to have a 32bit version manipulating 32bit words rather than 64bit words (so: give up on the current same-data-on-all-archs design), to get it working for armel
2. give up on runtime-events on armel, have a configure check that forbids this combination, and correspondingly disable the testsuite

(2) is of course vastly less work and it is not clear that anyone would care to use runtime-events on armel enough to justify (1). This sounds rather clear-cut?

[glondu]

FYI, all tests pass after replacing atomic_load_acquire by memcpy (3 occurrences). But this is probably not what we want...

give up on runtime-events on armel, have a configure check that forbids this combination, and correspondingly disable the testsuite

This seems reasonable.

[shindere]

2. give up on runtime-events on armel, have a configure check that forbids this combination, and correspondingly disable the testsuite

Am I understanding correctly that runtime-events should then actually be disabled on any 32bitsplatform, not jsut armel?

[lthls]

Am I understanding correctly that runtime-events should then actually be disabled on any 32bitsplatform, not jsut armel?

It's likely that the current issue is specific to armel, so I think we should keep support for the other 32-bit platforms unless we get further bug reports.

[gasche]

I wonder if other parts of the runtime suffer from the same issue. We use mmap liberally in the runtime (it is used for the minor heap, for small-size pools in the major heap, and for the call stacks in some configurations), but our atomic accesses are typically word-sized. The only other joint use of int64_t and atomics I found is in fiber.c, the static _Atomic int64_t fiber_id = 0; global declaration (operated with fetch_add), I suppose this is safe?