Infinite loop while allocating more memory than available via mmap (testing erlang-19.0)

[neeraj9]

While trying to run Erlang v19.0 I ran into a very strange issue of 100% CPU usage, which ultimately goes down to an infinite loop in the following function, where the math just doesn't let the loop break itself.
void * rump_hypermalloc(size_t howmuch, int alignment, bool waitok, const char *wmsg)

Note: Memory for VM is configured as 256 MB.

  rumprun "$virt" \
     -D 1234 \
     -p \
     -I if,vioif,'-net tap,script=no,ifname=tap0' \
     -W if,inet,static,"$ip/24","$gw" \
     -e ERL_INETRC="$erlpath/erl_inetrc" \
     -b images/erlang.iso,"$erlpath" \
     -b examples/app.iso,"$erlapp_path" \
     -M 256 \
     -i beam.hw.bin \
       "-- $epmd_opt -root $erlpath/lib/erlang \
        -progname erl +Mea min -- \
        -home $erlhome -noshell -pa $erlapp_path \
        $epmd_conf -s $module start"

The code snippet for the relevant portion (from rump_hypermalloc) is as follows for reference.

 limitagain:
        if (thelimit != RUMPMEM_UNLIMITED) {
                newmem = atomic_add_long_nv(&curphysmem, howmuch);
                if (newmem > thelimit) {
                        newmem = atomic_add_long_nv(&curphysmem, -howmuch);
                        if (!waitok) {
                                return NULL;
                        }
                        uvm_wait(wmsg);
                        goto limitagain;
                }
        }

As you will notice below that the math for curphysmem ping/pongs between 1076465664 and 2723840.

Breakpoint 4, rump_hypermalloc (howmuch=howmuch@entry=1073741824,
    alignment=alignment@entry=4096, waitok=waitok@entry=true,
    wmsg=wmsg@entry=0x6e8a48 "mmapmem")
    at /opt/rumprun/src-netbsd/sys/rump/librump/rumpkern/vm.c:1216
1216      newmem = atomic_add_long_nv(&curphysmem, howmuch);
(gdb) n
1217      if (newmem > thelimit) {
(gdb) p newmem
$29 = 1076465664
(gdb) p curphysmem
$30 = 1076465664
(gdb) p howmuch
$31 = 1073741824
(gdb) c
Continuing.

Breakpoint 4, rump_hypermalloc (howmuch=howmuch@entry=1073741824,
    alignment=alignment@entry=4096, waitok=waitok@entry=true,
    wmsg=wmsg@entry=0x6e8a48 "mmapmem")
    at /opt/rumprun/src-netbsd/sys/rump/librump/rumpkern/vm.c:1216
1216      newmem = atomic_add_long_nv(&curphysmem, howmuch);
(gdb) c
Continuing.
Hardware watchpoint 8: curphysmem

Old value = 2723840
New value = 1076465664
0x00000000005d13d8 in rumpns_atomic_add_ptr_nv ()
(gdb) c
Continuing.
Hardware watchpoint 8: curphysmem

Old value = 1076465664
New value = 2723840
0x00000000005d13d8 in rumpns_atomic_add_ptr_nv ()
(gdb) c
Continuing.

Breakpoint 4, rump_hypermalloc (howmuch=howmuch@entry=1073741824,
    alignment=alignment@entry=4096, waitok=waitok@entry=true,
    wmsg=wmsg@entry=0x6e8a48 "mmapmem")
    at /opt/rumprun/src-netbsd/sys/rump/librump/rumpkern/vm.c:1216
1216      newmem = atomic_add_long_nv(&curphysmem, howmuch);
(gdb) c
Continuing.
Hardware watchpoint 8: curphysmem

Old value = 2723840
New value = 1076465664
0x00000000005d13d8 in rumpns_atomic_add_ptr_nv ()
(gdb) c
Continuing.
Hardware watchpoint 8: curphysmem

Old value = 1076465664
New value = 2723840
0x00000000005d13d8 in rumpns_atomic_add_ptr_nv ()
(gdb) n
Single stepping until exit from function rumpns_atomic_add_ptr_nv,
which has no line number information.
rump_hypermalloc (howmuch=howmuch@entry=1073741824,
    alignment=alignment@entry=4096, waitok=waitok@entry=true,
    wmsg=wmsg@entry=0x6e8a48 "mmapmem")
    at /opt/rumprun/src-netbsd/sys/rump/librump/rumpkern/vm.c:1219
1219        if (!waitok) {
(gdb) p thelimit
$37 = 129478656
(gdb)

[anttikantee]

cross-referencing this with the mailing list discussion:
https://www.freelists.org/post/rumpkernel-users/Can-I-trace-rumprun-unikernel,6