hashmap debugging is not thread-safe

[teg]

I hit an assert from time to time when using nss-resolve (from firefox/chrome) with debug hashmap enabled.

Relevant part of the backtrace:
#2 0x00007f24938deecb in log_assert_failed (text=0x7f2493986003 "*_head == _item", file=0x7f2493985e77 "src/basic/hashmap.c", line=864, func=0x7f2493986450 <PRETTY_FUNCTION.8891> "hashmap_free_no_clear") at src/basic/log.c:773
#3 0x00007f24938d5b9b in hashmap_free_no_clear (h=0x387d7b7da000) at src/basic/hashmap.c:864
#4 0x00007f24938d5c75 in internal_hashmap_free_free (h=0x387d7b7da000) at src/basic/hashmap.c:892
#5 0x00007f2493912a0b in ordered_hashmap_free_free (h=0x387d7b7da000) at ./src/basic/hashmap.h:161
#6 0x00007f24939130cc in bus_free (b=0x387d7a78d000) at src/libsystemd/sd-bus/sd-bus.c:153
#7 0x00007f24939176b9 in sd_bus_unref (bus=0x387d7a78d000) at src/libsystemd/sd-bus/sd-bus.c:1474
#8 0x00007f24938946eb in message_free (m=0x387d7b052c00) at src/libsystemd/sd-bus/bus-message.c:137
#9 0x00007f2493896dc1 in sd_bus_message_unref (m=0x387d7b052c00) at src/libsystemd/sd-bus/bus-message.c:942
#10 0x00007f2493844f8a in sd_bus_message_unrefp (p=0x7f24a3c1a008) at ./src/shared/bus-util.h:148
#11 0x00007f249384653b in _nss_resolve_gethostbyname3_r (name=0x387d7b4718e8 "fsnyuzirjbelagg", af=2, result=0x7f24a3c1a6c0, buffer=0x7f24a3c1a1d0 "", buflen=912, errnop=0x7f24a3c1b660, h_errnop=0x7f24a3c1a6a4, ttlp=0x0, canonp=0x0) at src/nss-resolve/nss-resolve.c:305
#12 0x00007f2493846f10 in _nss_resolve_gethostbyname2_r (name=0x387d7b4718e8 "fsnyuzirjbelagg", af=2, host=0x7f24a3c1a6c0, buffer=0x7f24a3c1a1d0 "", buflen=912, errnop=0x7f24a3c1b660, h_errnop=0x7f24a3c1a6a4) at src/nss-resolve/nss-resolve.c:706

It seems we are trying to remove an entry from a list which is not linked into the list:

(gdb) frame 3
#3 0x00007f24938d5b9b in hashmap_free_no_clear (h=0x387d7b7da000) at src/basic/hashmap.c:864

864 LIST_REMOVE(debug_list, hashmap_debug_list, &h->debug);
(gdb) print h->debug
$10 = {debug_list_next = 0x0, debug_list_prev = 0x0, max_entries = 1, line = 1877, file = 0x7f249398fa08 "src/libsystemd/sd-bus/sd-bus.c",
func = 0x7f2493990d10 <func.13485> "sd_bus_call_async", put_count = 1, rem_count = 1, last_rem_idx = 0}

@michich, @poettering any ideas?

[poettering]

This might be a refocunting problem in nss-resolve rather than an issue in hashmap. We had reports about that before, but I am not sure how to reproduce this

[poettering]

Ah, my guess is that firefox is simply multitrheaded, but the debug hashmap list keeping is not. Not sure if it is worth fixing that though...

[teg]

Ah, good point. This is probably worth fixing so that developers can always run with hashmap debugging enabled... Should be straight-forward.

[teg]

I can confirm that adding locking to the hashmap debugging appears to solve the problem.

[dvdhrm]

Fixed.