Intermittent hangs with 10.16.1

[ofrobots]

Over at Google Cloud we are noticing intermittent hangs on some virtual machines with the latest 10.16.1. This is likely due to the OpenSSL upgrade due to 1.1.1c (see openssl/openssl#9078).

Hangs seem to coincide with when the system has low-entropy and has the following stack:

(gdb) bt
#0  0x00007fb4b2d7403f in __GI___select (nfds=18, readfds=0x7ffdd2bf5c30, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x00000000014d3c4c in rand_pool_acquire_entropy ()
#2  0x00000000014d2058 in rand_drbg_get_entropy ()
#3  0x00000000014cf5c4 in RAND_DRBG_instantiate ()
#4  0x00000000014cf8fe in drbg_setup ()
#5  0x00000000014cfbc7 in do_rand_drbg_init_ossl_ ()
#6  0x00007fb4b305d827 in __pthread_once_slow (once_control=0x2667838 <rand_drbg_init>, init_routine=0x14cfb80 <do_rand_drbg_init_ossl_>) at pthread_once.c:116
#7  0x00000000014febe9 in CRYPTO_THREAD_run_once ()
#8  0x00000000014ceb66 in drbg_status ()
#9  0x0000000000a104a0 in node::crypto::EntropySource(unsigned char*, unsigned long) ()
#10 0x00000000017aa753 in v8::base::RandomNumberGenerator::RandomNumberGenerator() ()
#11 0x00000000017a9901 in v8::base::CallOnceImpl(long*, std::function<void ()>) ()
#12 0x00000000017ac19f in v8::base::OS::GetRandomMmapAddr() ()
#13 0x0000000000af7947 in v8::internal::GetRandomMmapAddr() ()
#14 0x0000000000f037af in v8::internal::Heap::SetUp() ()
#15 0x0000000000fb6713 in v8::internal::Isolate::Init(v8::internal::StartupDeserializer*) ()
#16 0x00000000011ffd3a in v8::internal::Snapshot::Initialize(v8::internal::Isolate*) ()
#17 0x0000000000b14988 in v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) ()
#18 0x0000000000b149da in v8::Isolate::New(v8::Isolate::CreateParams const&) ()
#19 0x0000000000902954 in node::Start(int, char**) ()
#20 0x00007fb4b2c7eb97 in __libc_start_main (main=0x8bbd50 <main>, argc=2, argv=0x7ffdd2bf6788, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdd2bf6778) at ../csu/libc-start.c:310
#21 0x00000000008bbe85 in _start ()

[ofrobots]

Here's my experiment on some GCE virtual machines:

for i in $(seq 1 50) ; do echo run $i; cat /proc/sys/kernel/random/entropy_avail ; time npm >/dev/null; sleep 1; done
...
172

real	0m0.257s
user	0m0.222s
sys	0m0.032s
run 6
173

real	0m0.264s
user	0m0.216s
sys	0m0.044s
run 7
5
^C
real	33m50.490s
...

Note the entropy value of 5 at the time of the hang.

[addaleax]

I think this is openssl/openssl#9078 / openssl/openssl#9084, and cherry-picking the latter would resolve this?

[ofrobots]

I believe so. I will try this out in the failing environment tomorrow.

[targos]

I'm pretty sure I already saw an issue here about this a few days ago.

Edit: forget it, it was before 10.16.1

[MylesBorins]

Should we escalate a 10.16.2 to fix this? @ofrobots is there an easy way to test if I can get an rc together?

[MylesBorins]

/cc @nodejs/lts

[ofrobots]

#28983

[BethGriggs]

This should be fixed with the v10.16.2 release, please reopen if not.

[bernd-edlinger]

@ofrobots I am curious which linux version did you use where the available entropy
dropped to zero again?

[ofrobots]

This was a Ubuntu instance IIRC.

[bernd-edlinger]

I ask because the code with the code with the select should not be executed unless the
linux kernel is older than v3.17, that is rather old, ubuntu 14.04 uses linux v3.13 for instance,
but that is rather old, could you log in and try cat /proc/version, what does that look like?

[bernd-edlinger]

The other way round I observed a similar behaviour that you described, when
linux v4.8 or newer is used in an arm machine, and every 3-5 minutes when
the CRNG gets reseeded it pulls all entroy from the input pool.