Core dump when decoding long list in binary_to_term/1

[mikpe]

Describe the bug
When calling binary_to_term/1 on a binary containing the encoding of a list of length 1 bsl 30 or longer, the VM segfaults. There are three problems in the code:

• the code stores a 32-bit unsigned length in a 32-bit signed integer, causing some valid lengths to appear negative and pointer (hp) increments by those values to yield incorrect results (this is similar to Cannot allocate 18446744071562067968 bytes on erlang:binary_to_term/1 #6393)
• the code multiplies the length by 2 before incrementing hp by it, causing the sign flip to occur for many more valid lengths
• the code performs this multiplication in 32-bit precision on 64-bit platforms, causing the MSB of the length to be lost

The fix requires two changes: to use an unsigned type for the length, and to perform the multiplication in a wider type.

To Reproduce

> cat bug.erl
-module(bug).
-export([test/1]).

test(N) ->
  length(binary_to_term(make(N))).

%% This is term_to_binary(lists:duplicate(N, [])) without
%% the excessive intermediate memory usage.
make(N) ->
  <<131, 108, N:4/big-unsigned-integer-unit:8, (binary:copy(<<106>>, N + 1))/binary>>.
> erl
Erlang/OTP 25 [erts-13.1.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Eshell V13.1.2  (abort with ^G)
1> c(bug).
{ok,bug}
2> bug:test(1 bsl 30).
Segmentation fault (core dumped)

Expected behavior
The call should (eventually) succeed and return 1073741824.

Affected versions
All current releases, i.e., 25.1.2, 24.3.4.6, and 23.3.4.18. I didn't check older releases.

Additional context
Once the bug is fixed the reproducer should not crash the VM, but it will require lots of memory. I've run it with 24GB RAM and 32GB swap, but the swapping was painful.