error: to_string called on an uninitialized ip_address_t, addr_type: 0 compiling rethinkdb on Raspberry

[davide1995]

Just trying to compile and run rethinkdb on a Raspberry PI OS 64 bit (aarch64).
Compilation through:

./configure --allow-fetch
make

seems works correctly, thus no error is generated.
After that, when I move inside build/release and start the rethinkdb binary file following error occurs:

root@e521prod:/opt/rethinkdb/build/release# ./rethinkdb 
Recursively removing directory /opt/rethinkdb/build/release/rethinkdb_data/tmp
Initializing directory /opt/rethinkdb/build/release/rethinkdb_data
Running rethinkdb 2.4.3 (arm-linux-gnueabihf) (GCC 10.2.1)...
Running on Linux 6.1.41-v8+ aarch64
Loading data from directory /opt/rethinkdb/build/release/rethinkdb_data
Listening for intracluster connections on port 29015
Version: rethinkdb 2.4.3 (arm-linux-gnueabihf) (GCC 10.2.1)
error: Error in thread 4 in src/arch/address.cc at line 353:
error: to_string called on an uninitialized ip_address_t, addr_type: 0
error: Backtrace:
error: Tue Aug  1 13:34:05 2023
       
       1 [0x2c9ce4]: backtrace_t::backtrace_t() at vector.tcc:69 (discriminator 1)
       2 [0x2ca1f4]: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at backtrace.cc:309
       3 [0x2ca310]: format_backtrace[abi:cxx11](bool) at backtrace.cc:214
       4 [0x94199c]: report_fatal_error(char const*, int, char const*, ...) at errors.cc:87
       5 [0x3b6eb4]: ip_address_t::to_string[abi:cxx11]() const at address.cc:353
       6 [0x56b670]: service_address_ports_t::get_addresses_string[abi:cxx11](std::set<ip_address_t, std::less<ip_address_t>, std::allocator<ip_address_t> >) at char_traits.h:364
       7 [0x56d904]: do_serve(io_backender_t*, bool, base_path_t const&, metadata_file_t*, serve_info_t const&, os_signal_cond_t*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at stl_tree.h:991
       8 [0x5701b0]: serve(io_backender_t*, base_path_t const&, metadata_file_t*, serve_info_t const&, os_signal_cond_t*) at basic_string.h:187
       9 [0x58db58]: run_rethinkdb_serve(base_path_t const&, serve_info_t*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, file_direct_io_mode_t, int, optional<optional<unsigned long long> > const&, server_id_t const*, server_config_versioned_t const*, cluster_semilattice_metadata_t const*, directory_lock_t*, bool*) at command_line.cc:1284
       10 [0x58e940]: run_rethinkdb_porcelain(base_path_t const&, name_string_t const&, std::set<name_string_t, std::less<name_string_t>, std::allocator<name_string_t> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, file_direct_io_mode_t, int, optional<optional<unsigned long long> > const&, bool, serve_info_t*, directory_lock_t*, bool*) at stl_tree.h:991
       11 [0x58f030]: std::_Function_handler<void (), std::_Bind<void (*(base_path_t, name_string_t, std::set<name_string_t, std::less<name_string_t>, std::allocator<name_string_t> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, file_direct_io_mode_t, int, optional<optional<unsigned long long> >, bool, serve_info_t*, directory_lock_t*, bool*))(base_path_t const&, name_string_t const&, std::set<name_string_t, std::less<name_string_t>, std::allocator<name_string_t> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, file_direct_io_mode_t, int, optional<optional<unsigned long long> > const&, bool, serve_info_t*, directory_lock_t*, bool*)> >::_M_invoke(std::_Any_data const&) at std_function.h:293
       12 [0x3bff48]: starter_t::run_wrapper(std::function<void ()> const&) at runtime.cc:69
       13 [0x3bdb10]: coro_t::run() at coroutines.cc:310
error: Exiting.

[srh]

Previously seen in #3863 -- I guess it wasn't fixed as was claimed.

[srh]

My current theory is that it's a compilation or stdlib bug, or some other platform-specific glitch. I have only briefly looked over the code, but my understanding is:

1. do_serve calls get_addresses_string on serve_info.ports.local_addresses_{cluster|driver|http}.
2. In main_rethinkdb_serve, the serve_info_t is constructed with the variable address_ports.
3. The variable address_ports is set to get_service_address_ports(opts).
4. get_service_address_ports initializes those local_addresses_ fields with get_local_addresses(all_options(opts, "--bind-{cluster|driver|http}"), default_options, filter).
5. The return value result in get_local_addresses is iniitalized with get_local_ips(...).
6. get_local_ips constructs the ip_address_t with a sockaddr *, after checking sa_family is AF_INET or AF_INET6.
7. That ip_address_t constructor initializes addr_type with sanitize_address_family on sa_family which always returns RDB_IPV4_ADDR or RDB_IPV6_ADDR or invokes crash(...).
8. get_local_ips returns copies of that ip_address_t after filtering.

It is theoretically possible I have overlooked some place where an ip address could be passed uninitialized. It's also possible some other memory corruption bug is overwriting the memory with zero, and the problem isn't the logic around ip addresses.

Maybe if you pass --bind 127.0.0.1 or --bind all you will work around this problem.

[davide1995]

A little update. Not always the error is error: to_string called on an uninitialized ip_address_t, addr_type: 0. Sometimes is the following one:

root@e521prod:/opt/rethinkdb/build/release# ./rethinkdb
Running rethinkdb 2.4.3 (arm-linux-gnueabihf) (GCC 10.2.1)...
Running on Linux 6.1.41-v8+ aarch64
Loading data from directory /opt/rethinkdb/build/release/rethinkdb_data
Listening for intracluster connections on port 29015
Version: rethinkdb 2.4.3 (arm-linux-gnueabihf) (GCC 10.2.1)
error: Error in thread 4 in src/arch/runtime/thread_pool.cc at line 427:
error: Segmentation fault from reading the address 0x3fe00000.
error: Backtrace:
error: Wed Aug  2 01:32:37 2023
       
       1 [0x2c9ce4]: backtrace_t::backtrace_t() at vector.tcc:69 (discriminator 1)
       2 [0x2ca1f4]: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at backtrace.cc:309
       3 [0x2ca310]: format_backtrace[abi:cxx11](bool) at backtrace.cc:214
       4 [0x94199c]: report_fatal_error(char const*, int, char const*, ...) at errors.cc:87
       5 [0x3be250]: linux_thread_pool_t::fatal_signal_handler(int, siginfo_t*, void*) at thread_pool.cc:429 (discriminator 4)
       6 [0xf703b910]: __default_rt_sa_restorer+0 at 0xf703b910 (/lib/arm-linux-gnueabihf/libc.so.6)
       7 [0x2e11b8]: auto_drainer_t::lock_t::lock_t(auto_drainer_t::lock_t const&) at auto_drainer.cc:31
error: Exiting.

And passing --bind 127.0.0.1 or --bind all to the rethinkdb execution command, does not help.

Right now, I'm trying to compile it on an other Raspberry PI OS (aarch64), to check if there is also the same error.

[davide1995]

Okay, I tried with the other Raspberry and I can confirm that the same error occurs. Sometimes the error is

...
error: Error in thread 4 in src/arch/address.cc at line 353:
error: to_string called on an uninitialized ip_address_t, addr_type: 0
...

sometimes is the

...
error: Error in thread 4 in src/arch/runtime/thread_pool.cc at line 427:
error: Segmentation fault from reading the address 0x3fe00000.
...

GCC version on this Raspberry is also 10.2.1
Linux version on this Raspberry is also 6.1.41-v8

[davide1995]

Hi @srh
I really need some help on this issue. Do you have any updates?

[srh]

@davide1995 No, I haven't been thinking about this at all.