[DocDB] Can't start multiple tservers on the same machine

[eivanov89]

Jira Link: DB-6225

Description

Hi,

I try to start multiple tservers as suggested here. I use different "--rpc_bind_addresses" for each one. But I can start only the first tserver, the next one fails:

F20230416 21:27:53 ../../src/yb/tserver/tablet_server_main_impl.cc:223] Network error (yb/util/net/socket.cc:325): Error binding socket to [2a02:6b8:c34:14:0:1354:eb1f:2962]:9113: Cannot assign requested address (system error 99)
    @     0x563ffbc19a17  google::LogMessage::SendToLog()
    @     0x563ffbc1a95d  google::LogMessage::Flush()
    @     0x563ffbc1afd9  google::LogMessageFatal::~LogMessageFatal()
    @     0x563ffceab738  yb::tserver::TabletServerMain()
    @     0x563ffbbe8678  main
    @     0x7f7498ab5825  __libc_start_main
    @     0x563ffb93e02e  _start

For example if I have running

yb-tserver --rpc_bind_addresses=ydb-vla-dev04-002:9100 --tserver_master_addrs=ydb-vla-dev04-000:7100,ydb-vla-dev04-001:7100,ydb-vla-dev04-002:7100 --fs_data_dirs=/nvme0n1p2 --pgsql_proxy_bind_address ydb-vla-dev04-002:5433 --cql_proxy_bind_address ydb-vla-dev04-002:9042 --ysql_max_connections 4096 --db_block_cache_num_shard_bits=7 --default_memory_limit_to_ram_ratio 21

then start

/place/berkanavt/yugabyte/bin/yb-tserver --rpc_bind_addresses=ydb-vla-dev04-000:9113 --tserver_master_addrs=ydb-vla-dev04-000:7100,ydb-vla-dev04-001:7100,ydb-vla-dev04-002:7100 --fs_data_dirs=/nvme1n1p2 --pgsql_proxy_bind_address ydb-vla-dev04-000:5434 --cql_proxy_bind_address ydb-vla-dev04-000:9043 --ysql_max_connections 4096 --db_block_cache_num_shard_bits=7 --default_memory_limit_to_ram_ratio 21

I get that stacktrace. lsof shows that the port is free. Also if I start in parallel then the first process with arbitrary port succeeds and the rest fail, for example 9113 can succeed and 9100 will then fail instead.

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[ddorian]

@eivanov89

Can you also specify ports for --pgsql_proxy_webserver_port 13001, --cql_proxy_webserver_port 12001, --webserver_port 9001 --redis_proxy_webserver_port 11001 and try again?

Maybe it interfered with one of them but the error message was misleading:
https://docs.yugabyte.com/preview/reference/configuration/default-ports/

[eivanov89]

@ddorian

Now the port seems to be arbitrary:

F20230418 15:52:39 ../../src/yb/tserver/tablet_server_main_impl.cc:287] Network error (yb/util/net/socket.cc:325): Error binding socket to [2a02:6b8:c34:14:0:1354:eb1f:295a]:6379: Address already in use (system error 98)
    @     0x55caa9d54a17  google::LogMessage::SendToLog()
    @     0x55caa9d5595d  google::LogMessage::Flush()
    @     0x55caa9d55fd9  google::LogMessageFatal::~LogMessageFatal()
    @     0x55caaafe6a8c  yb::tserver::TabletServerMain()
    @     0x55caa9d23678  main
    @     0x7f8139386825  __libc_start_main
    @     0x55caa9a7902e  _start

The instance which was able to start:
/place/berkanavt/yugabyte/bin/yb-tserver --rpc_bind_addresses=ydb-vla-dev04-007:9100 --tserver_master_addrs=ydb-vla-dev04-005:7100,ydb-vla-dev04-006:7100,ydb-vla-dev04-007:7100 --pgsql_proxy_webserver_port 13001 --cql_proxy_webserver_port 12001 --webserver_port 9001 --redis_proxy_webserver_port 11001 --fs_data_dirs=/place/berkanavt/yugabyte/data/nvme0n1p2 --pgsql_proxy_bind_address ydb-vla-dev04-007:5433 --cql_proxy_bind_address ydb-vla-dev04-007:9042 --ysql_max_connections 4096 --db_block_cache_num_shard_bits=7 --default_memory_limit_to_ram_ratio 21

[jameshartig]

6379 is the Yedis (Redis) port. You might be able to use --start_redis_proxy=false