[ysql] [k8s] Transactional writes in a cluster with DNS addressing and TLS certs fail

[iSignal]

In a (k8s) cluster that is using DNS addresses (--server_broadcast_addresses and --rpc_bind_addresses set to DNS names) and that has TLS enabled, SQL inserts are failing as seen below. However, create table, select queries, single inserts work.

Repro steps:

Bring up a k8s cluster with tls enabled

$ kubectl create namespace sanketh-tmp-2; helm3 install yb-demo  ~/code/charts/stable/yugabyte/ --set tls.enabled=true --set resource.master.requests.cpu=0.25,resource.master.requests.memory=0.5Gi,resource.tserver.requests.cpu=0.25,resource.tserver.requests.memory=0.5Gi --namespace sanketh-tmp-2 --wait
$ kubectl exec -it yb-tserver-0 -n sanketh-tmp-2 -- bash

Run ysqlsh

[root@yb-tserver-0 cores]# /home/yugabyte/bin/ysqlsh -h yb-tserver-0.yb-tservers.sanketh-tmp-2.svc.cluster.local
ysqlsh (11.2-YB-2.3.2.0-b0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.
yugabyte=# create table foo(id int primary key);
CREATE TABLE
yugabyte=# select * from foo;
 id
----
(0 rows)
root@yb-tserver-0 yugabyte]# bin/ysqlsh -h yb-tserver-1.yb-tservers.sanketh-tmp-2.svc.cluster.local
ysqlsh (11.2-YB-2.3.2.0-b0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.
yugabyte=# insert into foo (select * from generate_series(1,10));
ERROR:  Network error: Handshake failed: certificate verify failed, address: 10.32.2.90, hostname: 10.32.2.90
yugabyte=#

This error means that postgres is attempting to connect to a tserver by IP address instead of DNS. Only conns that use DNS addresses work in this cluster because TLS has been set up to use DNS addresess.

The root cause of this appears to be a conn to the local tserver from postgres in the write txn code path -

yugabyte-db/src/yb/yql/pggate/pg_txn_manager.cc

Line 193 in 88e9d59

LOG(INFO) << "Using TServer endpoint: " << (**tserver_shared_object_).endpoint();

which is set from

yugabyte-db/src/yb/tserver/tablet_server.cc

Line 257 in e6e1b52

shared_object_->SetEndpoint(bound_addresses.front());

to the actual bound IP address of the server instead of its rpc_bind/server_broadcast address.

@tsmull-11 @m-iancu @psudheer21

[d-uspenskiy]

New use_node_hostname_for_local_tserver gflag can be used to overcome the issue.