[YSQL] Committed transactions lost when the cluster config placement was invalid

[FranckPachot]

Jira Link: DB-5811

Description

If we create a cluster without a valid cluster level data placement config, which is the default with yugabyted start, we cannot create tables and get an error Not enough tablet servers in the requested placements

However, I can set the placement at tablespace level, like with all leaders in one region and followers in the other regions, and can create tables and run transactions.

In this case it seems that the transaction table is not replicated (because of the invalid cluster-level config) which means that if one region fails we cannot read data anymore (ERROR: Query error: GetTransactionStatus RPC) until it is back, and if it never comes back the committed changes are lost.

This is easy to reproduce following all steps of https://dev.to/yugabyte/simulate-network-latency-in-a-yugabytedb-cluster-on-a-docker-lab-264a except the yugabyted configure data_placement. In this case we will see that the COMMIT (END; in PgBench) doesn't wait on other nodes to acknowlege and taking down the first node (where the leaders are) will make the committed changes unreadable.

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[nchandrappa]

@FranckPachot For multi-zone/multi-region setup using yugabyted, its mandatory to run the configure data_placement command so that the right cluster level config gets applied. Yugabyted computes the data placement constraint based on the values provided in cloud_location.

Currently, in yugabyted, when a 3rd node joins the cluster, we automatically apply the data placement with the default placement constraint cloud1.datacenter1.rack1 and change the RF to 3. I believe we should update this behavior if the cloud_location value is provided during the start.

Proposed change

During the cluster creation, when a 3rd node joins the cluster, apply the data placement constraint based on the cloud_location value.

• This solves the issue when someone is trying an RF-3 on single-zone
• For multi-zone/multi-region it's mandatory to run the configure data_placement command

[gargsans-yb]

Landed the changes to master and Backported to 2.17.3

[FranckPachot]

@gargsans-yb I see the changes for yugabyted to avoid this misconfiguration but shouldn't we raise an error in DocDB when the transaction table cannot be created correctly?
A user may do the same misconfiguration manually and will loose data one day.