#4 Primary/replica selection

[levkk]

Possible implementation for #4

Features

1. Add custom syntax SET SERVER ROLE TO '(primary|replica)' to route queries explicitly to either the primary or any replica.
2. By default, the query is routed with round-robin, so all servers are treated as replicas (incl. the primary).
3. More config validation before startup.
4. Removed round-robin atomic, using local instead. Start a client on a random replica for load-balancing.

Bug fixes

1. If a connection fails a health check, make sure to close it and remove it from the pool so the next query doesn't hit it.
2. Less borrowing calls in ConnectionPool::get, slightly faster server selection.

TODOs

1. Add more errors returned to the client instead of just in the logs, which will help diagnosing problems.

[NikolayS]

SET SERVER ROLE TO 'primary';

Worth noting that Postgres and libpq already offer something in this area - the ability to specify multiple hosts in the connection string + target_session_attrs: https://www.postgresql.org/docs/devel/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS. These things are relatively new and are still not well adapted by libraries and frameworks, in my opinion.

Good article: https://paquier.xyz/postgresql-2/postgres-10-libpq-read-write/

About the improvements in PG14: https://www.postgresql.fastware.com/blog/enhancement-to-target-session-attrs

[levkk]

I think it's similar to this, but maybe not quite the same, because we want clients to be able to select their desired server within the same connection. In many languages, e.g. Ruby, Python, and in large apps, connections are expensive so if we can let the client do a read against a replica and a write against a primary in the same connection, it would be pretty useful.

This also paves the way to have almost automatic read-write query routing in the pooler instead of in the app.