etcd-backend emits warning logs on IBM

[hugoShaka]

Expected behaviour:

Teleport does not spam logs when using an etcd backend with user/password setups.

Note: relying on user:password should be avoided as much as possible but IBM's etcd only offers user/pass auth.

Current behaviour:

{"level":"warn","ts":"2024-06-05T18:42:50.403476Z","logger":"etcd-client","caller":"v3@v3.5.13/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004f0000/7791fe81-7c94-4f98-a383-36ca417b103d.c0v4phir0ah9ul9trho0.databases.appdomain.cloud:31743","attempt":0,"error":"rpc error: code = Unauthenticated desc = etcdserver: invalid auth token"}

Note: I don't know where this logger comes from, but it's not configured properly and doesn't respect the Teleport log settings.

Bug details:

This is an etcd bug that was fixed, but the code was deemed too fragile and the fix reverted. A proper fix is not implemented yet.

The current workarounds are:

• Not using user:pass. This is impossible in IBM's cloud but is the recommended setup for everyone else who has access to a decent etcd setup.
• Re-create a new etcd client before each watch if etcd is configured with a user/password.
• (ask IBM to fix their etcd hosting and support mTLS? 🤷 )
• don't use IBM's etcd.

Parting from IBM's etcd is possible if we switch our backend to Postgres. This will improve the Teleport UX on IBM cloud. However their Postgres offering also has shortcomings and we must play with change_feed_conn_string to make the auth watchers work. This also means existing users will have to migrate the backend or live with the errors in logs (likely already the case).