Password authentication failed on RDS with scram-sha-256

[mintyhippoxyz]

PgBouncer 1.19.0 on Alpine Linux 3.18
RDS PostgreSQL 15.4 w/ TLS & scram-sha-256 (default parameter/options groups)

[databases]
test = host=<rds host> dbname=test

[pgbouncer]
logfile = /var/log/pgbouncer/pgbouncer.log
listen_addr = 127.0.0.1
listen_port = 6432
unix_socket_dir = /run/postgresql
unix_socket_mode = 0777
unix_socket_group = postgres
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt
stats_users = postgres
pool_mode = session
server_reset_query = DISCARD ALL
max_client_conn = 100
default_pool_size = 50
ignore_startup_parameters = options
server_tls_sslmode = require

userlist.txt contains the scram hash for "test" user. I've verified the way I'm generating them works in a non-aws environment. I've also noticed if the password hash in userlist.txt is incorrect or the wrong password is provided I get FATAL: SASL authentication failed instead so I think that further proves there isn't an issue with my scram hash.

RDS logs

FATAL: password authentication failed for user "test"
DETAIL: Connection matched pg_hba.conf line 13: "hostssl all all all md5"

The pg_hba matched line sent me down a bit of a rabbit hole trying to figure out why amazon doesn't add a hostssl all all all scram-sha-256 to pg_hba or if I could (not possible afaik) until I found this:

To ease transition from the md5 method to the newer SCRAM method, if md5 is specified as a method in pg_hba.conf but the user's password on the server is encrypted for SCRAM (see below), then SCRAM-based authentication will automatically be chosen instead.

Noted here: https://www.postgresql.org/docs/current/auth-password.html so I don't think that's the actual problem.

If nobody has any ideas/suggestions I may just have to reluctantly switch back to md5 as I'm at a loss at this point.

[mintyhippoxyz]

If I add the user (user=test) to the database line I get a FATAL: server login failed: wrong password type and cannot do SCRAM authentication: password is SCRAM secret but client authentication did not provide SCRAM keys in the pgbouncer log.

Both issues I found referencing that appear to be using auth_query whereas I am not. #774 #761

[JelteF]

PgBouncer cannot log in to Postgres, because you have the scram hash in userlist.txt. This is how scram works by design, the hash itself is not enough to authenticate. You'd need to store the plain text passwords in the userlist.txt file to make it work.

[JelteF]

To correct what I wrote here before: You can use scram hashes to authenticate, but they need be exactly the same as the one stored in the postgres server, not just ones created for the same password. See #1083 for improved documentation on that and a trick on how to get it.

Basically this issue seems to be a duplicate of #787