Add reader database cluster support

[shangyian]

Summary

This PR adds support for using a reader database cluster (for the metadata database) for read-only operations, along with the default writer database cluster for all other operations. It improves scalability by offloading read traffic from the writer database, and it sets up support for read/write-aware session management across both REST and GraphQL interfaces. The reader cluster configs are optional, so users who don't have a read replica setup can just set the writer cluster.

• For REST, we automatically detect when there is a GET request and route to the reader session.
• For GraphQL, we automatically detect when a request is not a mutation and route to the reader session (at the moment this is all GraphQL requests since we don't support mutations).

Settings

The settings for configuring a metadata database are now done via the nested writer_db and reader_db fields, each with a DatabaseConfig model to support detailed connection settings like pool_size, max_overflow, connect_timeout, etc.

.env files should be updated to include READER_DB__URI and other nested settings if reader DB is used:

# Writer DB (required)
WRITER_DB__URI=postgresql+psycopg://dj:dj@postgres_metadata:5432/dj
WRITER_DB__POOL_SIZE=20
WRITER_DB__MAX_OVERFLOW=20
WRITER_DB__POOL_TIMEOUT=10
WRITER_DB__CONNECT_TIMEOUT=5
WRITER_DB__POOL_PRE_PING=true
WRITER_DB__ECHO=false
WRITER_DB__KEEPALIVES=1
WRITER_DB__KEEPALIVES_IDLE=30
WRITER_DB__KEEPALIVES_INTERVAL=10
WRITER_DB__KEEPALIVES_COUNT=5

# Reader DB (optional)
READER_DB__URI=postgresql+psycopg://dj:dj@postgres_metadata:5432/dj
READER_DB__POOL_SIZE=10
READER_DB__MAX_OVERFLOW=10
READER_DB__POOL_TIMEOUT=5
READER_DB__CONNECT_TIMEOUT=5
READER_DB__POOL_PRE_PING=true
READER_DB__ECHO=false
READER_DB__KEEPALIVES=1
READER_DB__KEEPALIVES_IDLE=30
READER_DB__KEEPALIVES_INTERVAL=10
READER_DB__KEEPALIVES_COUNT=5

Test Plan

Locally. Modified unit tests.

• PR has an associated issue: #
• make check passes
• make test shows 100% unit test coverage

Deployment Plan

N/A

[netlify]

✅ Deploy Preview for thriving-cassata-78ae72 canceled.

Name	Link
🔨 Latest commit	29a4326
🔍 Latest deploy log	https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/6856622cf61b6c0008b3c236

[agorajek]

This is great. I think it will make our life better.

[agorajek]

Which of the above lines could trigger a GraphQLError ?

[shangyian]

If the user inputs an invalid GraphQL query, then calling parse(query) could result in a GraphQLError.

[agorajek]

very cool

[agorajek]

I think we may still want to do -1 here for better sleep, no?

[shangyian]

I changed this default to 100