PostgreSQL connection pooling

[tmikula-dev]

Overview

This pull request introduces connection caching and reuse for both PostgreSQL readers and writers, improving efficiency and reliability by maintaining a single connection per instance. It also adds robust reconnection logic, updates tests to reflect the new behavior, and enhances configuration for dependency updates.

Release Notes

• PostgreSQL connection pooling for

Related

Closes #115

Summary by CodeRabbit

Release Notes

• Bug Fixes

  • Improved database resilience with automatic retry logic for transient connection failures.
  • Enhanced error handling and recovery for database operations.

• Performance Improvements

  • Database connections are now cached and reused across operations, reducing overhead.

• Tests

  • Added integration and unit tests for connection reuse and retry behavior.

• Chores

  • Updated dependency management configuration for improved update consolidation.

[coderabbitai]

Walkthrough

This PR implements connection caching for PostgreSQL readers and writers to reuse single connections across Lambda invocations instead of opening new TCP connections per operation. Both WriterPostgres and ReaderPostgres now cache connections with automatic reconnection on connection failures, supported by comprehensive unit and integration tests.

Changes

Cohort / File(s)	Summary
Documentation & Configuration .github/copilot-instructions.md, .github/dependabot.yml	Added PostgreSQL-specific caching instructions and configured Dependabot to group dependency updates by pattern.
PostgreSQL Reader Implementation src/readers/reader_postgres.py	Introduced _get_connection() method for cached connection reuse. read_stats() now retries up to 2 times on OperationalError, clears stale connections, and provides improved error messages with connection cleanup.
PostgreSQL Writer Implementation src/writers/writer_postgres.py	Added _get_connection() method for cached connection reuse. write() now retries up to 2 times on OperationalError, clears stale connections on failure, and maintains connection state across invocations.
Reader Unit Tests tests/unit/readers/test_reader_postgres.py	Extracted _STATS_DESCRIPTION constant, updated mock setup with closed attribute, and added TestConnectionReuse suite covering reconnection, retry, and error handling scenarios.
Writer Unit Tests tests/unit/writers/test_writer_postgres.py	Added new unit test suite verifying reconnection on closed connections, retry behavior on transient failures, exhaustion handling, and connection cleanup on persistent errors.
Integration Tests tests/integration/test_connection_reuse.py	New integration test module with fixtures and two test classes (TestWriterConnectionReuse, TestReaderConnectionReuse) asserting that cached connection objects remain identical across sequential operations.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

• OOP for project writers #104: Refactored src/writers/writer_postgres.py into a WriterPostgres class; this PR adds connection caching and retry logic to that same writer implementation.

Suggested labels

refactoring

Suggested reviewers

• petr-pokorny-absa
• lsulak
• oto-macenauer-absa

Poem

🐰 A rabbit springs from cache to cache,
No TCP connections to detach—
With retries wise and pooled delight,
The Lambda stays forever bright!
Connection reuse, oh what a sight! 🌟

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	❓ Inconclusive	The description covers overview, release notes (though incomplete), and related issue. However, the release notes are unfinished with incomplete bullet points.	Complete the release notes section; the first bullet point ends abruptly with 'pooling for' and needs completion describing the key user-facing improvement.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: PostgreSQL connection pooling/caching replaces per-call connections in WriterPostgres and ReaderPostgres.
Linked Issues check	✅ Passed	All primary acceptance criteria from issue #115 are met: connection reuse via cached connection with lazy initialization, reconnection on stale/broken detection via OperationalError/closed checks, unit test mocks preserved, and integration tests added.
Out of Scope Changes check	✅ Passed	All changes are in scope for issue #115 and PR objectives. Copilot instructions update and Dependabot grouping rules align with connection pooling implementation and maintenance goals.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/postgresql-connection-pooling

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (2)

tests/unit/readers/test_reader_postgres.py (1)

385-405: Unused variable pagination flagged by static analysis.

The variable pagination on line 402 is unpacked but never used. Consider using an underscore prefix to indicate it's intentionally unused.

🔧 Suggested fix

-            rows, pagination = reader.read_stats(limit=10)
+            rows, _pagination = reader.read_stats(limit=10)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/unit/readers/test_reader_postgres.py` around lines 385 - 405, The test
test_retries_on_operational_error unpacks reader.read_stats into rows,
pagination but never uses pagination; rename the unused variable to _pagination
(or simply use an underscore `_`) to satisfy static analysis and indicate
intentional non-use. Update the line that calls reader.read_stats in
test_retries_on_operational_error to unpack as rows, _pagination (or rows, _) so
the test behavior (asserting connect call count and rows) remains unchanged
while eliminating the unused-variable warning.

src/writers/writer_postgres.py (1)

60-60: Consider adding explicit connection cleanup for non-Lambda deployments.

The cached connection has no explicit close() mechanism. While this works well for Lambda (connections naturally close when the container terminates), long-running processes or integration tests may benefit from explicit cleanup. The Writer base class could be extended with an optional close() method.

This is not blocking for the current Lambda use case.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/writers/writer_postgres.py` at line 60, Add an explicit cleanup hook that
closes the cached DB connection: extend the Writer base class with an optional
close() method and implement it in the Postgres writer to call and null out
self._connection.close() (or await if async) when a connection exists; update
any connection-creating methods that set self._connection to ensure close() will
be safe to call, and add a brief unit test or integration cleanup call to
exercise Writer.close() in long-running tests or processes to avoid leaked
connections.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/writers/writer_postgres.py`:
- Line 60: Add an explicit cleanup hook that closes the cached DB connection:
extend the Writer base class with an optional close() method and implement it in
the Postgres writer to call and null out self._connection.close() (or await if
async) when a connection exists; update any connection-creating methods that set
self._connection to ensure close() will be safe to call, and add a brief unit
test or integration cleanup call to exercise Writer.close() in long-running
tests or processes to avoid leaked connections.

In `@tests/unit/readers/test_reader_postgres.py`:
- Around line 385-405: The test test_retries_on_operational_error unpacks
reader.read_stats into rows, pagination but never uses pagination; rename the
unused variable to _pagination (or simply use an underscore `_`) to satisfy
static analysis and indicate intentional non-use. Update the line that calls
reader.read_stats in test_retries_on_operational_error to unpack as rows,
_pagination (or rows, _) so the test behavior (asserting connect call count and
rows) remains unchanged while eliminating the unused-variable warning.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6c04b6fc-8b9c-4a86-a7cd-246694e57bde

📥 Commits

Reviewing files that changed from the base of the PR and between ccc1639 and 452008d.

📒 Files selected for processing (7)

• .github/copilot-instructions.md
• .github/dependabot.yml
• src/readers/reader_postgres.py
• src/writers/writer_postgres.py
• tests/integration/test_connection_reuse.py
• tests/unit/readers/test_reader_postgres.py
• tests/unit/writers/test_writer_postgres.py