[Fix] Replace backoff with tenacity and enhance database connection handling

[JonnyTran]

Summary

This PR improves database connection reliability and error handling by replacing the library with and implementing enhanced retry mechanisms for database operations.

Changes

Dependency Updates

• Replaced with in pyproject.toml for more robust retry logic

Database Reliability Improvements

• New decorator: Replaced the custom decorator with a more robust retry policy using tenacity
• Enhanced error detection: Improved connection error filtering to only retry on actual connection issues, not SQL syntax errors
• Better logging: Added connection pool status logging for debugging connection issues
• Exponential backoff: Implemented proper exponential backoff with configurable wait times (0.1s to 2s max)

Database Health Checks

• Workspace doctor enhancement: Added comprehensive database connection health checks to the endpoint
• Autofix capability: Implemented automatic connection pool reset when database becomes unresponsive
• PostgreSQL monitoring: Added detailed connection pool statistics and stale transaction detection
• Safety timeout: Added 3-second timeout for health checks to prevent hanging

Code Simplification

• Unified retry logic: Consolidated retry decorators across the codebase (, , , )
• Cleaner error handling: Removed nested retry wrappers in favor of declarative decorators

Technical Details

Retry Policy Configuration

• Retry condition: Only retries on connection-related errors (OperationalError, DBAPIError, DisconnectionError, etc.)
• Wait strategy: Exponential backoff with 0.1s initial delay, max 2s delay
• Max attempts: 3 attempts total
• Logging: Warns on retry attempts and logs connection pool status

Database Health Check Features

• Liveness check: Simple query to verify connection
• PostgreSQL deep inspection: Monitors active connections and detects stale transactions
• Pool reset: Automatically disposes connection pool on timeout/connection errors
• Cross-database support: Handles PostgreSQL, SQLite, and other database types

Related Issues

• Fixes database connection reliability issues mentioned in Database connection error on initial login: connection is closed (asyncpg/sqlalchemy) #85
• Builds upon authentication improvements from [Fix] DB connection errors with JWT refresh token #150
• Addresses connection pool management and retry logic gaps

Testing

• All existing tests pass with the new retry mechanism
• Database health checks tested with various connection failure scenarios
• Retry logic validated with simulated network interruptions

Migration Notes

• No breaking changes to existing APIs
• Improved error messages and logging for better debugging
• Automatic pool reset may cause temporary connection recreation (expected behavior)