This is the first stable release of the Bank Statement Export Service — a Python-based service for exporting large-scale transaction data into CSV files, storing them securely in Amazon S3, and managing job lifecycles with deduplication and auditability.
✨ Features
• REST API (Flask)
• POST /export → Initiate an export (or reuse an existing one if available).
• GET /export/{reference_id} → Check job status and retrieve presigned S3 URL if completed.
• Export Management
• Dedicated Export Service Database (Postgres/MySQL) for tracking jobs.
• Deduplication rules prevent re-running exports for the same date range (unless it’s today or force_refresh is enabled).
• Job states: PENDING, IN_PROGRESS, COMPLETED, FAILED, SUPERSEDED.
• Background Processing (Celery)
• Asynchronous workers for large exports.
• Streaming queries from the Transactions Database (memory-safe, handles 10M+ rows).
• CSV generation with headers, optional gzip compression.
• Multipart upload to Amazon S3.
• Security
• JWT authentication for all endpoints.
• Read-only connections to the Transactions Database.
• Secure presigned S3 URLs with configurable expiry (default 24h).
• Observability
• Structured logging with reference_id correlation.
• Prometheus /metrics endpoint for monitoring job lifecycle and performance.
🛠️ Tech Stack
• Python 3.11+
• Flask (REST API)
• Celery + Redis/RabbitMQ (task queue)
• SQLAlchemy + Alembic (ORM + migrations)
• Amazon S3 (file storage)
• Docker & Docker Compose (local setup)
• Postgres/MySQL (Export DB)
📊 Performance Goals
• Support for 10M+ row exports without memory exhaustion.
• Typical 5M row export completes in under 15 minutes.