v0.7.1 - Local Dev Alignment & IO Manager Refactor

Summary

This release marks a significant milestone in aligning the local development environment with production logic, refactoring IO Managers for better consistency, and hardening the codebase for Kubernetes deployments.

Key Changes

1. Local Development & IO Manager Alignment (v0.7.0 - v0.7.1)

• Centralized Database Configuration: Introduced DatabaseConfig resource as the single source of truth for all IO Managers (S3, SQL, DuckDB, Filesystem).
• DuckDB Concurrency Fix: Implemented exponential backoff retries in DuckDBIOManager to handle parallel write locks during local development.
• Filesystem IO Manager Parity: Updated FilesystemIOManager to support "standalone" incremental loading, matching the S3 production logic.
• Lazy Loading: Made duckdb a lazy import to prevent dependency failures in production environments (K8s) where it isn't installed.

2. Infrastructure & CI/CD Enhancements (v0.6.0 - v0.6.3)

• Resource Optimization: Increased memory limits and requests for Dagster pods to prevent OOM errors during heavy transformations.
• CI/CD Improvements: Added branch and SHA tags to Docker images for better traceability.
• K8s Stability: Resolved PostgreSQL authentication issues during schedule evaluation in Kubernetes.

3. Data Modeling & Integrity (v0.5.1 - v0.5.4)

• Schema Hardening: Updated Ticker models with BigInteger for timestamps and improved metadata-driven table configuration.
• Type Safety: Resolved over 150+ linting and type-checking issues (Ruff, Mypy, Pyright).
• Documentation: Added comprehensive guides for DNS configuration and Git workflows.

Technical Debt & Fixes

• Standardized on v*.*.* tagging for automated Docker builds.
• Improved error handling in S3 loading logic and data merging.
• Refactored asset factories for better modularity.

Full Changelog: v0.5.0...v0.7.1

v0.5.0 - Kubernetes Deployment & CI/CD

🚀 Release v0.5.0 - Kubernetes Deployment & CI/CD

This release adds production-ready Kubernetes deployment manifests and a comprehensive CI/CD pipeline.

✨ New Features

Kubernetes Deployment

• 🎯 Helm chart override pattern (follows dagster-deployment conventions)
• 🔐 Sealed secrets support for credential management
• 📦 ConfigMap with environment variables matching Settings class
• 📚 Comprehensive deployment documentation

CI/CD Pipeline

• ✅ Sequential workflow: lint → test → build-image
• 🔍 Ruff linter and formatter checks
• 🔬 Mypy type checking
• 🧪 Pytest with coverage reporting
• 🐳 Docker image build and push to GHCR (on version tags)
• 🏷️ Smart tagging: semver + latest

Docker Image

• 🐍 Python 3.14 multi-stage build
• 🔒 Non-root user (UID 1000)
• 💊 Health checks included
• ⚡ Optimized for production

📦 Container Images

# Pull the image
podman pull ghcr.io/arookieds/dagster-code-location-crypto-data:0.5.0
podman pull ghcr.io/arookieds/dagster-code-location-crypto-data:latest

Available tags:

• 0.5.0 - This release
• 0.5 - Minor version
• 0 - Major version
• latest - Latest stable release

🚀 Quick Start

1. Generate sealed secrets:

cd manifests
cp .env.example .env
# Edit .env with your credentials
sops exec-env .env "nu create_sealed_secrets.nu"

2. Deploy to Kubernetes:

kubectl apply -k manifests/

3. Register with main Dagster instance:
Update workspace configuration:

dagster-webserver:
  workspace:
    servers:
      - host: "crypto-data-crypto-data"
        port: 3030
        name: "crypto-data"

📚 Documentation

• Deployment Guide
• CI/CD Workflow
• Development Guide

🔐 Security

• Non-root container execution
• Sealed secrets for credentials
• No hardcoded secrets
• Security context with dropped capabilities
• Pre-commit hooks for secret detection

📊 Stats

• 11 files changed
• 841 insertions
• 110 tests passing
• 82% code coverage

🙏 Contributors

@arookieds

---

Full Changelog: v0.4.0...v0.5.0