feat: add StorageAdapter for Phase 4 Dual-Write#5
Merged
nitaibezerra merged 1 commit intomainfrom Dec 25, 2025
Merged
Conversation
Implements StorageAdapter supporting multiple storage backends: - HUGGINGFACE: Write to HuggingFace only (legacy) - POSTGRES: Write to PostgreSQL only (target) - DUAL_WRITE: Write to both backends (migration phase) Features: - Unified interface compatible with DatasetManager - Lazy loading of backend managers - Automatic agency/theme resolution using PostgresManager cache - Environment variable configuration (STORAGE_BACKEND, STORAGE_READ_FROM) - Partial failure handling in dual-write mode Also includes comprehensive unit tests with 15 test cases.
5 tasks
mauriciomendonca
pushed a commit
that referenced
this pull request
Mar 20, 2026
Correcoes baseadas no code review: - #4 (ALTO): Remover BEGIN/COMMIT explicito do rollback SQL 005 que quebrava o commit atomico do runner (COMMIT prematuro antes do registro em migration_history) - #1 (MEDIO): Remover server-side cursor desnecessario em 006_migrate_unique_ids.py — fetchall() carrega tudo em memoria de qualquer forma, o name= so adiciona confusao semantica - #9 (MEDIO): Sanitizar target_version com regex ^[0-9]{3}$ no workflow e substituir eval $CMD por execucao direta com bash array para prevenir command injection - #5 (BAIXO): Extrair _execute_with_history() em migrate.py para eliminar ~60% de duplicacao entre execute_migration() e execute_rollback(), reduzindo risco de divergencia futura Todos os 44 testes continuam passando. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements StorageAdapter as part of Phase 4 (Dual-Write) of the HF → PostgreSQL migration.
Features
HUGGINGFACE: Write to HuggingFace only (legacy mode)POSTGRES: Write to PostgreSQL only (target mode)DUAL_WRITE: Write to both backends (migration phase)STORAGE_BACKEND,STORAGE_READ_FROM)Files
src/data_platform/managers/storage_adapter.py- Main adapter implementationsrc/data_platform/managers/__init__.py- Updated exportstests/unit/test_storage_adapter.py- 15 unit testsEnvironment Variables
STORAGE_BACKENDhuggingface,postgres,dual_writeSTORAGE_READ_FROMhuggingface,postgresTest plan
Related
Part of Phase 4 of the migration plan. Next steps: