[Silver Pipeline] - Parte 002 - Cria app ETL (Extract, Transform, Load)#650
Merged
pitangainnovare merged 26 commits intoMay 13, 2026
Merged
Conversation
pitangainnovare
force-pushed
the
scms-oca-etl-base
branch
from
d36aed0 to
80a0662
Compare
| oca_data: dict = field(default_factory=dict) | ||
|
|
||
| def __post_init__(self): | ||
| if not self.doc_id or not isinstance(self.doc_id, str): |
Contributor
There was a problem hiding this comment.
Pode ser um método de OcaModel e em BronzeDocument e SilverDocument faz um super()
| class DefaultStandardizer: | ||
| """Default source-agnostic transformation from bronze payload to silver.""" | ||
|
|
||
| def run(self, bronze_doc: BronzeDocument) -> SilverDocument: |
Contributor
There was a problem hiding this comment.
O dado sempre virá de um indice bronze? Se não, mudar o nome dá variável
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
O que esse PR faz?
Adiciona a estrutura base da app
etlpara geração e controle operacional de documentos Silver: modelo de estado, migration, contratos Bronze/Silver, mapping OpenSearch, orquestrador base, serviços de fila/processamento, command, tasks Celery e admin Wagtail mínimo.Onde a revisão poderia começar?
etl/models.py, depoisetl/pipeline/orchestrator.pyeetl/services.py.Como este poderia ser testado manualmente?
Rodar:
Opcionalmente, validar o command:
Algum cenário de contexto que queira dar?
Esta é a base operacional da história Silver. Ela ainda não implementa estratégias específicas de Article, Books, Preprints ou SciELO Data; esses fluxos devem entrar em PRs separados. Também mantém
search_gatewaycomo dono do helper genérico de OpenSearch, enquantoetlconsome essa integração.Screenshots
N/A.
Quais são tickets relevantes?
Resolve parte estrutural de #645.
Referências