You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Background
extract step does not generate any execution / tracing info, let's fix that. Should not be started before #757 is implemented and common interface of all step infos is clear
Tasks
implement ExtractInfo with asdict and asstr methods like the other infos. asdict must generate dlt friendly object that can be loaded into a relational structure, asstr must be friendly to our pipeline info command that displays traces
extract step may extract many sources. each extracted source is a separate operation and ExtractInfo must collect all of them.
a DAG of selected resources (by name - they are unique). mind that source has a method that generates DAGs
for each source and resource a list of applied hints
for each resource collect metrics: the count of the items, number of written bytes, the counts of extracted items per table + bytes, user defined metrics (ticket icoming)
for each table: the list of extracted files and their sizes
elapsed time for each resource
resource state, source state (this should be optional, maybe extractor config)
resource arguments (if created with decorator) - we have that partially implemented!
source arguments (if created with decorator) - to be implemented
Implementation
You must be able to get all the data above after execution is finished. You must be able to collect partial info also when we have an exception during extract (info is part of exception)
users can add custom metrics to resources that will be part of extract info. please read #755 for some implementation hints. we can collect metrics in organized way even before custom metrics are added
The text was updated successfully, but these errors were encountered:
Background
extract step does not generate any execution / tracing info, let's fix that. Should not be started before #757 is implemented and common interface of all step infos is clear
Tasks
ExtractInfo
withasdict
andasstr
methods like the other infos.asdict
must generatedlt
friendly object that can be loaded into a relational structure,asstr
must be friendly to ourpipeline info
command that displays tracesExtractInfo
must collect all of them.NormalizeInfo
andLoadInfo
#757extract info should contain the following information
in extract info per source
Implementation
You must be able to get all the data above after execution is finished. You must be able to collect partial info also when we have an exception during extract (info is part of exception)
users can add custom metrics to resources that will be part of extract info. please read #755 for some implementation hints. we can collect metrics in organized way even before custom metrics are added
The text was updated successfully, but these errors were encountered: