implement LoadInfo
and ExtractInfo
missing tracing
#853
Labels
tech-debt
Leftovers from previous sprint that should be fixed over time
Background
We release a refactored tracing for
0.4.1
that unified the shapes of data across pipeline step. A few important features are still missingTasks
LoadInfo
(after #757 )load_id
so extend load info to keep this informationelapsed
fromLoadJobInfo
. Instead implement proper metrics for jobs generated inload.py
. We should get start, stop times of the jobs initially. mind that you need to persist the metrics as this step is not atomic and may be restartedExtractInfo
(after #754)We should be careful what we send. Some of the arguments are secrets so initially we should only send arguments that are typed and not typed as secrets. for any other arguments we should just information that argument got set
Tests
LoadInfo
foruse_single_dataset
pipeline setting which will generate separate dataset for each schema (we should move this config to load config or destination config)hints
trace too wellThe text was updated successfully, but these errors were encountered: