-
Notifications
You must be signed in to change notification settings - Fork 550
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Add freeform task execution metadata to task executions table #325
Comments
This has to be done in two parts |
@EngHabu we already do have https://github.com/flyteorg/flyteidl/blob/da3f0f7695147f79a65f0348ba0f626f054ad28e/protos/flyteidl/event/event.proto#L147 although it is a struct rather than map. this is accessible for calls to get and list task executions https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/admin/task_execution.proto#L98 |
Signed-off-by: Sean Lin <sean@union.ai>
* add retry wip Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * add tests Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix lint Signed-off-by: Sonja Ericsson <sonjae@spotify.com>
…g#325) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.25.11 to 1.26.5. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.25.11...1.26.5) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* supporting data loading config for k8s pod type Signed-off-by: Daniel Rammer <daniel@union.ai> * should be checking dataLoadingConfig for nil Signed-off-by: Daniel Rammer <daniel@union.ai> * updated flyteidl Signed-off-by: Daniel Rammer <daniel@union.ai> --------- Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Sean Lin <sean@union.ai>
* add retry wip Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * add tests Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix lint Signed-off-by: Sonja Ericsson <sonjae@spotify.com>
* supporting data loading config for k8s pod type Signed-off-by: Daniel Rammer <daniel@union.ai> * should be checking dataLoadingConfig for nil Signed-off-by: Daniel Rammer <daniel@union.ai> * updated flyteidl Signed-off-by: Daniel Rammer <daniel@union.ai> --------- Signed-off-by: Daniel Rammer <daniel@union.ai>
Motivation: Why do you think this is important?
Allow additional data to be passed as task execution metadata. It's unclear how useful each of these pieces is at the moment but surfacing them might be a good way to figure that out.
Goal: What should the final outcome look like, ideally?
Add execution metadata to task execution events.
An example of this task metadata could be - nodeId for a pod where the node was executed
Cluster id for the hive cluster etc
Qubole Command ID
Resource Token that was associated
podID
AWS Batch Job ID
Spark application id
etc
Thoughts: A simple Map<string,string> called executionMetadata, which we can show as tabular information in the UI
For node name it is part of the podspec - https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#pod-v1-core
Flyte component
The text was updated successfully, but these errors were encountered: