Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add freeform task execution metadata to task executions table #325

Closed
1 of 13 tasks
EngHabu opened this issue May 27, 2020 · 2 comments
Closed
1 of 13 tasks
Assignees
Labels
enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers

Comments

@EngHabu
Copy link
Contributor

EngHabu commented May 27, 2020

Motivation: Why do you think this is important?
Allow additional data to be passed as task execution metadata. It's unclear how useful each of these pieces is at the moment but surfacing them might be a good way to figure that out.

Goal: What should the final outcome look like, ideally?
Add execution metadata to task execution events.
An example of this task metadata could be - nodeId for a pod where the node was executed
Cluster id for the hive cluster etc
Qubole Command ID
Resource Token that was associated
podID
AWS Batch Job ID
Spark application id
etc
Thoughts: A simple Map<string,string> called executionMetadata, which we can show as tabular information in the UI
For node name it is part of the podspec - https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#pod-v1-core

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other
@EngHabu EngHabu added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels May 27, 2020
@kumare3
Copy link
Contributor

kumare3 commented Jun 10, 2020

This has to be done in two parts
PART 1: Event has to be updated to add this metadata
PART 2: Update admin to add it to the table and expose it in the API

@katrogan
Copy link
Contributor

@katrogan katrogan self-assigned this Mar 15, 2021
@katrogan katrogan closed this as completed Apr 7, 2021
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
Signed-off-by: Sean Lin <sean@union.ai>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
* add retry wip

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* add tests

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix lint

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
…g#325)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.25.11 to 1.26.5.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](urllib3/urllib3@1.25.11...1.26.5)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Jul 24, 2023
* supporting data loading config for k8s pod type

Signed-off-by: Daniel Rammer <daniel@union.ai>

* should be checking dataLoadingConfig for nil

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated flyteidl

Signed-off-by: Daniel Rammer <daniel@union.ai>

---------

Signed-off-by: Daniel Rammer <daniel@union.ai>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
Signed-off-by: Sean Lin <sean@union.ai>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
* add retry wip

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* add tests

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>

* fix lint

Signed-off-by: Sonja Ericsson <sonjae@spotify.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
* supporting data loading config for k8s pod type

Signed-off-by: Daniel Rammer <daniel@union.ai>

* should be checking dataLoadingConfig for nil

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated flyteidl

Signed-off-by: Daniel Rammer <daniel@union.ai>

---------

Signed-off-by: Daniel Rammer <daniel@union.ai>
troychiu pushed a commit that referenced this issue Jul 8, 2024
…#325)

## Overview
This PR fixes an issue where FlytePropeller restarts and the fasttask plugin checks the status of a task execution where the task execution context does not exist. Rather than fail, it creates the task execution context (that will be cleaned up later) and returns running.

## Test Plan
This was tested locally under a variety of failure scenarios.

## Rollout Plan (if applicable)
This may be rolled out immediately.

## Upstream Changes
Should this change be upstreamed to OSS (flyteorg/flyte)? If not, please uncheck this box, which is used for auditing. Note, it is the responsibility of each developer to actually upstream their changes. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F).
- [ ] To be upstreamed to OSS

## Issue
https://linear.app/unionai/issue/COR-1128/fasttask-plugin-checks-status-of-task-execution-failure-on-restart

## Checklist
* [x] Added tests
* [ ] Ran a deploy dry run and shared the terraform plan
* [ ] Added logging and metrics
* [ ] Updated [dashboards](https://unionai.grafana.net/dashboards) and [alerts](https://unionai.grafana.net/alerting/list)
* [ ] Updated documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers
Projects
None yet
Development

No branches or pull requests

3 participants