[Core feature] Observability: expose Flyte runtime metrics

[cosmicBboy]

Motivation: Why do you think this is important?

Flyte performance is becoming increasingly important as users scale executions. Currently, Flyte exposes very elementary metrcis (ex. "started", "updated") without any description. This makes analysis of workflow execution performance very difficult. As a result, identifying system-level bottlenecks and implementing mitigations is extremely difficult.

Goal: What should the final outcome look like, ideally?

Flyte should expose fine-grained breakdowns of workflow, node, and task execution details. These should attribute the time spent during execution to various categories (ex. "flyte overhead", "plugin overhead", "user code", etc).

Describe alternatives you've considered

Discussion on the RFC link below may contain more information on alternative approaches.

Propose: Link/Inline OR Additional context

This work is described as "runtime metrics" in the performance observability RFC.

PRs:

• Add endpoint exposing runtime metrics flyteadmin#524
• Create service for runtime metrics flyteidl#367
• Added metrics command to pyflyte flytekit#1513
• Add reported_at timestamps to events flytepropeller#529
• Fixing pod plugin event reporting timestamps flyteplugins#307

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[cosmicBboy]

After the 5 above PRs are merged, we need to create a new issue for the frontend work to add the runtime metrics to flyteconsole