More customizable configuration of logging links

[fg91]

I propose two enhancements that would make the configuration of logging links as documented here more powerful:

1. Logging links are only shown once the pod starts running while useful information could be provided before to users.

   A pod in queued status, no log links are shown:
   

   Once the pod is running, we link to the pod's overview page in the GCP cloud console:
   

   Even before a pod starts running, this page has useful information, e.g. in case the pod cannot be scheduled when the cluster cannot fulfil the task's resource requests.

   Having the ability to show some log links before the pod has started would give users not comfortable with kubectl an easy way to access this information directly from the Flyte console.

   This option could for instance be exposed like this:

   task_logs:
     plugins:
       logs:
         templates:
           - displayName: <name-to-show>
             templateUris:
               - "..."
             messageFormat: "json" # "unknown" | "csv" | "json"
             show_before_running: true
   

2. Providing more parameters for link templating:

   Currently these parameters can be used to template links.

   We would like to be able to add links to experiment tracking servers like Mlflow, Wandb, ...
   We typically add tags with e.g. the flyte task project, domain, version, and execution id (retrieved from the flyte context) to the runs in our experiment tracking server. If we had access to these values during link templating, we could provide links to the run/experiment in e.g. Wandb in the Flyte console. Currently we log this link and have to go to the stackdriver logs to retrieve it.

[fg91]

Discussion is continuing in #3696

[fg91]

The discussion continued for a bit in #3696. Let's take it back to the incubator.

Current state of the discussion

User story: as a user I want to run a long training and log the training progress in an experiment tracking server like Mlflow, Wandb, ClearML, Aim, ... I want to be able to navigate from the Flyte console to my run in the experiment tracking server while the task is still running, e.g. to monitor the training progress and decided whether to continue or abort the run.

1. Option:

Create templateable links in the task decorator

@task(log_links="{"wandb": f"https://my-experiment-tracking-server.com/?tag={{.executionId}}")

Pro:

• The implementation will be straight forward

Con:

• If the user omits to actually create the experiment in the experiment racking server or doesn't set the correct tags there that have been used for templating the link in the task decorator, Flyte console will show a dead link regardless.
• This approach only works in case the link is templateable. Some experiment tracking servers use uuids for run names so the link cannot be configured upfront in the task decorator (see details).

2. Option:

Allow flytekit to communicate a log link to flytepropeller even before the task has finished. This could for instance be done by allowing the user to flush the deck before the task finished:

@task
def train():
    run = some_experiment_tracking_api.start_run()
    deck = flytekit.Deck("Summary", ...)
    deck.append(MarkdownRenderer.to_html(f"... {run.link} ..."))
    deck.flush()

Pro:

• A lot more flexible, works with any experiment tracking server (and any other information users want to visualize in the flytyeconsole deck while the task is still running)
• The link would only be shown in the deck once it actually "points to something".

Con:

• Currently there is no communication channel from flytekit to propeller while the task is running. The implementation would be more complicated.

[btang-stripe]

Of the two options /2 seems more complete as it is able to handle urls of arbitrary complexity.

• /1 won't meet the needs of services that require an id that's generated at runtime (that's not the flyte execution id eg., wandb runs by default).
• For /2 is this purely a solution for flyte deck? ie., it will hide the log link under the "Flyte Deck" button, and then after opening the modal a user can navigate to the link? While this would work, it does make the information a bit less accessible. It would be more ideal if it were possible to display the link prominently at the top level along w/ the other log links

[vkaiser-mb]

I agree with @btang-stripe
For 1/ I think you still to have the option to set the id (e.g. to W&Bs) to the id of the flyte task. I guess I would prefer it anyways like this, so you have the same id in both tools. But I agree that this might not work for all tools?

But I totally share the opinion on /2. The UX via Flyte Deck would make this cumbersome from my POV, thats why I would prefer 1/ over 2/. Or even better to do it similar as in 2/, but give the option to add a log link directly (to the UI sidecar of the task).

[fg91]

See flyteorg/flytekit#1704 which might be used to solve this problem.

[davidmirror-ops]

2023-11-09 Contributor's meetup notes: this discussion will be used as an argument/use case for the RFC that will come out of this entry: #3838