Close mlflow run when a pipeline fails in interactive mode

[Galileo-Galilei]

Context

Today, you can execute a kedro pipeline interactively. The logic would be to load the context, and then to run the pipeline.

from kedro.context import load_context
local_context = load_context(".")
local_context.run(pipeline=local_context.pipelines[PIPELINE_NAME],
                             catalog=local_context.catalog)

Description

If the execution fails for some reason (bug in the pipeline), the mlflow run is not closed. This creates unintended side effects: for instance, if you rerun the pipeline, the new run will be nested in the failing runs and the mllflow database will become very messy.

This bug does not occur when running from the command line since the mlflow run is automatically closed when exiting.

Possible Implementation

Implement a on_pipeline_error kedro Hook to close the mlflow run when the pipeline fails.