[FR] set_experiment not safe for concurrency

[amueller]

Willingness to contribute

Yes. I would be willing to contribute this feature with guidance from the MLflow community.

Proposal Summary

Currently calling set_experiment with a new experiment name in multiple processes in parallel can lead to concurrency issues, in that all the processes will look for the experiment, not find it, try to create it, and all but one will find that it has already been created and error out.
It would be nice to catch the error and try to retrieve the experiment again once it already exists.

Motivation

What is the use case for this feature?

I'm doing hyper-parameter optimization in parallel, and I create the experiment on the workers. I could also ensure the experiment is created prior to calling the workers on the head node, but it seem like this would be easy to do as a feature in MLFlow.

Why is this use case valuable to support for MLflow users in general?

Having multiple workers try to access a non-existing experiment in parallel seems like a common scenario.

Why is this use case valuable to support for your project(s) or organization?

Currently I do a retry in my code. That seems the wrong place to do the retry. I'd rather not have mlflow specific logic in my application.

Why is it currently difficult to achieve this use case?

Not very difficult, but it seems more appropriate to support in mlflow directly (according to my understanding of mlflow scope, which is not very extensive)

Details

No response

What component(s) does this bug affect?

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/gateway: AI Gateway service, Gateway client APIs, third-party Gateway integrations
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

What language(s) does this bug affect?

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

[github-actions]

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.