Async calls to log_metric

[fariasfc]

Is it possible to call log_metrics and execute it asynchronously?

In my situation, I am doing experiments with simple models and simple datasets that are very fast to train/test, but when I try to log_metrics every epoch, it takes much longer due to the logs on mlflow. I was thinking that we should build some queue that could be processed in parallel with the original code, without blocking the main processes of train and test.

[fabboe]

also this would make it more fault-tolerant. currently an experiment that has an temporarily unreachable Tracking URI defined, will die.
Sacred does this, for example.

[juntai-zheng]

A potential workaround to the lag due to logging every epoch is instead using mlflow.log_metrics, or the lower-level MlflowClient.log_batch. Having an async process definitely would be useful as a built-in feature, though.

[dannyfriar]

Also interested in this. Would be very useful for both fault tolerance and for cases where the running time of an epoch is short.

[ygean]

@apurva-koti Is there any new progress now?

[dbczumar]

Hi folks, the MLflow fluent API, including mlflow.log_metrics(), is not designed for asynchronous execution (is not thread safe). We recommend using MlflowClient().log_batch() to record metrics asynchronously.

[you-n-g]

We created a simple async wrapper in our project which leverages MLflow

https://github.com/microsoft/qlib/blob/d7d19feb4ebb0c4318ac3bfda32a34c56e28a6a0/qlib/workflow/recorder.py#L298
https://github.com/microsoft/qlib/blob/d7d19feb4ebb0c4318ac3bfda32a34c56e28a6a0/qlib/workflow/recorder.py#L368
https://github.com/microsoft/qlib/blob/d7d19feb4ebb0c4318ac3bfda32a34c56e28a6a0/qlib/utils/paral.py#L94

hope it will be helpful.

[phelps-matthew]

The primary quantities that have a demand for async logging tend to be ones that are not simple parameters and metrics. For my use case, MlflowClient().log_batch() is clearly insufficient. I needed a manner to log images, plots, and analytics much faster.

@you-n-g 's solution was of immense help. I added some multi-threading capability to their code and simplified a few downstream patterns, for anyone who may find that helpful.

https://github.com/phelps-matthew/dl-schema/blob/torch-advanced/dl_schema/recorder_base.py