Seq2Seq Trainer Handling for MLFlow Exception

[njbrake]

System Info

Transformers 4.28.1, torch 1.13.1

When using the Seq2SeqTrainer with the MLFlow integration enabled, if I lose my connection to mlflow after the training has begun (if the server crashes or if there is a network error), MLFlow throws the exception:

raise MlflowException(f"API request to {url} failed with exception {e}")

I don't know if this is a bug, or if you have a recommended way to continue: I would like the option to have the seq2seqtrainer log the mlflow connection error but then continue training with the mlflow integration disabled.

I imagine that this behavior would apply to any integration, not just mlflow.

Who can help?

@sgugger

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Start training with Seq2Seq Trainer with a connection to MLflow
2. During training, stop the mlflow server
3. Seq2Seq Trainer raises an MLFlow Exception

Expected behavior

I would expect that there would be an option to allow for mlflow exception to disable the integration but continue training.

[sgugger]

We don't maintain the MlFlow integration ourselves, so I can't really help. Maybe try to tag the persons who added it?

[njbrake]

Thanks for the quick reply! Although my question involves MLFlow, I think the question more broadly is:

if an integration callback throws an error, how can we disable that integration and continue with training?

I think that might be a question for the huggingface team and not for the MLFlow integrators?

[sgugger]

Oh I dind't realize you were asking for that. This is what the report_to argument is for :-)

[njbrake]

Ah. Perfect. Thanks so much!