Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MLFlowLogger's status is "RUNNING" even after training failed #12291

Closed
ritsuki1227 opened this issue Mar 10, 2022 · 0 comments · Fixed by #12292
Closed

MLFlowLogger's status is "RUNNING" even after training failed #12291

ritsuki1227 opened this issue Mar 10, 2022 · 0 comments · Fixed by #12292
Assignees
Labels
feature Is an improvement or enhancement logger: mlflow
Milestone

Comments

@ritsuki1227
Copy link
Contributor

ritsuki1227 commented Mar 10, 2022

🐛 Bug

If a trainer with MLFlowLogger raises an error, the user should be able to see the MLflow's screen to check the training has been failed.
MLflow's status remains "RUNNING" even after trainer.fit raises an error in the current implementation, so the user cannot know whether the training is still in progress or failed.

Current behavior when training finished with an error:

スクリーンショット 2022-03-10 21 41 18

Expected behavior:

スクリーンショット 2022-03-10 21 31 23

To Reproduce

class CustomModel(BoringModel):
    def training_step(self, batch, batch_idx):
        super().training_step(batch, batch_idx)
        raise BaseException
trainer = Trainer(logger=MLFlowLogger("test"))
try:
    trainer.fit(CustomModel())
finally:
    print(trainer.logger.experiment.get_run(trainer.logger.run_id).info.status) # This should be 'FAILED'

cc @Borda

@akihironitta akihironitta added bug Something isn't working logger: mlflow labels Mar 10, 2022
@carmocca carmocca added this to the 1.7 milestone Apr 6, 2022
@carmocca carmocca added feature Is an improvement or enhancement and removed bug Something isn't working labels Apr 6, 2022
@carmocca carmocca modified the milestones: pl:1.7, pl:future Jul 19, 2022
@awaelchli awaelchli modified the milestones: pl:future, pl:1.8 Sep 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Is an improvement or enhancement logger: mlflow
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants