Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] No training log when the num of iterations smaller than default interval #935

Closed
2 tasks done
Tau-J opened this issue Feb 17, 2023 · 2 comments · Fixed by #1046
Closed
2 tasks done

[Bug] No training log when the num of iterations smaller than default interval #935

Tau-J opened this issue Feb 17, 2023 · 2 comments · Fixed by #1046
Assignees
Labels
bug Something isn't working

Comments

@Tau-J
Copy link
Contributor

Tau-J commented Feb 17, 2023

Prerequisite

Environment

mmengine == 0.5.0

Reproduces the problem - code sample

Reproduces the problem - command or script

Reproduces the problem - error message

Additional information

When the number of iterations per epoch is smaller than the log interval, there's no training info be printed. Since important information like eta is in training logs, I suggest printing at least one line of training logs per epoch.

Here's an example of current training logs:

2023/01/31 13:44:41 - mmengine - INFO - Saving checkpoint at 200 epochs
2023/01/31 13:44:45 - mmengine - INFO - Evaluating MpiiPCKAccuracy (normalized by ``"head_size"``)...
2023/01/31 13:44:45 - mmengine - INFO - Epoch(val) [200][12/12]  Head PCK: 96.958305  Shoulder PCK: 96.375085  Elbow PCK: 90.350029  Wrist PCK: 85.442129  Hip PCK: 89.800562  Knee PCK: 86.871427  Ankle PCK: 82.686027  PCK: 90.257132  PCK@0.1: 33.794920
2023/01/31 13:45:05 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:45:35 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:45:56 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:46:17 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:46:30 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:46:37 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:46:57 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:47:18 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:47:38 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:47:58 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:48:19 - mmengine - INFO - Exp name: rtmpose-m_finetune-aic-coco_8xb64-210e_mpii_20230131_123528
2023/01/31 13:48:19 - mmengine - INFO - Saving checkpoint at 210 epochs
2023/01/31 13:48:22 - mmengine - INFO - Evaluating MpiiPCKAccuracy (normalized by ``"head_size"``)...
@Tau-J Tau-J added the bug Something isn't working label Feb 17, 2023
@HAOCHENYE
Copy link
Collaborator

You can set ignore_last=False in the configuration of LoggerHook. Considering that the number of samples in the last iteration is uncertain and the averaged value could be relatively unstable, the log for the last iteration will be ignored by default.

@Tau-J
Copy link
Contributor Author

Tau-J commented Feb 20, 2023

Hi @HAOCHENYE , I mean, users need to know the eta, but for now, this kind of information will not be printed when the number of samples is small.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants