Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maybe a small bug about test progress bar in multi_gpu_test(). #921

Open
MisakaMikt opened this issue Dec 8, 2023 · 1 comment
Open

Comments

@MisakaMikt
Copy link

During distributed training, sot model use multi_gpu_test() func to test all videos on several GPUs. But the progress bar is updated on rank 0.

https://github.com/open-mmlab/mmtracking/blob/e79491ec8f0b8c86fda947fbaaa824c66ab2a991/mmtrack/apis/test.py#L160C34-L160C34

The number of frames on each rank is not equal, in my run, the number of frames on rank0 is less than total_frames / world_size, so the progress bar will be less than the actual total number of frames. For example, I used 3 gpu, and total 21007 frames(got10k val), and here is some out info. the final progress only 5565*3=16695.

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ] 16695/21007, 99.7 task/s, elapsed: 167s, ETA: 43s
rank 0 world_size 3 cnt 5565.
rank 1 world_size 3 cnt 7072.
rank 2 world_size 3 cnt 8370.

@tericalpha
Copy link

tericalpha commented Dec 18, 2023

Hello @MisakaMikt
Haven't you got this MMCV compatibility error?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants