Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

About training under different version of pytorch and cuda #14

Closed
shangbuhuan13 opened this issue Feb 8, 2022 · 3 comments
Closed

About training under different version of pytorch and cuda #14

shangbuhuan13 opened this issue Feb 8, 2022 · 3 comments

Comments

@shangbuhuan13
Copy link

Thanks for your great work!
I am now training the code under pytorch 1.10 and cuda 11.0, because I don't have a proper GPU that satisfies the environment in README. However, I got a much lower result in AP40 moderate: 13.69, compared to the given ckpt 16.23.
Do you have some ideas about why the performance deteriorate sharply under different environments?
Thanks very much

@SuperMHP
Copy link
Owner

I do not know. Please mention that the model has a certain range of jitter. But 13.69 is very low. Most times will not happen. We also are developing the model in the higher torch.

@shangbuhuan13
Copy link
Author

Thanks for the reply.
We now can reproduce the results.
But a weird thing is that training under 3 GPUs outperforms training under a single GPU.
We first trained the network with 1 A100, and got 13.69.
Then we trained it with 3 A100s, and got 15.70.
Other conditions were kept the same.

All in all, we get the right results.
Thanks for your work again.

@Senwang98
Copy link

@SuperMHP
I used 3 * V100 to train GUPNet, But get 15mAP.
I use pt1.7 and cuda11 to train model, I think 15mAP is a little abnormal.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants