Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

terrible output after 2k itr #157

Open
Continue7777 opened this issue Aug 6, 2021 · 2 comments
Open

terrible output after 2k itr #157

Continue7777 opened this issue Aug 6, 2021 · 2 comments

Comments

@Continue7777
Copy link

what happen here,and what can i do?

dataset:deepfashion landmark filter about 1.6w.
cfg:default

itr:2k
image

itr:3k
image

about loss
image
image
image

@thusinh1969
Copy link

Same. After 4,000 itr, FiD wildly jumps all over the places, from 58 to 634 and hanging over there ... Shall we stop training and do it again ? What is going on ?

Steve

@vsemecky
Copy link

vsemecky commented May 3, 2022

Since it looked pretty good in 2 kimg, I'd try to slow it down. What learning rate did you use?

I often use LR 0.003, but when the network becomes unstable, I need to lower it to 0.002 or even 0.001.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants