-
Notifications
You must be signed in to change notification settings - Fork 6.8k
-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Performance differs between 0.9.4 and 0.9.5 #5816
Comments
@magic282 What's the optimizer you are using? We've fixed a bug in RMSProp and changed it to be non-centered. |
@sxjscience I am using Adam. |
that's interesting. can you please do a git bisect to locate which PR makes the diff? |
@mli Since it is too slow to build under Windows(takes 30 mins on my server), I just tested some nightly builds:
where error means my code crashed for these builds.
So my guess is, something after 0405 introduced bugs. |
@magic282 Is it a language model task? If so, which dataset do you use? PTB? |
The last three bads means build failure, and other bad means performance problem. |
Did you use clip_gradient? @magic282 |
the ssd commit b35148e only adds io, contrib operators and ssd examples, i think it should not affect the results. also the last bad commit 1383446 only updated documents, and it should also be fine. it is possible due to the numerical instabilitiy? say if you repeat it multiple times, some of them are good, some of them are not |
@alanhuang1990 Yes I use clip_gradient. |
@magic282 You can try the old python implementation of Adam optimizer. I have the same convergence problem in this version using RMSProp, but the old optimizer implementation works well. |
@alanhuang1990 So it's the optimizer's fault? |
@magic282 Not sure about it. You can try it in your problem. |
both adam and rmsprop are updated on mar 1
7510ef1
why compiler error? because you didn't compile with opencv? you can hack it
by remove the .cc files that caused the problem.
…On Wed, Apr 19, 2017 at 8:37 PM, alanhuang1990 ***@***.***> wrote:
@magic282 <https://github.com/magic282> Not sure about it. You can try it
in your problem.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#5816 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAZv4dDPj3qPodYS9g86uOOjQyiswCe0ks5rxtLcgaJpZM4M8P6d>
.
|
@mli Hi Mu, I removed the files that caused compile error, but it still reports compile errors:
Any hints about solving this? |
Sorry the previous error is not correct. I redo the cmake process, the following is the error mesage:
|
Hi, I did the git bisect under an linux server and got the result.
The |
Close it now. @magic282 Feel free to reopen it if you have other findings. |
Hi,
I found that the latest mxnet 0.9.5 I cloned yesterday performs very differently from 0.9.4.
The same code generates very different results with random seed fixed. I am doing an NMT task, for 0.9.4(20170325_mxnet_x64_vc14_gpu.7z), the perplexity drops fast:
and after 10 epoches, the perp drops to less than 2.0;
However, for 0.9.5 (cloned yesterday: 6179974):
and after 10 epoches, the perp is still about 40.
So I infer that something went wrong between the two versions.
The text was updated successfully, but these errors were encountered: