Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Intel MKL] SparseAdam: an optimizer to improve the accuracy of LazyAdam #24788

Closed
wants to merge 7 commits into
base: master
from

Conversation

5 participants
@mpjlu
Copy link
Contributor

mpjlu commented Jan 9, 2019

LazyAdam is not convergent on NCF (MLPerf requires NCF to reach 0.635 HR in 10 epchos, but with LazyAdam HR can’t reach the goal).

The main problem of LazyAdam is it doesn't update m and v when the gradient is 0. Actually, m and v should be updated when other weight changed. So the convergence accuracy of LazyAdam is lower than Adam.

I have designed a method to optimize Adam on Sparse data (named SparseAdam) : SparseAdam provides similar semantics as the original Adam algorithm, so the convergence is also close to Adam, and the TPT is about the same as LazyAdam.

The only change of our method to LazyAdam is we compute the learning rate based the the steps skipped.

Fig 1. The convergence comparing of Adam, LazyAdam and SparseAdam on NCF (Hit Rate score)
adam lazyadam and sparseadam comparing

Fig 2. The thoughput comparing of Adam, LazyAdam and SparseAdam
ncf throughput comparing

Fig 3. SparseAdam algorithm:
sparseadam2

agramesh1 and others added some commits Nov 20, 2018

Temporarily disabling Conv2D fusions in grappler (#442)
* Temorarily commented out Conv2D fusion

* Commented out Conv2D fusion tests.
sparse Adam as a new optimizer
Change-Id: I6a842c30c5838f7bc701156df0129cf2bf73abd2
add AparseAdam test cases and format fix
Change-Id: I3274f135231d89f956a252f4e82d4f8bc6931ee8

@mpjlu mpjlu requested review from alextp and strategist333 as code owners Jan 9, 2019

@googlebot googlebot added the cla: yes label Jan 9, 2019

@ymodak ymodak self-assigned this Jan 9, 2019

@ymodak ymodak added this to Assigned Reviewer in PR Queue via automation Jan 9, 2019

@alextp

This comment has been minimized.

Copy link
Member

alextp commented Jan 9, 2019

Normally I'd accept this, but we're in the process of transitioning out of contrib and into the new addons SIG ( https://groups.google.com/a/tensorflow.org/forum/#!forum/addons for more information). So until that release is done I'd like to soft-freeze contrib to make the migration easier.

@alextp alextp closed this Jan 9, 2019

PR Queue automation moved this from Assigned Reviewer to Closed/Rejected Jan 9, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.