New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DL4J Spark fixes (Gradient Sharing) #5858

Merged
merged 9 commits into from Jul 9, 2018

Conversation

Projects
None yet
1 participant
@AlexDBlack
Copy link
Member

AlexDBlack commented Jul 9, 2018

Fixes/improvements for gradinet sharing.

Specifically:

  • Bug fix in repartitioning for gradient sharing
  • Makes repartitioning more configurable (adds Repartitioner interface)
  • Improves repartitioning (new default algorithm allows spark scheduler to do proper load balancing)
  • Post training, logs number of minibatches fit for each JVM

@AlexDBlack AlexDBlack changed the title [WIP] DL4J Spark fixes (Gradient Sharing) DL4J Spark fixes (Gradient Sharing) Jul 9, 2018

@AlexDBlack AlexDBlack merged commit 2ead48e into master Jul 9, 2018

0 of 2 checks passed

codeclimate Code Climate is analyzing this code.
Details
continuous-integration/jenkins/pr-merge This commit is being built
Details

@AlexDBlack AlexDBlack deleted the ab_sparkdebug branch Jul 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment