Skip to content

Implement distributed learning iterations over mini-batches #83

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 75 commits into from
Apr 3, 2020
Merged

Implement distributed learning iterations over mini-batches #83

merged 75 commits into from
Apr 3, 2020

Conversation

@bowni
Copy link
Member

@bowni bowni commented Mar 27, 2020 •
edited
Loading

Implements and closes #4:

  • for each epoch, splits the nodes datasets into mini-batches
  • iterates over mini-batches for training individual models and aggregating them

@bowni bowni added the enhancement New feature or request label Mar 27, 2020
@bowni bowni added this to the 0.3 Consolidation milestone Mar 27, 2020
@bowni bowni self-assigned this Mar 27, 2020
@bowni bowni changed the title Iss4 mini batches Implement distributed learning iterations over mini-batches Mar 27, 2020
Thomas-Galtier
Thomas-Galtier reviewed Mar 28, 2020
View reviewed changes
Thomas-Galtier
Thomas-Galtier reviewed Mar 28, 2020
View reviewed changes
Copy link
Collaborator

@Thomas-Galtier Thomas-Galtier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not tested the code on my machine yet but it looks like a great work! 👍

I have a remark on a machine learning issue: concerning the early stopping, should we rather yield the model with le lower loss rather than the last one which has an increased loss?

@Thomas-Galtier Thomas-Galtier self-requested a review March 28, 2020 15:01
@bowni bowni mentioned this pull request Mar 30, 2020
Closed
@bowni
Copy link
Member Author

bowni commented Mar 30, 2020

I just rebased with master (following recent merge of PR #80) to enable an easier review, test and merge @RomainGoussault.

RomainGoussault
RomainGoussault requested changes Mar 30, 2020
View reviewed changes
Copy link
Member

@RomainGoussault RomainGoussault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on a complicated and tricky subject Eric !!
Looking forward to test the performance improvement :)

bowni and others added 26 commits April 2, 2020 18:02
Fix rebase mistake and restore normalization of aggregation weights
b67acf4
Rename data-volume to data_volume
c03827a
Rename not-corrupted to not_corrupted
d0cd599
Validate on global validation dataset instead of local one
b9eee14
Remove default values for aggregation_weighting and minibatch_count
4ed18f7
Make minibatch_count a configurable scenario parameter
05d3ac0
Clarify index for iteration over mini-batches
5395183
Remove default values for aggregation_weighting and minibatch_count
3e18aab
Remove empty first piece of the split
40ee792
Fix case where minibatch_count is 1
6aed9bb
@RomainGoussault
Add minibatch_count to csv output file
7d72f97
Align prints of epochs and mini-batches
99462d8
Clarify model aggregation for early stopping evaluation
1dd3864
Align format of early stopping print statements
995cb89
Shuffle dataset before split in mini-batches and finetune console pri…
d5fc6c7 
…nts for better lisibility
@RomainGoussault
Add federated_computation_time in scenario
d8d4c4c
@RomainGoussault
Add minibatch count analysis
eea7817
Add minibatch_count to results_summary.txt
199e5c9
Put the mini-batch split in a dedicated function
cf51990
Finetune comments for lisibility
b91c8e1
Move initialization of model within iteration over nodes
0c56a3d
Aggregate model once at the beginning of each mini-batch and use it f…
5bb053d 
…or each node
Create a distinct aggregated model for each node at the beginning of …
b24e4b7 
…each iteration
Decrease BATCH_SIZE to 1024 and refine certain prints
1dd4299
Clarify comments
9bc878c
Merge branch 'iss4-mini-batches' of https://github.com/SubstraFoundat…
a41f65b 
…ion/distributed-learning-contributivity into iss4-mini-batches
@bowni
Copy link
Member Author

bowni commented Apr 2, 2020

Just rebased with recent merge on master.

RomainGoussault
RomainGoussault reviewed Apr 3, 2020
View reviewed changes
Copy link
Member

@RomainGoussault RomainGoussault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥇

@RomainGoussault RomainGoussault merged commit 2880cee into master Apr 3, 2020
@RomainGoussault RomainGoussault deleted the iss4-mini-batches branch April 9, 2020 06:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RomainGoussault RomainGoussault RomainGoussault requested changes

@Thomas-Galtier Thomas-Galtier Thomas-Galtier approved these changes

@natct10 natct10 Awaiting requested review from natct10 natct10 is a code owner

Assignees

@bowni bowni

Labels

enhancement New feature or request

Projects

None yet

Milestone

0.3 Consolidation

Development

Successfully merging this pull request may close these issues.

Parameterize size of mini-batches

4 participants

@bowni @RomainGoussault @Thomas-Galtier