Multigpu #13
Merged
Multigpu #13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* tf.split removed from graph, the placeholders are now in a list * split the operations of the graph according to the number of working towers * modify compute_chunk_size function in utils to compute the chunks directly
* Improve the structure by reusing code and using dictionaries
* Select the parts of the graph we need, depending on the number of GPUs being used at each iteration (at runtime!) * Select the summaries we care about, depending on the number of GPUs being used at each iteration (at runtime!) * Create one graph per GPU and dataset split * Collect all the summaries inside build_graph, as should be
Request that all the sequences in a minibatch belong to the same subset
fvisin
force-pushed
the
multigpu
branch
2 times, most recently
from
5d461f9
to
23c2396
Compare
* Improve names * Allow empty name in write_IoUs_summaries * Use the number of epochs for the x-axis
fvisin
force-pushed
the
multigpu
branch
2 times, most recently
from
637f051
to
66db7c4
Compare
Force each of the grad update ops to depend only on the variables that are actually related to the devices linked to that op
marcociccone
approved these changes
Jun 8, 2017
There was a problem hiding this comment.
LGTM, I tested several times and it seems to be working correctly! The only thing I noticed it's that when you reload the parameters and continue the training, the plot of the IoU metrics become a mess... I don't know if it's related to this PR so we can merge and look at it later on
|
Great thanks! I'll merge this one and open an issue with what you reported then! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Completes #12.
This PR adds multiGPU support for training. The code loads N batches of size
batchsizewhen N devices are specified. It then feeds each device with one batch. The code also deals with the case where the dataset returns a batch that is smaller thanN*batchsize(e.g., at the end of a video when the number of sequences is not a multiple ofN*batchsize)