-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix array indexing in create_multi_node_evaluator
#8568
Fix array indexing in create_multi_node_evaluator
#8568
Conversation
create_multi_node_evaluator
create_multi_node_evaluator
pfnci, test this please. |
Jenkins CI test (for commit 813a518, target branch master) succeeded! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Test are failing. This does not seem to be related to this PR, but could you take a look at it?
Yes, tests has been failing in PfnCI for a while due to out of memory errors, Sorry and thank you again. |
All the CI issues were fixed, the current failing job is due to having the env variables set on job creation so it can't be overridden with the new values unless the CIs are kicked again |
pfnCI, test this please. |
Jenkins CI test (for commit 813a518, target branch master) failed with status FAILURE. |
Jenkins, test this please |
Jenkins CI test (for commit 813a518, target branch master) succeeded! |
@keisukefukuda we fixed the CIs and now all test passed :D |
Closes #8546
In some cases
local_mean_dict.values()
is empty. For example in the above issue, it is empty for the rank 1 since the example explicitly creates an empty dataset for this rank.chainer/examples/chainermn/mnist/train_mnist_dual_parallel.py
Lines 124 to 126 in 8b9e9c0
Also metrics are not reported for rank 1.
chainer/examples/chainermn/mnist/train_mnist_dual_parallel.py
Lines 107 to 110 in 8b9e9c0