Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AsyncDataSetIterator not used by default when using an EarlyStoppingTrainer #7190

Closed
madhs53 opened this issue Feb 19, 2019 · 1 comment

Comments

@madhs53
Copy link

commented Feb 19, 2019

When using an EarlyStoppingTrainer, the trainer itself calls next() on the provided DataSetIterator and then fit(dataset) on the MultiLayerNetwork. This means we do not get the benefit of the AsyncDataSetIterator by default, and have to wrap the provided iterator manually before passing to EarlyStoppingTrainer.

This might be by design, but I thought I'd raise this issue in case it isn't. When I read that MultiLayerNetwork uses AsyncDataSetIterator by default, I expected that to be the case even when using EarlyStoppingTrainer.

Suggest updating EarlyStoppingTrainer or updating documentation on EarlyStoppingTrainer to suggest wrapping the provided iterator with an AsyncDataSetIterator.

@AlexDBlack AlexDBlack self-assigned this Feb 21, 2019

AlexDBlack added a commit that referenced this issue Feb 21, 2019
AlexDBlack added a commit that referenced this issue Feb 21, 2019
[WIP] DL4J embedding layer and early stopping async iter optimizations (
#7212)

* Switch Embedding layers to use Nd4j.scatterUpdate

* #7190 Add AsyncDataSetIterator to BaseEarlyStoppingTrainer (when supported)

* some logging

* some logging

* one test

* some more logging

* some more logging

* some more logging

* some more logging

* first set of tweaks

* few fixes
@lock

This comment has been minimized.

Copy link

commented Mar 23, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 23, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
2 participants
You can’t perform that action at this time.