-
Notifications
You must be signed in to change notification settings - Fork 74k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improving local run behavior in estimator.train_and_evaluate.
Current behavior is unintuitive (depends on throttle_secs) and leads to frequent checkpoint than desired. This CL makes evaluation synchronized with checkpointing. It also makes the behavior more closer to distributed setting in following ways: * in distributed setting we do create input_pipeline only once, in current behavior of local run we do recreate input pipeline in a loop. This cl creates training input pipeline only once. * in distributed setting evaluator job waits for checkpoints which are dumped by training job. In current behavior of local run evaluator controls the checkpoint schedule. In this cl, we give back the control to trainer. PiperOrigin-RevId: 201085814
- Loading branch information
1 parent
f91b5b0
commit 3edb609
Showing
2 changed files
with
231 additions
and
251 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.