Support data.experimental.distribued_save
and add wait
to load
.
#64610
+165
−66
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support
data.experimental.distribued_save
and addwait
toload
.distribued_save
uses tf.data service(https://www.tensorflow.org/api_docs/python/tf/data/experimental/service)
to write distributed dataset snapshots. The call is non-blocking and
returns without waiting for the snapshot to finish. Setting
wait=True
totf.data.Dataset.load
allows the snapshots to be read while they arebeing written. The default is
False
for backward compatibility. It willraise an error if the requested snapshot does not exist.