Fix resuming arrow format (#6964)

* fix resuming in arrow format * one more * fix arrow resuming
huggingface · Jun 14, 2024 · ef2fb35 · ef2fb35
1 parent 087671d
commit ef2fb35
Show file tree

Hide file tree

Showing 3 changed files with 357 additions and 215 deletions.
diff --git a/docs/source/stream.mdx b/docs/source/stream.mdx
@@ -415,6 +415,6 @@ This can be used with the `StatefulDataLoader` from `torchdata`:
 
 <Tip>
 
-Resuming returns exactly where the checkpoint was saved except in two cases: 1) examples from shuffle buffers are lost when resuming and the buffers are refilled with new data and 2) combinations of `.with_format(arrow)` and batched `.map()` may skip one batch.
+Resuming returns exactly where the checkpoint was saved except if `.shuffle()` is used: examples from shuffle buffers are lost when resuming and the buffers are refilled with new data.
 
 </Tip>