Repository navigation
-
New features:
- Added the
py_permanently_disable_statsconfig option
(--grain_py_permanently_disable_stats) to permanently disable dataset
stats collection and avoid stats tree initialization overhead. - Added
HFIterDatasetto wrap Hugging Face streaming datasets, supporting
sharding, slicing (set_slice), and state checkpointing. MapDataset.slicenow accepts aSequence[int](including a
MapDataset[int]) in addition to asliceobject, enabling arbitrary index
remapping similarly to NumPy's advanced indexing.- Added
wrap_aroundoption toMapDataset.slice/SliceMapDatasetto
restore modular index wrapping when transformations such asshuffleor
mixsit betweensliceandrepeat. - Added
ElasticIterDatasetIterator(and OrbaxCheckpointHandlersupport)
for scaling the number of shards up and down between checkpoints, with
support for interleaves, mixtures, and zip datasets inElasticIterator. - Added the
/grain/python/dataset/source_read_time_ustelemetry metric,
recording source read latency in microseconds. The existing
/grain/python/dataset/source_read_time_nshistogram overflows at ~4s,
which hid the tail of very slow reads.
- Added the
-
Bug fixes:
- Fixed installing Grain from source (
pip install git+https://github.com/google/grain.git)
by building protobuf bindings and pybind11 extensions insetup.py. - Added exponential bucketer for
/grain/python/data_loader/iterator_get_next
and/grain/python/dataset/prefetch_buffer_ready_countmetrics so that
histograms capture values properly instead of overflowing or using defaults. - Fixes an issue where interpreter shutdown stops responding in
ProcessPrefetchDatasetIteratorafterset_state()is called. - Fixed bug in
DataLoaderwhere sharding remainder was dropped even when
ShardOptions.drop_remainder=False. - Support recursive in-place sharding (
set_slice) inBatchMapDatasetand
RepeatMapDataset, fixing remainder and length inconsistencies when
sharding across workers withsequential_slice=True. - Fixes
OverflowErrorwhen indexingShuffleMapDataset,
WindowShuffleMapDataset,MixedMapDataset, or concatenatedMapDatasets
with narrow NumPy integer scalars (e.g.np.int32). - Downcasts advanced indexing results on
SharedMemoryArrayto regular
np.ndarrayinstead of returning an unbackedSharedMemoryArray. - Fixes
UnparsedFlagAccessErrorwhen checking multiprocess worker profiling
status beforeabsl.flagsare parsed. - Fixes reference cycle in dataset stats and
BatchMapDataset. - Fixes
start_prefetchnot starting for sub-iterators of
InterleaveDatasetIterator, and fixesDatasetOptionspropagation into
nestedInterleaveIterDatasets. - Logs missing optional profiler dependencies at
DEBUGinstead ofWARNING
to avoid unnecessary stderr warnings. WindowShuffleIterDatasetwas generating a different sequence than
WindowShuffleMapDataset. A new optionreverse=Falseallows generating an
equivalent sequence (backward compatibility is preserved).- Fixes Grain Prometheus telemetry metrics not being exported when
PROMETHEUS_MULTIPROC_DIRis not set before import.
- Fixed installing Grain from source (
Full Changelog: v0.2.18...v0.2.19