Skip to content

Grain 0.2.19

Latest

Choose a tag to compare

@yk5 yk5 released this 13 Nov 20:13
· 6 commits to main since this release
  • New features:

    • Added the py_permanently_disable_stats config option
      (--grain_py_permanently_disable_stats) to permanently disable dataset
      stats collection and avoid stats tree initialization overhead.
    • Added HFIterDataset to wrap Hugging Face streaming datasets, supporting
      sharding, slicing (set_slice), and state checkpointing.
    • MapDataset.slice now accepts a Sequence[int] (including a
      MapDataset[int]) in addition to a slice object, enabling arbitrary index
      remapping similarly to NumPy's advanced indexing.
    • Added wrap_around option to MapDataset.slice / SliceMapDataset to
      restore modular index wrapping when transformations such as shuffle or
      mix sit between slice and repeat.
    • Added ElasticIterDatasetIterator (and Orbax CheckpointHandler support)
      for scaling the number of shards up and down between checkpoints, with
      support for interleaves, mixtures, and zip datasets in ElasticIterator.
    • Added the /grain/python/dataset/source_read_time_us telemetry metric,
      recording source read latency in microseconds. The existing
      /grain/python/dataset/source_read_time_ns histogram overflows at ~4s,
      which hid the tail of very slow reads.
  • Bug fixes:

    • Fixed installing Grain from source (pip install git+https://github.com/google/grain.git)
      by building protobuf bindings and pybind11 extensions in setup.py.
    • Added exponential bucketer for /grain/python/data_loader/iterator_get_next
      and /grain/python/dataset/prefetch_buffer_ready_count metrics so that
      histograms capture values properly instead of overflowing or using defaults.
    • Fixes an issue where interpreter shutdown stops responding in
      ProcessPrefetchDatasetIterator after set_state() is called.
    • Fixed bug in DataLoader where sharding remainder was dropped even when
      ShardOptions.drop_remainder=False.
    • Support recursive in-place sharding (set_slice) in BatchMapDataset and
      RepeatMapDataset, fixing remainder and length inconsistencies when
      sharding across workers with sequential_slice=True.
    • Fixes OverflowError when indexing ShuffleMapDataset,
      WindowShuffleMapDataset, MixedMapDataset, or concatenated MapDatasets
      with narrow NumPy integer scalars (e.g. np.int32).
    • Downcasts advanced indexing results on SharedMemoryArray to regular
      np.ndarray instead of returning an unbacked SharedMemoryArray.
    • Fixes UnparsedFlagAccessError when checking multiprocess worker profiling
      status before absl.flags are parsed.
    • Fixes reference cycle in dataset stats and BatchMapDataset.
    • Fixes start_prefetch not starting for sub-iterators of
      InterleaveDatasetIterator, and fixes DatasetOptions propagation into
      nested InterleaveIterDatasets.
    • Logs missing optional profiler dependencies at DEBUG instead of WARNING
      to avoid unnecessary stderr warnings.
    • WindowShuffleIterDataset was generating a different sequence than
      WindowShuffleMapDataset. A new option reverse=False allows generating an
      equivalent sequence (backward compatibility is preserved).
    • Fixes Grain Prometheus telemetry metrics not being exported when
      PROMETHEUS_MULTIPROC_DIR is not set before import.

Full Changelog: v0.2.18...v0.2.19