Commits

Branch selector

 main

User selector

All users

All time

Commit History

Commits on Jan 14, 2026

• Replace unicode escaped characters in ipynb files

  Show description for da74a25
  

  AleksMat

  authored and

  AirIO

  committed
  da74a25

  Copy full SHA for da74a25

Commits on Dec 16, 2025

• Deprecates the experimental API for `MultiprocessPrefetchIterDataset`.

  Show description for b6ca0f1
  

  iindyk

  authored and

  AirIO

  committed
  b6ca0f1

  Copy full SHA for b6ca0f1

Commits on Oct 27, 2025

• no-op

  Show description for 3c50084
  

  gauravmishra

  authored and

  AirIO

  committed
  3c50084

  Copy full SHA for 3c50084

• Publish AirIO docs for OSS code.

  Show description for b4d5035
  

  gauravmishra

  authored and

  AirIO

  committed
  b4d5035

  Copy full SHA for b4d5035

Commits on Oct 22, 2025

• Add slice-and-queue packing impl to airio pygrain.

  Show description for b76ba57
  

  gauravmishra

  authored and

  AirIO

  committed
  b76ba57

  Copy full SHA for b76ba57

Commits on Sep 24, 2025

• Update grain version in airio

  Show description for 4e3b376
  

  gauravmishra

  authored and

  AirIO

  committed
  4e3b376

  Copy full SHA for 4e3b376

Commits on May 7, 2025

• Simplify state of `MapDatasetIterator`.

  Show description for 7feb9dd
  

  iindyk

  authored and

  AirIO

  committed
  7feb9dd

  Copy full SHA for 7feb9dd

Commits on Apr 28, 2025

• No public description

  Show description for debc14b
  

  h-joo

  authored and

  AirIO

  committed
  debc14b

  Copy full SHA for debc14b

Commits on Apr 23, 2025

• Fix AirIO hashing in Mixtures. hash() is not stable across processes.

  Show description for 167cd61
  

  timshen91

  authored and

  AirIO

  committed
  167cd61

  Copy full SHA for 167cd61

Commits on Apr 18, 2025

• Validate that all leaf tasks in AirIO mixtures are unique and update the ordering of `leaf_tasks` based on insertion order rather than task name.

  Show description for e6451bf
  

  AirIO Team

  authored and

  AirIO

  committed
  e6451bf

  Copy full SHA for e6451bf

• Fix airio benchmark tests.

  Show description for 059560a
  

  gauravmishra

  authored and

  AirIO

  committed
  059560a

  Copy full SHA for 059560a

Commits on Apr 10, 2025

• Configure tests to explicitly use jax_threefry_partitionable=False.

  Show description for 751e506
  

  jburnim

  authored and

  AirIO

  committed
  751e506

  Copy full SHA for 751e506

Commits on Mar 4, 2025

• Migrate `tf.data.experimental.RandomDataset` to `tf.data.Dataset.random`.

  Show description for c6531b3
  

  yangustc07

  authored and

  AirIO

  committed
  c6531b3

  Copy full SHA for c6531b3

Commits on Oct 18, 2024

• Introduces DatasetIterator._parents and DatasetIterator._parent.

  Show description for 37109ff
  

  iindyk

  authored and

  AirIO

  committed
  37109ff

  Copy full SHA for 37109ff

Commits on Aug 19, 2024

• Improve error propagation.

  Show description for d08752e
  

  iindyk

  authored and

  AirIO

  committed
  d08752e

  Copy full SHA for d08752e

Commits on Aug 6, 2024

• Switch to graduated grain.Dataset APIs.

  Show description for 815f40e
  

  iindyk

  authored and

  AirIO

  committed
  815f40e

  Copy full SHA for 815f40e

Commits on Jul 3, 2024

• Publicly expose UnigramVocabulary

  Show description for 994e213
  

  AirIO Team

  authored and

  AirIO

  committed
  994e213

  Copy full SHA for 994e213

Commits on Jul 2, 2024

• No public description

  Show description for a456edd
  

  AirIO Team

  authored and

  AirIO

  committed
  a456edd

  Copy full SHA for a456edd

Commits on Jun 28, 2024

• Add batching support to t5 denoise preprocessor.

  Show description for cbeb815
  

  gauravmishra

  authored and

  AirIO

  committed
  cbeb815

  Copy full SHA for cbeb815

Commits on Jun 20, 2024

• Fix `ataset_providers_test.py`.

  Show description for 2c48ef0
  

  AirIO Team

  authored and

  AirIO

  committed
  2c48ef0

  Copy full SHA for 2c48ef0

Commits on Jun 18, 2024

• Improve batching error messages.

  Show description for e7620ba
  

  Marvin182

  authored and

  AirIO

  committed
  e7620ba

  Copy full SHA for e7620ba

Commits on Jun 13, 2024

• Add a UnigramVocabulary

  Show description for 66721d6
  

  AirIO Team

  authored and

  AirIO

  committed
  66721d6

  Copy full SHA for 66721d6

Commits on May 16, 2024

• Replace deprecated `jax.tree_*` functions with `jax.tree.*`

  Show description for 2582307
  

  Jake VanderPlas

  authored and

  AirIO

  committed
  2582307

  Copy full SHA for 2582307

Commits on Apr 24, 2024

• Fix infinite repeat for GrainTask.get_lazy_datasets.

  Show description for d96858e
  

  gauravmishra

  authored and

  AirIO

  committed
  d96858e

  Copy full SHA for d96858e

Commits on Apr 22, 2024

• Fix a bug in Task.get_dataset() when using lazy_dataset and preprocessors that have update_runtime_args.

  Show description for 36a6650
  

  gauravmishra

  authored and

  AirIO

  committed
  36a6650

  Copy full SHA for 36a6650

• Add eos support in airio-pygrain tokenizer.

  Show description for 3af8663
  

  gauravmishra

  authored and

  AirIO

  committed
  3af8663

  Copy full SHA for 3af8663

Commits on Apr 19, 2024

• Fix post-mix preprocessing for AirIO-PyGrain Mixtures.

  Show description for f0fe9e0
  

  gauravmishra

  authored and

  AirIO

  committed
  f0fe9e0

  Copy full SHA for f0fe9e0

Commits on Apr 18, 2024

• Convert tensors to numpy at the end of span_corruption preprocessor in AirIO-PyGrain

  Show description for 133d00a
  

  gauravmishra

  authored and

  AirIO

  committed
  133d00a

  Copy full SHA for 133d00a

Commits on Apr 17, 2024

• internal

  Show description for 7401d08
  

  gauravmishra

  authored and

  AirIO

  committed
  7401d08

  Copy full SHA for 7401d08

Commits on Apr 16, 2024

• Add tests to load and parse tf.Examples from files with airio pygrain datasources.

  Show description for a296623
  

  gauravmishra

  authored and

  AirIO

  committed
  a296623

  Copy full SHA for a296623

Commits on Apr 15, 2024

• remove with_eos from Tokenizer class.

  Show description for 0cf7619
  

  gauravmishra

  authored and

  AirIO

  committed
  0cf7619

  Copy full SHA for 0cf7619

Commits on Apr 11, 2024

• Update test data to use tf.train.Example protos.

  Show description for dee4d99
  

  gauravmishra

  authored and

  AirIO

  committed
  dee4d99

  Copy full SHA for dee4d99

Commits on Apr 9, 2024

• Remove extra features before packing.

  Show description for 2f93084
  

  gauravmishra

  authored and

  AirIO

  committed
  2f93084

  Copy full SHA for 2f93084

Commits on Apr 5, 2024

• Merge pull request #226 from gspschmid/gschmid/bump-grain-007

  Show description for cfca4a1
  texasmichelle

  authored
  cfca4a1

  Copy full SHA for cfca4a1

• Merge branch 'main' into gschmid/bump-grain-007

  texasmichelle

  authored
  267efd5

  Copy full SHA for 267efd5