IterableDataset raises FileNotFoundError instead of retrying

[bauwenst]

Describe the bug

In #6843 it was noted that the streaming feature of datasets is highly susceptible to outages and doesn't back off for long (or even at all).

I was training a model while streaming SlimPajama and training crashed with a FileNotFoundError. I can only assume that this was due to a momentary outage considering the file in question, train/chunk9/example_train_3889.jsonl.zst, exists like all other files in SlimPajama.

...
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 2226, in __iter__
    for key, example in ex_iterable:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1499, in __iter__
    for x in self.ex_iterable:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1067, in __iter__
    yield from self._iter()
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1231, in _iter
    for key, transformed_example in iter_outputs():
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1207, in iter_outputs
    for i, key_example in inputs_iterator:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1111, in iter_inputs
    for key, example in iterator:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 371, in __iter__
    for key, pa_table in self.generate_tables_fn(**gen_kwags):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/packaged_modules/json/json.py", line 99, in _generate_tables
    for file_idx, file in enumerate(itertools.chain.from_iterable(files)):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/track.py", line 50, in __iter__
    for x in self.generator(*self.args):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 1378, in _iter_from_urlpaths
    raise FileNotFoundError(urlpath)
FileNotFoundError: zstd://example_train_3889.jsonl::hf://datasets/cerebras/SlimPajama-627B@2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk9/example_train_3889.jsonl.zst

That final raise is at the bottom of the following snippet:

datasets/src/datasets/utils/file_utils.py

Lines 1354 to 1379 in f693f4e

	class FilesIterable(TrackedIterableFromGenerator):
	"""An iterable of paths from a list of directories or files"""
	@classmethod
	def _iter_from_urlpaths(
	cls, urlpaths: Union[str, list[str]], download_config: Optional[DownloadConfig] = None
	) -> Generator[str, None, None]:
	if not isinstance(urlpaths, list):
	urlpaths = [urlpaths]
	for urlpath in urlpaths:
	if xisfile(urlpath, download_config=download_config):
	yield urlpath
	elif xisdir(urlpath, download_config=download_config):
	for dirpath, dirnames, filenames in xwalk(urlpath, download_config=download_config):
	# in-place modification to prune the search
	dirnames[:] = sorted([dirname for dirname in dirnames if not dirname.startswith((".", "__"))])
	if xbasename(dirpath).startswith((".", "__")):
	# skipping hidden directories
	continue
	for filename in sorted(filenames):
	if filename.startswith((".", "__")):
	# skipping hidden files
	continue
	yield xjoin(dirpath, filename)
	else:
	raise FileNotFoundError(urlpath)

So clearly, something choked up in xisfile.

Steps to reproduce the bug

This happens when streaming a dataset and iterating over it. In my case, that iteration is done in Trainer's inner_training_loop, but this is not relevant to the iterator.

  File "/miniconda3/envs/draft/lib/python3.11/site-packages/accelerate/data_loader.py", line 835, in __iter__
    next_batch, next_batch_info = self._fetch_batches(main_iterator)

Expected behavior

This bug and the linked issue have one thing in common: when streaming fails to retrieve an example, the entire program gives up and crashes. As users, we cannot even protect ourselves from this: when we are iterating over a dataset, we can't make datasets skip over a bad example or wait a little longer to retry the iteration, because when a Python generator/iterator raises an error, it loses all its context.

In other words: if you have something that looks like for b in a: for c in b: for d in c:, errors in the innermost loop can only be caught by a try ... except in c.__iter__(). There should be such exception handling in datasets and it should have a configurable exponential back-off: first wait and retry after 1 minute, then 2 minutes, then 4 minutes, then 8 minutes, ... and after a given amount of retries, skip the bad example, and only after skipping a given amount of examples, give up and crash. This was requested in #6843 too, since currently there is only linear backoff and it is clearly not applied to xisfile.

Environment info

• datasets version: 3.3.2 (the latest version)
• Platform: Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28
• Python version: 3.11.7
• huggingface_hub version: 0.26.5
• PyArrow version: 15.0.0
• Pandas version: 2.2.0
• fsspec version: 2024.10.0

[bauwenst]

I have since been training more models with identical architectures over the same dataset, and it is completely unstable. One has now failed at chunk9/1215, whilst others have gotten past that.

FileNotFoundError: zstd://example_train_1215.jsonl::hf://datasets/cerebras/SlimPajama-627B@2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk9/example_train_1215.jsonl.zst

Below is the full training log, where you can clearly see the intermittent dataset issues. Note again that this model only got to epoch 0.11, whereas I have other models training on the exact same dataset right now that have gotten way beyond that. This is quickly turning into a highly expensive bug which I didn't have issues with in the past half year of using the same setup.

Training log of failed run

  1%|          | 64/8192 [56:27<87:25:33, 38.72s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 5ef28452-e903-4bd8-946d-f0c77f558a2a)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk5/example_holdout_4799.jsonl.zst
  1%|          | 64/8192 [56:51<87:25:33, 38.72s/it]Retrying in 1s [Retry 1/5].
  2%|▏         | 192/8192 [2:40:14<85:29:44, 38.47s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: ba6e4c51-f4a4-407e-9934-3772550b7ce9)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk1/example_holdout_2770.jsonl.zst
  2%|▏         | 192/8192 [2:40:53<85:29:44, 38.47s/it]Retrying in 1s [Retry 1/5].
  2%|▏         | 192/8192 [2:40:53<85:29:44, 38.47s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: bdf2cfaa-7e0b-46a0-bec1-b1e573fa7998)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_4386.jsonl.zst
  2%|▏         | 192/8192 [2:42:16<85:29:44, 38.47s/it]Retrying in 1s [Retry 1/5].
  2%|▏         | 192/8192 [2:42:16<85:29:44, 38.47s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 1dc5e455-8042-4c7b-9b97-5ded33dfea34)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk1/example_holdout_1763.jsonl.zst
  2%|▏         | 192/8192 [2:42:30<85:29:44, 38.47s/it]Retrying in 1s [Retry 1/5].
  2%|▏         | 192/8192 [2:42:30<85:29:44, 38.47s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 9cf29917-8111-41fe-80aa-953df65c5803)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_5509.jsonl.zst
  2%|▏         | 192/8192 [2:44:31<85:29:44, 38.47s/it]Retrying in 1s [Retry 1/5].
  2%|▏         | 192/8192 [2:44:31<85:29:44, 38.47s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 2515a0b0-3d81-409f-940c-e78ed5e2dbf8)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_3093.jsonl.zst
  2%|▏         | 192/8192 [2:45:13<85:29:44, 38.47s/it]Retrying in 1s [Retry 1/5].
  2%|▏         | 192/8192 [2:45:13<85:29:44, 38.47s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: a4c1e0c7-1c7a-4377-bc7e-6f076473072b)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_3422.jsonl.zst
  2%|▏         | 192/8192 [2:46:26<85:29:44, 38.47s/it]Retrying in 1s [Retry 1/5].
  2%|▏         | 192/8192 [2:46:26<85:29:44, 38.47s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: c7b0d366-db86-4d0c-a4e0-be251d26519e)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_2250.jsonl.zst
  2%|▏         | 192/8192 [2:47:24<85:29:44, 38.47s/it]Retrying in 1s [Retry 1/5].
  2%|▏         | 192/8192 [2:47:24<85:29:44, 38.47s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: b0df5a1a-4836-46cf-8e45-58a7c1553309)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_6161.jsonl.zst
  2%|▏         | 192/8192 [2:49:10<85:29:44, 38.47s/it]Retrying in 1s [Retry 1/5].
  2%|▏         | 192/8192 [2:49:10<85:29:44, 38.47s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: c1d97368-c0ae-45bb-ae10-5559b3ebc4e4)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_5782.jsonl.zst
  2%|▏         | 192/8192 [2:49:30<85:29:44, 38.47s/it]Retrying in 1s [Retry 1/5].
{'eval_loss': 10.482319831848145, 'eval_runtime': 902.7516, 'eval_samples_per_second': 18.149, 'eval_steps_per_second': 0.142, 'epoch': 0, 'num_input_tokens_seen': 0}
{'loss': 10.4895, 'grad_norm': 2.9147818088531494, 'learning_rate': 3.90625e-06, 'epoch': 0.0, 'num_input_tokens_seen': 1048576}
{'loss': 10.4832, 'grad_norm': 2.8206892013549805, 'learning_rate': 7.8125e-06, 'epoch': 0.0, 'num_input_tokens_seen': 2097152}
{'loss': 10.4851, 'grad_norm': 2.910552978515625, 'learning_rate': 1.171875e-05, 'epoch': 0.0, 'num_input_tokens_seen': 3145728}
{'loss': 10.486, 'grad_norm': 2.8042073249816895, 'learning_rate': 1.5625e-05, 'epoch': 0.0, 'num_input_tokens_seen': 4194304}
{'loss': 10.4719, 'grad_norm': 2.83260440826416, 'learning_rate': 1.953125e-05, 'epoch': 0.0, 'num_input_tokens_seen': 5242880}
{'loss': 10.4482, 'grad_norm': 2.916527032852173, 'learning_rate': 2.34375e-05, 'epoch': 0.0, 'num_input_tokens_seen': 6291456}
{'loss': 10.4113, 'grad_norm': 2.911870241165161, 'learning_rate': 2.734375e-05, 'epoch': 0.0, 'num_input_tokens_seen': 7340032}
{'loss': 10.3863, 'grad_norm': 2.8873367309570312, 'learning_rate': 3.125e-05, 'epoch': 0.0, 'num_input_tokens_seen': 8388608}
{'loss': 10.3557, 'grad_norm': 2.7183432579040527, 'learning_rate': 3.5156250000000004e-05, 'epoch': 0.0, 'num_input_tokens_seen': 9437184}
{'loss': 10.2795, 'grad_norm': 2.6743927001953125, 'learning_rate': 3.90625e-05, 'epoch': 0.0, 'num_input_tokens_seen': 10485760}
{'loss': 10.2148, 'grad_norm': 2.3173940181732178, 'learning_rate': 4.296875e-05, 'epoch': 0.0, 'num_input_tokens_seen': 11534336}
{'loss': 10.1482, 'grad_norm': 2.09787917137146, 'learning_rate': 4.6875e-05, 'epoch': 0.0, 'num_input_tokens_seen': 12582912}
{'loss': 10.1024, 'grad_norm': 1.889390468597412, 'learning_rate': 5.0781250000000004e-05, 'epoch': 0.0, 'num_input_tokens_seen': 13631488}
{'loss': 10.0418, 'grad_norm': 1.8319090604782104, 'learning_rate': 5.46875e-05, 'epoch': 0.0, 'num_input_tokens_seen': 14680064}
{'loss': 10.0081, 'grad_norm': 1.7302652597427368, 'learning_rate': 5.859375e-05, 'epoch': 0.0, 'num_input_tokens_seen': 15728640}
{'loss': 9.9525, 'grad_norm': 1.767600417137146, 'learning_rate': 6.25e-05, 'epoch': 0.0, 'num_input_tokens_seen': 16777216}
{'loss': 9.9326, 'grad_norm': 2.1608240604400635, 'learning_rate': 6.640625e-05, 'epoch': 0.0, 'num_input_tokens_seen': 17825792}
{'loss': 9.8478, 'grad_norm': 1.7399269342422485, 'learning_rate': 7.031250000000001e-05, 'epoch': 0.0, 'num_input_tokens_seen': 18874368}
{'loss': 9.8215, 'grad_norm': 1.6564425230026245, 'learning_rate': 7.421875e-05, 'epoch': 0.0, 'num_input_tokens_seen': 19922944}
{'loss': 9.7732, 'grad_norm': 1.6452653408050537, 'learning_rate': 7.8125e-05, 'epoch': 0.0, 'num_input_tokens_seen': 20971520}
{'loss': 9.6896, 'grad_norm': 1.7053238153457642, 'learning_rate': 8.203125e-05, 'epoch': 0.0, 'num_input_tokens_seen': 22020096}
{'loss': 9.6356, 'grad_norm': 1.7050201892852783, 'learning_rate': 8.59375e-05, 'epoch': 0.0, 'num_input_tokens_seen': 23068672}
{'loss': 9.5781, 'grad_norm': 1.7155998945236206, 'learning_rate': 8.984375e-05, 'epoch': 0.0, 'num_input_tokens_seen': 24117248}
{'loss': 9.5355, 'grad_norm': 1.697864294052124, 'learning_rate': 9.375e-05, 'epoch': 0.0, 'num_input_tokens_seen': 25165824}
{'loss': 9.4718, 'grad_norm': 1.7598071098327637, 'learning_rate': 9.765625e-05, 'epoch': 0.0, 'num_input_tokens_seen': 26214400}
{'loss': 9.3972, 'grad_norm': 1.7407673597335815, 'learning_rate': 0.00010156250000000001, 'epoch': 0.0, 'num_input_tokens_seen': 27262976}
{'loss': 9.3303, 'grad_norm': 1.7710134983062744, 'learning_rate': 0.00010546875, 'epoch': 0.0, 'num_input_tokens_seen': 28311552}
{'loss': 9.2973, 'grad_norm': 1.716180682182312, 'learning_rate': 0.000109375, 'epoch': 0.0, 'num_input_tokens_seen': 29360128}
{'loss': 9.2049, 'grad_norm': 1.7579947710037231, 'learning_rate': 0.00011328125, 'epoch': 0.0, 'num_input_tokens_seen': 30408704}
{'loss': 9.1656, 'grad_norm': 1.6988558769226074, 'learning_rate': 0.0001171875, 'epoch': 0.0, 'num_input_tokens_seen': 31457280}
{'loss': 9.0966, 'grad_norm': 1.7036350965499878, 'learning_rate': 0.00012109375, 'epoch': 0.0, 'num_input_tokens_seen': 32505856}
{'loss': 9.0107, 'grad_norm': 1.752451777458191, 'learning_rate': 0.000125, 'epoch': 0.0, 'num_input_tokens_seen': 33554432}
{'loss': 8.9788, 'grad_norm': 1.6769776344299316, 'learning_rate': 0.00012890625, 'epoch': 0.0, 'num_input_tokens_seen': 34603008}
{'loss': 8.9155, 'grad_norm': 1.6497987508773804, 'learning_rate': 0.0001328125, 'epoch': 0.0, 'num_input_tokens_seen': 35651584}
{'loss': 8.8008, 'grad_norm': 1.722798466682434, 'learning_rate': 0.00013671875, 'epoch': 0.0, 'num_input_tokens_seen': 36700160}
{'loss': 8.7727, 'grad_norm': 1.6046854257583618, 'learning_rate': 0.00014062500000000002, 'epoch': 0.0, 'num_input_tokens_seen': 37748736}
{'loss': 8.682, 'grad_norm': 1.6132164001464844, 'learning_rate': 0.00014453125, 'epoch': 0.0, 'num_input_tokens_seen': 38797312}
{'loss': 8.6516, 'grad_norm': 1.558968424797058, 'learning_rate': 0.0001484375, 'epoch': 0.0, 'num_input_tokens_seen': 39845888}
{'loss': 8.5935, 'grad_norm': 1.6083673238754272, 'learning_rate': 0.00015234375, 'epoch': 0.0, 'num_input_tokens_seen': 40894464}
{'loss': 8.4852, 'grad_norm': 1.5469273328781128, 'learning_rate': 0.00015625, 'epoch': 0.0, 'num_input_tokens_seen': 41943040}
{'loss': 8.4342, 'grad_norm': 1.46219801902771, 'learning_rate': 0.00016015625, 'epoch': 0.01, 'num_input_tokens_seen': 42991616}
{'loss': 8.3213, 'grad_norm': 1.473191261291504, 'learning_rate': 0.0001640625, 'epoch': 0.01, 'num_input_tokens_seen': 44040192}
{'loss': 8.3193, 'grad_norm': 1.4024137258529663, 'learning_rate': 0.00016796875000000001, 'epoch': 0.01, 'num_input_tokens_seen': 45088768}
{'loss': 8.1853, 'grad_norm': 1.3591463565826416, 'learning_rate': 0.000171875, 'epoch': 0.01, 'num_input_tokens_seen': 46137344}
{'loss': 8.1109, 'grad_norm': 1.3547109365463257, 'learning_rate': 0.00017578125, 'epoch': 0.01, 'num_input_tokens_seen': 47185920}
{'loss': 8.0741, 'grad_norm': 1.268977403640747, 'learning_rate': 0.0001796875, 'epoch': 0.01, 'num_input_tokens_seen': 48234496}
{'loss': 8.0032, 'grad_norm': 1.222671389579773, 'learning_rate': 0.00018359375, 'epoch': 0.01, 'num_input_tokens_seen': 49283072}
{'loss': 7.9346, 'grad_norm': 1.154278039932251, 'learning_rate': 0.0001875, 'epoch': 0.01, 'num_input_tokens_seen': 50331648}
{'loss': 7.8823, 'grad_norm': 1.1396397352218628, 'learning_rate': 0.00019140625, 'epoch': 0.01, 'num_input_tokens_seen': 51380224}
{'loss': 7.8444, 'grad_norm': 1.0608373880386353, 'learning_rate': 0.0001953125, 'epoch': 0.01, 'num_input_tokens_seen': 52428800}
{'loss': 7.7794, 'grad_norm': 1.0165436267852783, 'learning_rate': 0.00019921875000000001, 'epoch': 0.01, 'num_input_tokens_seen': 53477376}
{'loss': 7.7567, 'grad_norm': 0.8742461204528809, 'learning_rate': 0.00020312500000000002, 'epoch': 0.01, 'num_input_tokens_seen': 54525952}
{'loss': 7.6489, 'grad_norm': 0.8699902296066284, 'learning_rate': 0.00020703125, 'epoch': 0.01, 'num_input_tokens_seen': 55574528}
{'loss': 7.6062, 'grad_norm': 0.809831440448761, 'learning_rate': 0.0002109375, 'epoch': 0.01, 'num_input_tokens_seen': 56623104}
{'loss': 7.5511, 'grad_norm': 0.7423847317695618, 'learning_rate': 0.00021484375, 'epoch': 0.01, 'num_input_tokens_seen': 57671680}
{'loss': 7.4435, 'grad_norm': 0.7614696025848389, 'learning_rate': 0.00021875, 'epoch': 0.01, 'num_input_tokens_seen': 58720256}
{'loss': 7.564, 'grad_norm': 0.5147746801376343, 'learning_rate': 0.00022265625, 'epoch': 0.01, 'num_input_tokens_seen': 59768832}
{'loss': 7.5278, 'grad_norm': 0.4705545902252197, 'learning_rate': 0.0002265625, 'epoch': 0.01, 'num_input_tokens_seen': 60817408}
{'loss': 7.5479, 'grad_norm': 0.3745419979095459, 'learning_rate': 0.00023046875000000001, 'epoch': 0.01, 'num_input_tokens_seen': 61865984}
{'loss': 7.4759, 'grad_norm': 0.3893500566482544, 'learning_rate': 0.000234375, 'epoch': 0.01, 'num_input_tokens_seen': 62914560}
{'loss': 7.5032, 'grad_norm': 0.31959569454193115, 'learning_rate': 0.00023828125, 'epoch': 0.01, 'num_input_tokens_seen': 63963136}
{'loss': 7.421, 'grad_norm': 0.3203206956386566, 'learning_rate': 0.0002421875, 'epoch': 0.01, 'num_input_tokens_seen': 65011712}
{'loss': 7.4998, 'grad_norm': 0.2730390429496765, 'learning_rate': 0.00024609375, 'epoch': 0.01, 'num_input_tokens_seen': 66060288}
{'loss': 7.4157, 'grad_norm': 0.34872403740882874, 'learning_rate': 0.00025, 'epoch': 0.01, 'num_input_tokens_seen': 67108864}
[2025-03-10 16:17:04 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 5ef28452-e903-4bd8-946d-f0c77f558a2a)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk5/example_holdout_4799.jsonl.zst
[2025-03-10 16:17:04 WARNING] Retrying in 1s [Retry 1/5].
{'eval_loss': 7.471163749694824, 'eval_runtime': 651.4801, 'eval_samples_per_second': 25.149, 'eval_steps_per_second': 0.196, 'epoch': 0.01, 'num_input_tokens_seen': 67108864}
{'loss': 7.5083, 'grad_norm': 0.339502215385437, 'learning_rate': 0.00025390625, 'epoch': 0.01, 'num_input_tokens_seen': 68157440}
{'loss': 7.7083, 'grad_norm': 0.6426190137863159, 'learning_rate': 0.0002578125, 'epoch': 0.01, 'num_input_tokens_seen': 69206016}
{'loss': 7.446, 'grad_norm': 0.9138129353523254, 'learning_rate': 0.00026171875, 'epoch': 0.01, 'num_input_tokens_seen': 70254592}
{'loss': 7.3747, 'grad_norm': 1.2179911136627197, 'learning_rate': 0.000265625, 'epoch': 0.01, 'num_input_tokens_seen': 71303168}
{'loss': 7.367, 'grad_norm': 0.7108445167541504, 'learning_rate': 0.00026953125, 'epoch': 0.01, 'num_input_tokens_seen': 72351744}
{'loss': 7.4751, 'grad_norm': 0.7580183744430542, 'learning_rate': 0.0002734375, 'epoch': 0.01, 'num_input_tokens_seen': 73400320}
{'loss': 7.3405, 'grad_norm': 0.7545790076255798, 'learning_rate': 0.00027734375000000003, 'epoch': 0.01, 'num_input_tokens_seen': 74448896}
{'loss': 7.4194, 'grad_norm': 0.4764443039894104, 'learning_rate': 0.00028125000000000003, 'epoch': 0.01, 'num_input_tokens_seen': 75497472}
{'loss': 7.2826, 'grad_norm': 0.5942808985710144, 'learning_rate': 0.00028515625, 'epoch': 0.01, 'num_input_tokens_seen': 76546048}
{'loss': 7.3945, 'grad_norm': 0.5272891521453857, 'learning_rate': 0.0002890625, 'epoch': 0.01, 'num_input_tokens_seen': 77594624}
{'loss': 7.3492, 'grad_norm': 0.465085506439209, 'learning_rate': 0.00029296875, 'epoch': 0.01, 'num_input_tokens_seen': 78643200}
{'loss': 7.3658, 'grad_norm': 0.6932719349861145, 'learning_rate': 0.000296875, 'epoch': 0.01, 'num_input_tokens_seen': 79691776}
{'loss': 7.3554, 'grad_norm': 0.49396172165870667, 'learning_rate': 0.00030078125, 'epoch': 0.01, 'num_input_tokens_seen': 80740352}
{'loss': 7.2916, 'grad_norm': 0.3178255558013916, 'learning_rate': 0.0003046875, 'epoch': 0.01, 'num_input_tokens_seen': 81788928}
{'loss': 7.2871, 'grad_norm': 0.5465154647827148, 'learning_rate': 0.00030859375, 'epoch': 0.01, 'num_input_tokens_seen': 82837504}
{'loss': 7.262, 'grad_norm': 0.4718130826950073, 'learning_rate': 0.0003125, 'epoch': 0.01, 'num_input_tokens_seen': 83886080}
{'loss': 7.2845, 'grad_norm': 0.5033366680145264, 'learning_rate': 0.00031640625, 'epoch': 0.01, 'num_input_tokens_seen': 84934656}
{'loss': 7.2525, 'grad_norm': 0.5601146817207336, 'learning_rate': 0.0003203125, 'epoch': 0.01, 'num_input_tokens_seen': 85983232}
{'loss': 7.1971, 'grad_norm': 0.5764456987380981, 'learning_rate': 0.00032421875, 'epoch': 0.01, 'num_input_tokens_seen': 87031808}
{'loss': 7.1988, 'grad_norm': 0.6154745817184448, 'learning_rate': 0.000328125, 'epoch': 0.01, 'num_input_tokens_seen': 88080384}
{'loss': 7.1987, 'grad_norm': 0.6701765656471252, 'learning_rate': 0.00033203125, 'epoch': 0.01, 'num_input_tokens_seen': 89128960}
{'loss': 7.3324, 'grad_norm': 0.5648972988128662, 'learning_rate': 0.00033593750000000003, 'epoch': 0.01, 'num_input_tokens_seen': 90177536}
{'loss': 7.2233, 'grad_norm': 0.5782461166381836, 'learning_rate': 0.00033984375000000003, 'epoch': 0.01, 'num_input_tokens_seen': 91226112}
{'loss': 7.1995, 'grad_norm': 0.540762722492218, 'learning_rate': 0.00034375, 'epoch': 0.01, 'num_input_tokens_seen': 92274688}
{'loss': 7.1214, 'grad_norm': 0.9524508118629456, 'learning_rate': 0.00034765625, 'epoch': 0.01, 'num_input_tokens_seen': 93323264}
{'loss': 7.1603, 'grad_norm': 1.4820659160614014, 'learning_rate': 0.0003515625, 'epoch': 0.01, 'num_input_tokens_seen': 94371840}
{'loss': 7.2364, 'grad_norm': 0.6124428510665894, 'learning_rate': 0.00035546875, 'epoch': 0.01, 'num_input_tokens_seen': 95420416}
{'loss': 7.0258, 'grad_norm': 0.8897235989570618, 'learning_rate': 0.000359375, 'epoch': 0.01, 'num_input_tokens_seen': 96468992}
{'loss': 7.1182, 'grad_norm': 0.9263321757316589, 'learning_rate': 0.00036328125, 'epoch': 0.01, 'num_input_tokens_seen': 97517568}
{'loss': 7.109, 'grad_norm': 0.5800505876541138, 'learning_rate': 0.0003671875, 'epoch': 0.01, 'num_input_tokens_seen': 98566144}
{'loss': 7.0449, 'grad_norm': 0.6776424050331116, 'learning_rate': 0.00037109375, 'epoch': 0.01, 'num_input_tokens_seen': 99614720}
{'loss': 7.1272, 'grad_norm': 0.7616431713104248, 'learning_rate': 0.000375, 'epoch': 0.01, 'num_input_tokens_seen': 100663296}
{'loss': 7.046, 'grad_norm': 0.5346249938011169, 'learning_rate': 0.00037890625, 'epoch': 0.01, 'num_input_tokens_seen': 101711872}
{'loss': 7.0713, 'grad_norm': 0.6108944416046143, 'learning_rate': 0.0003828125, 'epoch': 0.01, 'num_input_tokens_seen': 102760448}
{'loss': 7.1459, 'grad_norm': 0.4430749714374542, 'learning_rate': 0.00038671875, 'epoch': 0.01, 'num_input_tokens_seen': 103809024}
{'loss': 7.0709, 'grad_norm': 0.6020255088806152, 'learning_rate': 0.000390625, 'epoch': 0.01, 'num_input_tokens_seen': 104857600}
{'loss': 7.0144, 'grad_norm': 0.5525627732276917, 'learning_rate': 0.00039453125, 'epoch': 0.01, 'num_input_tokens_seen': 105906176}
{'loss': 7.0926, 'grad_norm': 0.6909684538841248, 'learning_rate': 0.00039843750000000003, 'epoch': 0.01, 'num_input_tokens_seen': 106954752}
{'loss': 7.0289, 'grad_norm': 0.5576740503311157, 'learning_rate': 0.00040234375000000003, 'epoch': 0.01, 'num_input_tokens_seen': 108003328}
{'loss': 6.9173, 'grad_norm': 0.48874178528785706, 'learning_rate': 0.00040625000000000004, 'epoch': 0.01, 'num_input_tokens_seen': 109051904}
{'loss': 6.9777, 'grad_norm': 0.3904782831668854, 'learning_rate': 0.00041015625, 'epoch': 0.01, 'num_input_tokens_seen': 110100480}
{'loss': 6.9473, 'grad_norm': 0.3953755795955658, 'learning_rate': 0.0004140625, 'epoch': 0.01, 'num_input_tokens_seen': 111149056}
{'loss': 6.9071, 'grad_norm': 0.43107134103775024, 'learning_rate': 0.00041796875, 'epoch': 0.01, 'num_input_tokens_seen': 112197632}
{'loss': 6.9277, 'grad_norm': 0.33989447355270386, 'learning_rate': 0.000421875, 'epoch': 0.01, 'num_input_tokens_seen': 113246208}
{'loss': 6.914, 'grad_norm': 0.3267095983028412, 'learning_rate': 0.00042578125, 'epoch': 0.01, 'num_input_tokens_seen': 114294784}
{'loss': 6.6865, 'grad_norm': 0.4201946556568146, 'learning_rate': 0.0004296875, 'epoch': 0.01, 'num_input_tokens_seen': 115343360}
{'loss': 6.8229, 'grad_norm': 0.345426082611084, 'learning_rate': 0.00043359375, 'epoch': 0.01, 'num_input_tokens_seen': 116391936}
{'loss': 6.8599, 'grad_norm': 0.4104400873184204, 'learning_rate': 0.0004375, 'epoch': 0.01, 'num_input_tokens_seen': 117440512}
{'loss': 6.7656, 'grad_norm': 0.6487549543380737, 'learning_rate': 0.00044140625, 'epoch': 0.01, 'num_input_tokens_seen': 118489088}
{'loss': 6.8654, 'grad_norm': 1.5497283935546875, 'learning_rate': 0.0004453125, 'epoch': 0.01, 'num_input_tokens_seen': 119537664}
{'loss': 6.8207, 'grad_norm': 1.9772824048995972, 'learning_rate': 0.00044921875, 'epoch': 0.01, 'num_input_tokens_seen': 120586240}
{'loss': 6.7802, 'grad_norm': 0.9341455101966858, 'learning_rate': 0.000453125, 'epoch': 0.01, 'num_input_tokens_seen': 121634816}
{'loss': 6.8017, 'grad_norm': 1.3528856039047241, 'learning_rate': 0.00045703125, 'epoch': 0.01, 'num_input_tokens_seen': 122683392}
{'loss': 6.8344, 'grad_norm': 0.5852281451225281, 'learning_rate': 0.00046093750000000003, 'epoch': 0.01, 'num_input_tokens_seen': 123731968}
{'loss': 6.8259, 'grad_norm': 0.9776580929756165, 'learning_rate': 0.00046484375000000003, 'epoch': 0.01, 'num_input_tokens_seen': 124780544}
{'loss': 6.7581, 'grad_norm': 1.0398296117782593, 'learning_rate': 0.00046875, 'epoch': 0.01, 'num_input_tokens_seen': 125829120}
{'loss': 6.7795, 'grad_norm': 1.1206268072128296, 'learning_rate': 0.00047265625, 'epoch': 0.01, 'num_input_tokens_seen': 126877696}
{'loss': 6.5667, 'grad_norm': 0.6790318489074707, 'learning_rate': 0.0004765625, 'epoch': 0.01, 'num_input_tokens_seen': 127926272}
{'loss': 6.7297, 'grad_norm': 1.2275055646896362, 'learning_rate': 0.00048046875, 'epoch': 0.02, 'num_input_tokens_seen': 128974848}
{'loss': 6.7104, 'grad_norm': 1.1354466676712036, 'learning_rate': 0.000484375, 'epoch': 0.02, 'num_input_tokens_seen': 130023424}
{'loss': 6.7025, 'grad_norm': 0.9035728573799133, 'learning_rate': 0.00048828125, 'epoch': 0.02, 'num_input_tokens_seen': 131072000}
{'loss': 6.6391, 'grad_norm': 1.3942680358886719, 'learning_rate': 0.0004921875, 'epoch': 0.02, 'num_input_tokens_seen': 132120576}
{'loss': 6.6011, 'grad_norm': 0.7435236573219299, 'learning_rate': 0.00049609375, 'epoch': 0.02, 'num_input_tokens_seen': 133169152}
{'loss': 6.5135, 'grad_norm': 0.5970368385314941, 'learning_rate': 0.0005, 'epoch': 0.02, 'num_input_tokens_seen': 134217728}
{'eval_loss': 6.573822021484375, 'eval_runtime': 629.9441, 'eval_samples_per_second': 26.009, 'eval_steps_per_second': 0.203, 'epoch': 0.02, 'num_input_tokens_seen': 134217728}
{'loss': 6.5509, 'grad_norm': 0.7936264276504517, 'learning_rate': 0.00050390625, 'epoch': 0.02, 'num_input_tokens_seen': 135266304}
{'loss': 6.6008, 'grad_norm': 0.6225885152816772, 'learning_rate': 0.0005078125, 'epoch': 0.02, 'num_input_tokens_seen': 136314880}
{'loss': 6.4821, 'grad_norm': 0.5519376993179321, 'learning_rate': 0.00051171875, 'epoch': 0.02, 'num_input_tokens_seen': 137363456}
{'loss': 6.3411, 'grad_norm': 0.5908603668212891, 'learning_rate': 0.000515625, 'epoch': 0.02, 'num_input_tokens_seen': 138412032}
{'loss': 6.3464, 'grad_norm': 0.5101401209831238, 'learning_rate': 0.00051953125, 'epoch': 0.02, 'num_input_tokens_seen': 139460608}
{'loss': 6.3638, 'grad_norm': 0.7352246046066284, 'learning_rate': 0.0005234375, 'epoch': 0.02, 'num_input_tokens_seen': 140509184}
{'loss': 6.3429, 'grad_norm': 0.49651673436164856, 'learning_rate': 0.00052734375, 'epoch': 0.02, 'num_input_tokens_seen': 141557760}
{'loss': 6.2987, 'grad_norm': 0.4835755527019501, 'learning_rate': 0.00053125, 'epoch': 0.02, 'num_input_tokens_seen': 142606336}
{'loss': 6.2982, 'grad_norm': 0.5940163731575012, 'learning_rate': 0.00053515625, 'epoch': 0.02, 'num_input_tokens_seen': 143654912}
{'loss': 6.267, 'grad_norm': 0.7658674120903015, 'learning_rate': 0.0005390625, 'epoch': 0.02, 'num_input_tokens_seen': 144703488}
{'loss': 6.2102, 'grad_norm': 0.6704416275024414, 'learning_rate': 0.00054296875, 'epoch': 0.02, 'num_input_tokens_seen': 145752064}
{'loss': 6.1956, 'grad_norm': 0.6615312099456787, 'learning_rate': 0.000546875, 'epoch': 0.02, 'num_input_tokens_seen': 146800640}
{'loss': 6.286, 'grad_norm': 0.7957404255867004, 'learning_rate': 0.0005507812500000001, 'epoch': 0.02, 'num_input_tokens_seen': 147849216}
{'loss': 6.2483, 'grad_norm': 0.6477276682853699, 'learning_rate': 0.0005546875000000001, 'epoch': 0.02, 'num_input_tokens_seen': 148897792}
{'loss': 6.0944, 'grad_norm': 0.5753227472305298, 'learning_rate': 0.0005585937500000001, 'epoch': 0.02, 'num_input_tokens_seen': 149946368}
{'loss': 6.0995, 'grad_norm': 0.5871054530143738, 'learning_rate': 0.0005625000000000001, 'epoch': 0.02, 'num_input_tokens_seen': 150994944}
{'loss': 6.112, 'grad_norm': 0.7046136856079102, 'learning_rate': 0.00056640625, 'epoch': 0.02, 'num_input_tokens_seen': 152043520}
{'loss': 6.102, 'grad_norm': 0.9357424378395081, 'learning_rate': 0.0005703125, 'epoch': 0.02, 'num_input_tokens_seen': 153092096}
{'loss': 6.1407, 'grad_norm': 1.0577837228775024, 'learning_rate': 0.00057421875, 'epoch': 0.02, 'num_input_tokens_seen': 154140672}
{'loss': 5.9836, 'grad_norm': 0.7795257568359375, 'learning_rate': 0.000578125, 'epoch': 0.02, 'num_input_tokens_seen': 155189248}
{'loss': 6.1041, 'grad_norm': 0.8117634057998657, 'learning_rate': 0.00058203125, 'epoch': 0.02, 'num_input_tokens_seen': 156237824}
{'loss': 5.9474, 'grad_norm': 0.8311094045639038, 'learning_rate': 0.0005859375, 'epoch': 0.02, 'num_input_tokens_seen': 157286400}
{'loss': 5.9365, 'grad_norm': 0.8269851803779602, 'learning_rate': 0.00058984375, 'epoch': 0.02, 'num_input_tokens_seen': 158334976}
{'loss': 5.9668, 'grad_norm': 0.701510488986969, 'learning_rate': 0.00059375, 'epoch': 0.02, 'num_input_tokens_seen': 159383552}
{'loss': 5.9874, 'grad_norm': 0.49938252568244934, 'learning_rate': 0.00059765625, 'epoch': 0.02, 'num_input_tokens_seen': 160432128}
{'loss': 5.8505, 'grad_norm': 0.6981683969497681, 'learning_rate': 0.0006015625, 'epoch': 0.02, 'num_input_tokens_seen': 161480704}
{'loss': 6.0156, 'grad_norm': 0.5023297071456909, 'learning_rate': 0.00060546875, 'epoch': 0.02, 'num_input_tokens_seen': 162529280}
{'loss': 5.8299, 'grad_norm': 0.6075630187988281, 'learning_rate': 0.000609375, 'epoch': 0.02, 'num_input_tokens_seen': 163577856}
{'loss': 5.8203, 'grad_norm': 0.6051607728004456, 'learning_rate': 0.00061328125, 'epoch': 0.02, 'num_input_tokens_seen': 164626432}
{'loss': 5.7705, 'grad_norm': 0.6384783983230591, 'learning_rate': 0.0006171875, 'epoch': 0.02, 'num_input_tokens_seen': 165675008}
{'loss': 5.791, 'grad_norm': 0.5084705948829651, 'learning_rate': 0.00062109375, 'epoch': 0.02, 'num_input_tokens_seen': 166723584}
{'loss': 5.6743, 'grad_norm': 0.4278322160243988, 'learning_rate': 0.000625, 'epoch': 0.02, 'num_input_tokens_seen': 167772160}
{'loss': 5.7112, 'grad_norm': 0.5151192545890808, 'learning_rate': 0.00062890625, 'epoch': 0.02, 'num_input_tokens_seen': 168820736}
{'loss': 5.5128, 'grad_norm': 0.6542677283287048, 'learning_rate': 0.0006328125, 'epoch': 0.02, 'num_input_tokens_seen': 169869312}
{'loss': 5.6735, 'grad_norm': 0.6016008257865906, 'learning_rate': 0.00063671875, 'epoch': 0.02, 'num_input_tokens_seen': 170917888}
{'loss': 5.6525, 'grad_norm': 0.48695647716522217, 'learning_rate': 0.000640625, 'epoch': 0.02, 'num_input_tokens_seen': 171966464}
{'loss': 5.6051, 'grad_norm': 0.5894989371299744, 'learning_rate': 0.00064453125, 'epoch': 0.02, 'num_input_tokens_seen': 173015040}
{'loss': 5.6377, 'grad_norm': 0.7626883387565613, 'learning_rate': 0.0006484375, 'epoch': 0.02, 'num_input_tokens_seen': 174063616}
{'loss': 5.6038, 'grad_norm': 0.745198130607605, 'learning_rate': 0.00065234375, 'epoch': 0.02, 'num_input_tokens_seen': 175112192}
{'loss': 5.5465, 'grad_norm': 0.7876908779144287, 'learning_rate': 0.00065625, 'epoch': 0.02, 'num_input_tokens_seen': 176160768}
{'loss': 5.5903, 'grad_norm': 0.7416785359382629, 'learning_rate': 0.00066015625, 'epoch': 0.02, 'num_input_tokens_seen': 177209344}
{'loss': 5.4993, 'grad_norm': 0.4493878185749054, 'learning_rate': 0.0006640625, 'epoch': 0.02, 'num_input_tokens_seen': 178257920}
{'loss': 5.5612, 'grad_norm': 0.5095419883728027, 'learning_rate': 0.00066796875, 'epoch': 0.02, 'num_input_tokens_seen': 179306496}
{'loss': 5.378, 'grad_norm': 0.6330733895301819, 'learning_rate': 0.0006718750000000001, 'epoch': 0.02, 'num_input_tokens_seen': 180355072}
{'loss': 5.4875, 'grad_norm': 0.4710595905780792, 'learning_rate': 0.0006757812500000001, 'epoch': 0.02, 'num_input_tokens_seen': 181403648}
{'loss': 5.4221, 'grad_norm': 0.5276287198066711, 'learning_rate': 0.0006796875000000001, 'epoch': 0.02, 'num_input_tokens_seen': 182452224}
{'loss': 5.308, 'grad_norm': 0.6985499858856201, 'learning_rate': 0.0006835937500000001, 'epoch': 0.02, 'num_input_tokens_seen': 183500800}
{'loss': 5.4455, 'grad_norm': 0.4874110519886017, 'learning_rate': 0.0006875, 'epoch': 0.02, 'num_input_tokens_seen': 184549376}
{'loss': 5.476, 'grad_norm': 0.5807638764381409, 'learning_rate': 0.00069140625, 'epoch': 0.02, 'num_input_tokens_seen': 185597952}
{'loss': 5.2876, 'grad_norm': 0.5431288480758667, 'learning_rate': 0.0006953125, 'epoch': 0.02, 'num_input_tokens_seen': 186646528}
{'loss': 5.3881, 'grad_norm': 0.7681945562362671, 'learning_rate': 0.00069921875, 'epoch': 0.02, 'num_input_tokens_seen': 187695104}
{'loss': 5.4006, 'grad_norm': 0.7372023463249207, 'learning_rate': 0.000703125, 'epoch': 0.02, 'num_input_tokens_seen': 188743680}
{'loss': 5.3813, 'grad_norm': 0.7354347109794617, 'learning_rate': 0.00070703125, 'epoch': 0.02, 'num_input_tokens_seen': 189792256}
{'loss': 5.3393, 'grad_norm': 0.5908933281898499, 'learning_rate': 0.0007109375, 'epoch': 0.02, 'num_input_tokens_seen': 190840832}
{'loss': 5.3024, 'grad_norm': 0.5665153861045837, 'learning_rate': 0.00071484375, 'epoch': 0.02, 'num_input_tokens_seen': 191889408}
{'loss': 5.2782, 'grad_norm': 0.5930947661399841, 'learning_rate': 0.00071875, 'epoch': 0.02, 'num_input_tokens_seen': 192937984}
{'loss': 5.3199, 'grad_norm': 0.5926457643508911, 'learning_rate': 0.00072265625, 'epoch': 0.02, 'num_input_tokens_seen': 193986560}
{'loss': 5.2949, 'grad_norm': 0.548610270023346, 'learning_rate': 0.0007265625, 'epoch': 0.02, 'num_input_tokens_seen': 195035136}
{'loss': 5.3143, 'grad_norm': 0.6023995280265808, 'learning_rate': 0.00073046875, 'epoch': 0.02, 'num_input_tokens_seen': 196083712}
{'loss': 5.2982, 'grad_norm': 1.0335254669189453, 'learning_rate': 0.000734375, 'epoch': 0.02, 'num_input_tokens_seen': 197132288}
{'loss': 5.2933, 'grad_norm': 1.2596269845962524, 'learning_rate': 0.00073828125, 'epoch': 0.02, 'num_input_tokens_seen': 198180864}
{'loss': 5.2524, 'grad_norm': 0.6956535577774048, 'learning_rate': 0.0007421875, 'epoch': 0.02, 'num_input_tokens_seen': 199229440}
{'loss': 5.3543, 'grad_norm': 0.946761429309845, 'learning_rate': 0.00074609375, 'epoch': 0.02, 'num_input_tokens_seen': 200278016}
{'loss': 5.1616, 'grad_norm': 0.9568974375724792, 'learning_rate': 0.00075, 'epoch': 0.02, 'num_input_tokens_seen': 201326592}
[2025-03-10 18:01:06 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: ba6e4c51-f4a4-407e-9934-3772550b7ce9)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk1/example_holdout_2770.jsonl.zst
[2025-03-10 18:01:06 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-10 18:02:30 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: bdf2cfaa-7e0b-46a0-bec1-b1e573fa7998)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_4386.jsonl.zst
[2025-03-10 18:02:30 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-10 18:02:44 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 1dc5e455-8042-4c7b-9b97-5ded33dfea34)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk1/example_holdout_1763.jsonl.zst
[2025-03-10 18:02:44 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-10 18:04:45 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 9cf29917-8111-41fe-80aa-953df65c5803)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_5509.jsonl.zst
[2025-03-10 18:04:45 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-10 18:05:26 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 2515a0b0-3d81-409f-940c-e78ed5e2dbf8)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_3093.jsonl.zst
[2025-03-10 18:05:26 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-10 18:06:39 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: a4c1e0c7-1c7a-4377-bc7e-6f076473072b)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_3422.jsonl.zst
[2025-03-10 18:06:39 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-10 18:07:37 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: c7b0d366-db86-4d0c-a4e0-be251d26519e)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_2250.jsonl.zst
[2025-03-10 18:07:37 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-10 18:09:23 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: b0df5a1a-4836-46cf-8e45-58a7c1553309)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_6161.jsonl.zst
[2025-03-10 18:09:23 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-10 18:09:44 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: c1d97368-c0ae-45bb-ae10-5559b3ebc4e4)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_5782.jsonl.zst
[2025-03-10 18:09:44 WARNING] Retrying in 1s [Retry 1/5].
{'eval_loss': 5.276012420654297, 'eval_runtime': 754.8295, 'eval_samples_per_second': 21.706, 'eval_steps_per_second': 0.17, 'epoch': 0.02, 'num_input_tokens_seen': 201326592}
{'loss': 5.2363, 'grad_norm': 0.8435476422309875, 'learning_rate': 0.00075390625, 'epoch': 0.02, 'num_input_tokens_seen': 202375168}
{'loss': 5.1035, 'grad_norm': 1.1267820596694946, 'learning_rate': 0.0007578125, 'epoch': 0.02, 'num_input_tokens_seen': 203423744}
{'loss': 5.3017, 'grad_norm': 0.8555666208267212, 'learning_rate': 0.00076171875, 'epoch': 0.02, 'num_input_tokens_seen': 204472320}
{'loss': 5.1679, 'grad_norm': 0.7608171105384827, 'learning_rate': 0.000765625, 'epoch': 0.02, 'num_input_tokens_seen': 205520896}
{'loss': 5.2326, 'grad_norm': 0.6787221431732178, 'learning_rate': 0.00076953125, 'epoch': 0.02, 'num_input_tokens_seen': 206569472}
{'loss': 5.144, 'grad_norm': 0.6404955983161926, 'learning_rate': 0.0007734375, 'epoch': 0.02, 'num_input_tokens_seen': 207618048}
{'loss': 5.1933, 'grad_norm': 0.6099393367767334, 'learning_rate': 0.00077734375, 'epoch': 0.02, 'num_input_tokens_seen': 208666624}
{'loss': 5.0498, 'grad_norm': 0.5971768498420715, 'learning_rate': 0.00078125, 'epoch': 0.02, 'num_input_tokens_seen': 209715200}
{'loss': 5.1443, 'grad_norm': 0.642633318901062, 'learning_rate': 0.00078515625, 'epoch': 0.02, 'num_input_tokens_seen': 210763776}
{'loss': 5.2125, 'grad_norm': 0.706398606300354, 'learning_rate': 0.0007890625, 'epoch': 0.02, 'num_input_tokens_seen': 211812352}
{'loss': 5.1882, 'grad_norm': 0.817449688911438, 'learning_rate': 0.00079296875, 'epoch': 0.02, 'num_input_tokens_seen': 212860928}
{'loss': 5.0905, 'grad_norm': 0.9392185807228088, 'learning_rate': 0.0007968750000000001, 'epoch': 0.02, 'num_input_tokens_seen': 213909504}
{'loss': 5.059, 'grad_norm': 0.5305852890014648, 'learning_rate': 0.0008007812500000001, 'epoch': 0.03, 'num_input_tokens_seen': 214958080}
{'loss': 5.0838, 'grad_norm': 0.7662672996520996, 'learning_rate': 0.0008046875000000001, 'epoch': 0.03, 'num_input_tokens_seen': 216006656}
{'loss': 5.0112, 'grad_norm': 0.5768160223960876, 'learning_rate': 0.0008085937500000001, 'epoch': 0.03, 'num_input_tokens_seen': 217055232}
{'loss': 4.9684, 'grad_norm': 0.5972586870193481, 'learning_rate': 0.0008125000000000001, 'epoch': 0.03, 'num_input_tokens_seen': 218103808}
{'loss': 5.0764, 'grad_norm': 0.559498131275177, 'learning_rate': 0.00081640625, 'epoch': 0.03, 'num_input_tokens_seen': 219152384}
{'loss': 5.0117, 'grad_norm': 0.555585503578186, 'learning_rate': 0.0008203125, 'epoch': 0.03, 'num_input_tokens_seen': 220200960}
{'loss': 5.1955, 'grad_norm': 0.6180793046951294, 'learning_rate': 0.00082421875, 'epoch': 0.03, 'num_input_tokens_seen': 221249536}
{'loss': 5.1265, 'grad_norm': 0.5784006118774414, 'learning_rate': 0.000828125, 'epoch': 0.03, 'num_input_tokens_seen': 222298112}
{'loss': 5.03, 'grad_norm': 0.5200456380844116, 'learning_rate': 0.00083203125, 'epoch': 0.03, 'num_input_tokens_seen': 223346688}
{'loss': 5.051, 'grad_norm': 0.5112505555152893, 'learning_rate': 0.0008359375, 'epoch': 0.03, 'num_input_tokens_seen': 224395264}
{'loss': 5.0994, 'grad_norm': 0.44979697465896606, 'learning_rate': 0.00083984375, 'epoch': 0.03, 'num_input_tokens_seen': 225443840}
{'loss': 4.94, 'grad_norm': 0.46642380952835083, 'learning_rate': 0.00084375, 'epoch': 0.03, 'num_input_tokens_seen': 226492416}
{'loss': 5.0562, 'grad_norm': 0.49667519330978394, 'learning_rate': 0.00084765625, 'epoch': 0.03, 'num_input_tokens_seen': 227540992}
{'loss': 4.9217, 'grad_norm': 0.4302496314048767, 'learning_rate': 0.0008515625, 'epoch': 0.03, 'num_input_tokens_seen': 228589568}
{'loss': 4.8588, 'grad_norm': 0.5326887369155884, 'learning_rate': 0.00085546875, 'epoch': 0.03, 'num_input_tokens_seen': 229638144}
{'loss': 4.8501, 'grad_norm': 0.45604026317596436, 'learning_rate': 0.000859375, 'epoch': 0.03, 'num_input_tokens_seen': 230686720}
{'loss': 4.8774, 'grad_norm': 0.4497997462749481, 'learning_rate': 0.00086328125, 'epoch': 0.03, 'num_input_tokens_seen': 231735296}
{'loss': 5.0143, 'grad_norm': 0.526670515537262, 'learning_rate': 0.0008671875, 'epoch': 0.03, 'num_input_tokens_seen': 232783872}
{'loss': 4.9512, 'grad_norm': 0.5823948979377747, 'learning_rate': 0.00087109375, 'epoch': 0.03, 'num_input_tokens_seen': 233832448}
{'loss': 4.915, 'grad_norm': 0.6516634821891785, 'learning_rate': 0.000875, 'epoch': 0.03, 'num_input_tokens_seen': 234881024}
{'loss': 4.9318, 'grad_norm': 0.7564677596092224, 'learning_rate': 0.00087890625, 'epoch': 0.03, 'num_input_tokens_seen': 235929600}
{'loss': 4.9041, 'grad_norm': 0.7170491814613342, 'learning_rate': 0.0008828125, 'epoch': 0.03, 'num_input_tokens_seen': 236978176}
{'loss': 4.9727, 'grad_norm': 0.7671059966087341, 'learning_rate': 0.00088671875, 'epoch': 0.03, 'num_input_tokens_seen': 238026752}
{'loss': 4.7895, 'grad_norm': 0.8752806782722473, 'learning_rate': 0.000890625, 'epoch': 0.03, 'num_input_tokens_seen': 239075328}
{'loss': 4.8845, 'grad_norm': 0.8313667178153992, 'learning_rate': 0.00089453125, 'epoch': 0.03, 'num_input_tokens_seen': 240123904}
{'loss': 4.8325, 'grad_norm': 0.9223323464393616, 'learning_rate': 0.0008984375, 'epoch': 0.03, 'num_input_tokens_seen': 241172480}
{'loss': 4.8991, 'grad_norm': 0.7362072467803955, 'learning_rate': 0.00090234375, 'epoch': 0.03, 'num_input_tokens_seen': 242221056}
{'loss': 4.7443, 'grad_norm': 0.6667400598526001, 'learning_rate': 0.00090625, 'epoch': 0.03, 'num_input_tokens_seen': 243269632}
{'loss': 4.8913, 'grad_norm': 0.5431771874427795, 'learning_rate': 0.00091015625, 'epoch': 0.03, 'num_input_tokens_seen': 244318208}
{'loss': 4.8997, 'grad_norm': 0.5542160272598267, 'learning_rate': 0.0009140625, 'epoch': 0.03, 'num_input_tokens_seen': 245366784}
{'loss': 4.8448, 'grad_norm': 0.6110911965370178, 'learning_rate': 0.0009179687500000001, 'epoch': 0.03, 'num_input_tokens_seen': 246415360}
{'loss': 4.7975, 'grad_norm': 0.5550041794776917, 'learning_rate': 0.0009218750000000001, 'epoch': 0.03, 'num_input_tokens_seen': 247463936}
{'loss': 4.87, 'grad_norm': 0.4778221547603607, 'learning_rate': 0.0009257812500000001, 'epoch': 0.03, 'num_input_tokens_seen': 248512512}
{'loss': 4.7594, 'grad_norm': 0.35899603366851807, 'learning_rate': 0.0009296875000000001, 'epoch': 0.03, 'num_input_tokens_seen': 249561088}
{'loss': 4.8338, 'grad_norm': 0.494094580411911, 'learning_rate': 0.0009335937500000001, 'epoch': 0.03, 'num_input_tokens_seen': 250609664}
{'loss': 4.7424, 'grad_norm': 0.4671477675437927, 'learning_rate': 0.0009375, 'epoch': 0.03, 'num_input_tokens_seen': 251658240}
{'loss': 4.7593, 'grad_norm': 0.4691649079322815, 'learning_rate': 0.00094140625, 'epoch': 0.03, 'num_input_tokens_seen': 252706816}
{'loss': 4.7869, 'grad_norm': 0.6212939023971558, 'learning_rate': 0.0009453125, 'epoch': 0.03, 'num_input_tokens_seen': 253755392}
{'loss': 4.7925, 'grad_norm': 0.621306300163269, 'learning_rate': 0.00094921875, 'epoch': 0.03, 'num_input_tokens_seen': 254803968}
{'loss': 4.7714, 'grad_norm': 0.6991429328918457, 'learning_rate': 0.000953125, 'epoch': 0.03, 'num_input_tokens_seen': 255852544}
{'loss': 5.2726, 'grad_norm': 1.016664743423462, 'learning_rate': 0.00095703125, 'epoch': 0.03, 'num_input_tokens_seen': 256901120}
{'loss': 4.9125, 'grad_norm': 1.3091747760772705, 'learning_rate': 0.0009609375, 'epoch': 0.03, 'num_input_tokens_seen': 257949696}
{'loss': 4.839, 'grad_norm': 1.2617076635360718, 'learning_rate': 0.00096484375, 'epoch': 0.03, 'num_input_tokens_seen': 258998272}
{'loss': 4.8412, 'grad_norm': 0.9403041005134583, 'learning_rate': 0.00096875, 'epoch': 0.03, 'num_input_tokens_seen': 260046848}
{'loss': 5.0193, 'grad_norm': 0.9802642464637756, 'learning_rate': 0.00097265625, 'epoch': 0.03, 'num_input_tokens_seen': 261095424}
{'loss': 4.7372, 'grad_norm': 0.9636861085891724, 'learning_rate': 0.0009765625, 'epoch': 0.03, 'num_input_tokens_seen': 262144000}
{'loss': 4.7878, 'grad_norm': 0.7803710699081421, 'learning_rate': 0.00098046875, 'epoch': 0.03, 'num_input_tokens_seen': 263192576}
{'loss': 4.8126, 'grad_norm': 0.7087182402610779, 'learning_rate': 0.000984375, 'epoch': 0.03, 'num_input_tokens_seen': 264241152}
{'loss': 4.7252, 'grad_norm': 0.7220279574394226, 'learning_rate': 0.00098828125, 'epoch': 0.03, 'num_input_tokens_seen': 265289728}
{'loss': 4.7419, 'grad_norm': 0.6956494450569153, 'learning_rate': 0.0009921875, 'epoch': 0.03, 'num_input_tokens_seen': 266338304}
{'loss': 4.8041, 'grad_norm': 0.8009976148605347, 'learning_rate': 0.00099609375, 'epoch': 0.03, 'num_input_tokens_seen': 267386880}
{'loss': 4.7016, 'grad_norm': 0.6665300130844116, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 268435456}
{'eval_loss': 4.753816604614258, 'eval_runtime': 661.8529, 'eval_samples_per_second': 24.755, 'eval_steps_per_second': 0.193, 'epoch': 0.03, 'num_input_tokens_seen': 268435456}
{'loss': 4.6762, 'grad_norm': 0.5311985611915588, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 269484032}
{'loss': 4.6296, 'grad_norm': 0.5160760879516602, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 270532608}
{'loss': 4.7422, 'grad_norm': 0.5964047312736511, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 271581184}
{'loss': 4.7396, 'grad_norm': 0.4793979227542877, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 272629760}
{'loss': 4.733, 'grad_norm': 0.5280688405036926, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 273678336}
{'loss': 4.9591, 'grad_norm': 0.8669152855873108, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 274726912}
{'loss': 4.7953, 'grad_norm': 0.8417720198631287, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 275775488}
{'loss': 4.7972, 'grad_norm': 0.9349585175514221, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 276824064}
{'loss': 4.7233, 'grad_norm': 0.8441230654716492, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 277872640}
{'loss': 4.8032, 'grad_norm': 0.7163352370262146, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 278921216}
{'loss': 4.4369, 'grad_norm': 1.0364480018615723, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 279969792}
{'loss': 4.557, 'grad_norm': 1.012042760848999, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 281018368}
{'loss': 4.7696, 'grad_norm': 1.1818541288375854, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 282066944}
{'loss': 4.7835, 'grad_norm': 0.8296499848365784, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 283115520}
{'loss': 4.761, 'grad_norm': 0.6920194625854492, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 284164096}
{'loss': 4.6239, 'grad_norm': 0.8495435118675232, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 285212672}
{'loss': 4.6914, 'grad_norm': 0.6536931991577148, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 286261248}
{'loss': 4.776, 'grad_norm': 0.7161967754364014, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 287309824}
{'loss': 4.7096, 'grad_norm': 0.5441194176673889, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 288358400}
{'loss': 4.7278, 'grad_norm': 0.5437328219413757, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 289406976}
{'loss': 4.6126, 'grad_norm': 0.49404028058052063, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 290455552}
{'loss': 4.6594, 'grad_norm': 0.4274217188358307, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 291504128}
{'loss': 4.6365, 'grad_norm': 0.48871853947639465, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 292552704}
{'loss': 4.5999, 'grad_norm': 0.5101707577705383, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 293601280}
{'loss': 4.5869, 'grad_norm': 0.4579870104789734, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 294649856}
{'loss': 4.5993, 'grad_norm': 0.44694098830223083, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 295698432}
{'loss': 4.6369, 'grad_norm': 0.42955130338668823, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 296747008}
{'loss': 4.5973, 'grad_norm': 0.532283365726471, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 297795584}
{'loss': 4.3953, 'grad_norm': 0.5553389191627502, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 298844160}
{'loss': 4.5501, 'grad_norm': 0.4733176529407501, 'learning_rate': 0.001, 'epoch': 0.03, 'num_input_tokens_seen': 299892736}
{'loss': 4.4896, 'grad_norm': 0.5510519742965698, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 300941312}
{'loss': 4.348, 'grad_norm': 0.5312983393669128, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 301989888}
{'loss': 4.4, 'grad_norm': 0.4173823297023773, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 303038464}
{'loss': 4.4971, 'grad_norm': 0.4799824357032776, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 304087040}
{'loss': 4.5507, 'grad_norm': 0.4494017958641052, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 305135616}
{'loss': 4.5655, 'grad_norm': 0.36501485109329224, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 306184192}
{'loss': 4.5189, 'grad_norm': 0.4833853840827942, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 307232768}
{'loss': 4.5387, 'grad_norm': 0.5214531421661377, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 308281344}
{'loss': 4.5509, 'grad_norm': 0.5383253693580627, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 309329920}
{'loss': 4.4112, 'grad_norm': 0.5364778637886047, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 310378496}
{'loss': 4.568, 'grad_norm': 0.3624066114425659, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 311427072}
{'loss': 4.5289, 'grad_norm': 0.5469081401824951, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 312475648}
{'loss': 4.4953, 'grad_norm': 0.5212593674659729, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 313524224}
{'loss': 4.4614, 'grad_norm': 0.36742305755615234, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 314572800}
{'loss': 4.4757, 'grad_norm': 0.43591663241386414, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 315621376}
{'loss': 4.5321, 'grad_norm': 0.483548104763031, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 316669952}
{'loss': 4.449, 'grad_norm': 0.3971082866191864, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 317718528}
{'loss': 4.4539, 'grad_norm': 0.3416251540184021, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 318767104}
{'loss': 4.3456, 'grad_norm': 0.45731472969055176, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 319815680}
{'loss': 4.4179, 'grad_norm': 0.4462226331233978, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 320864256}
{'loss': 4.3691, 'grad_norm': 0.3393065631389618, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 321912832}
{'loss': 4.4361, 'grad_norm': 0.39659640192985535, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 322961408}
{'loss': 4.4166, 'grad_norm': 0.42212849855422974, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 324009984}
{'loss': 4.3931, 'grad_norm': 0.3403238356113434, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 325058560}
{'loss': 4.3003, 'grad_norm': 0.3405858278274536, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 326107136}
{'loss': 4.4339, 'grad_norm': 0.42516669631004333, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 327155712}
{'loss': 4.4258, 'grad_norm': 0.4387160539627075, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 328204288}
{'loss': 4.3774, 'grad_norm': 0.3546140193939209, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 329252864}
{'loss': 4.3261, 'grad_norm': 0.3842155933380127, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 330301440}
{'loss': 4.2843, 'grad_norm': 0.32807183265686035, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 331350016}
{'loss': 4.3627, 'grad_norm': 0.3635430932044983, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 332398592}
{'loss': 4.3304, 'grad_norm': 0.32113364338874817, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 333447168}
{'loss': 4.258, 'grad_norm': 0.3261938989162445, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 334495744}
{'loss': 4.392, 'grad_norm': 0.35287028551101685, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 335544320}
{'eval_loss': 4.340233325958252, 'eval_runtime': 641.4064, 'eval_samples_per_second': 25.544, 'eval_steps_per_second': 0.2, 'epoch': 0.04, 'num_input_tokens_seen': 335544320}
{'loss': 4.4095, 'grad_norm': 0.30875736474990845, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 336592896}
{'loss': 3.8896, 'grad_norm': 0.6334038972854614, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 337641472}
{'loss': 4.449, 'grad_norm': 0.5519331693649292, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 338690048}
{'loss': 4.4388, 'grad_norm': 0.4262654185295105, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 339738624}
{'loss': 4.3918, 'grad_norm': 0.4348645508289337, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 340787200}
{'loss': 4.3677, 'grad_norm': 0.3858915865421295, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 341835776}
{'loss': 4.3343, 'grad_norm': 0.4542510509490967, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 342884352}
{'loss': 4.3196, 'grad_norm': 0.4413583278656006, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 343932928}
{'loss': 4.322, 'grad_norm': 0.5200892686843872, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 344981504}
{'loss': 4.2409, 'grad_norm': 0.4969848692417145, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 346030080}
{'loss': 4.2263, 'grad_norm': 0.43436068296432495, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 347078656}
{'loss': 4.2271, 'grad_norm': 0.4760046899318695, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 348127232}
{'loss': 4.3567, 'grad_norm': 0.43881112337112427, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 349175808}
{'loss': 4.2606, 'grad_norm': 0.5361112952232361, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 350224384}
{'loss': 4.3831, 'grad_norm': 0.5959597229957581, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 351272960}
{'loss': 4.2899, 'grad_norm': 0.6709368824958801, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 352321536}
{'loss': 4.2263, 'grad_norm': 0.6585149168968201, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 353370112}
{'loss': 4.3428, 'grad_norm': 0.5447191596031189, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 354418688}
{'loss': 4.3642, 'grad_norm': 0.576545238494873, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 355467264}
{'loss': 4.025, 'grad_norm': 0.7567218542098999, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 356515840}
{'loss': 4.2593, 'grad_norm': 0.6053742170333862, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 357564416}
{'loss': 4.2864, 'grad_norm': 0.54949551820755, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 358612992}
{'loss': 4.3183, 'grad_norm': 0.4792100489139557, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 359661568}
{'loss': 4.2957, 'grad_norm': 0.4366244077682495, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 360710144}
{'loss': 4.3502, 'grad_norm': 0.5610309839248657, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 361758720}
{'loss': 4.2673, 'grad_norm': 0.42132946848869324, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 362807296}
{'loss': 4.2565, 'grad_norm': 0.45927727222442627, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 363855872}
{'loss': 4.3009, 'grad_norm': 0.40793168544769287, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 364904448}
{'loss': 4.2584, 'grad_norm': 0.3818293511867523, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 365953024}
{'loss': 4.3187, 'grad_norm': 0.4942944645881653, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 367001600}
{'loss': 4.2056, 'grad_norm': 0.5316190719604492, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 368050176}
{'loss': 4.2403, 'grad_norm': 0.4738222658634186, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 369098752}
{'loss': 4.244, 'grad_norm': 0.41153445839881897, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 370147328}
{'loss': 4.2876, 'grad_norm': 0.35864201188087463, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 371195904}
{'loss': 4.2457, 'grad_norm': 0.4317127466201782, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 372244480}
{'loss': 4.2138, 'grad_norm': 0.4922076165676117, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 373293056}
{'loss': 4.1875, 'grad_norm': 0.5150508880615234, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 374341632}
{'loss': 4.1485, 'grad_norm': 0.40701162815093994, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 375390208}
{'loss': 4.1062, 'grad_norm': 0.40378910303115845, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 376438784}
{'loss': 4.226, 'grad_norm': 0.4435281753540039, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 377487360}
{'loss': 4.2034, 'grad_norm': 0.37908127903938293, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 378535936}
{'loss': 4.1502, 'grad_norm': 0.408202588558197, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 379584512}
{'loss': 4.1623, 'grad_norm': 0.4542413651943207, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 380633088}
{'loss': 4.206, 'grad_norm': 0.5084658861160278, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 381681664}
{'loss': 4.1867, 'grad_norm': 0.432908833026886, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 382730240}
{'loss': 4.2377, 'grad_norm': 0.38273656368255615, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 383778816}
{'loss': 4.1443, 'grad_norm': 0.39886555075645447, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 384827392}
{'loss': 4.16, 'grad_norm': 0.4073260724544525, 'learning_rate': 0.001, 'epoch': 0.04, 'num_input_tokens_seen': 385875968}
{'loss': 4.0871, 'grad_norm': 0.46062660217285156, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 386924544}
{'loss': 4.1655, 'grad_norm': 0.3555128574371338, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 387973120}
{'loss': 4.1993, 'grad_norm': 0.35318323969841003, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 389021696}
{'loss': 4.0745, 'grad_norm': 0.3469637632369995, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 390070272}
{'loss': 4.1844, 'grad_norm': 0.3650517761707306, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 391118848}
{'loss': 4.1744, 'grad_norm': 0.4310692846775055, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 392167424}
{'loss': 4.1896, 'grad_norm': 0.465585857629776, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 393216000}
{'loss': 4.0568, 'grad_norm': 0.5539769530296326, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 394264576}
{'loss': 4.2642, 'grad_norm': 0.5437971949577332, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 395313152}
{'loss': 4.1705, 'grad_norm': 0.6534202694892883, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 396361728}
{'loss': 3.9844, 'grad_norm': 0.7271204590797424, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 397410304}
{'loss': 4.105, 'grad_norm': 0.7395262122154236, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 398458880}
{'loss': 4.2332, 'grad_norm': 0.9734097719192505, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 399507456}
{'loss': 4.1501, 'grad_norm': 1.1519765853881836, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 400556032}
{'loss': 4.0756, 'grad_norm': 0.7837873697280884, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 401604608}
{'loss': 4.013, 'grad_norm': 0.8097010850906372, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 402653184}
{'eval_loss': 4.120734214782715, 'eval_runtime': 626.8806, 'eval_samples_per_second': 26.136, 'eval_steps_per_second': 0.204, 'epoch': 0.05, 'num_input_tokens_seen': 402653184}
{'loss': 4.0955, 'grad_norm': 0.6811020970344543, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 403701760}
{'loss': 4.0917, 'grad_norm': 0.5382081270217896, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 404750336}
{'loss': 4.0414, 'grad_norm': 0.4250117242336273, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 405798912}
{'loss': 4.1051, 'grad_norm': 0.4233124256134033, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 406847488}
{'loss': 4.1475, 'grad_norm': 0.41960859298706055, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 407896064}
{'loss': 4.0322, 'grad_norm': 0.4991297423839569, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 408944640}
{'loss': 4.0664, 'grad_norm': 0.43890711665153503, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 409993216}
{'loss': 4.1126, 'grad_norm': 0.38538315892219543, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 411041792}
{'loss': 4.0591, 'grad_norm': 0.41170960664749146, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 412090368}
{'loss': 4.1145, 'grad_norm': 0.42465972900390625, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 413138944}
{'loss': 4.0393, 'grad_norm': 0.4215935468673706, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 414187520}
{'loss': 3.9509, 'grad_norm': 0.5031537413597107, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 415236096}
{'loss': 3.9314, 'grad_norm': 0.5212794542312622, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 416284672}
{'loss': 4.062, 'grad_norm': 0.5779813528060913, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 417333248}
{'loss': 4.0264, 'grad_norm': 0.5523960590362549, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 418381824}
{'loss': 4.0366, 'grad_norm': 0.501869797706604, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 419430400}
{'loss': 4.016, 'grad_norm': 0.390077143907547, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 420478976}
{'loss': 3.9438, 'grad_norm': 0.39393457770347595, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 421527552}
{'loss': 3.9882, 'grad_norm': 0.3395244777202606, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 422576128}
{'loss': 3.95, 'grad_norm': 0.3985426425933838, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 423624704}
{'loss': 3.9708, 'grad_norm': 0.4353885352611542, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 424673280}
{'loss': 3.9959, 'grad_norm': 0.39546582102775574, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 425721856}
{'loss': 3.9475, 'grad_norm': 0.3725046217441559, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 426770432}
{'loss': 3.8599, 'grad_norm': 0.5391167998313904, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 427819008}
{'loss': 3.9765, 'grad_norm': 0.5383077263832092, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 428867584}
{'loss': 3.8999, 'grad_norm': 0.4455236494541168, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 429916160}
{'loss': 4.0357, 'grad_norm': 0.4489726722240448, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 430964736}
{'loss': 3.992, 'grad_norm': 0.45914894342422485, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 432013312}
{'loss': 3.9556, 'grad_norm': 0.5718650817871094, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 433061888}
{'loss': 3.9797, 'grad_norm': 0.5529163479804993, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 434110464}
{'loss': 3.9479, 'grad_norm': 0.4689369201660156, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 435159040}
{'loss': 3.9358, 'grad_norm': 0.448303759098053, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 436207616}
{'loss': 3.9699, 'grad_norm': 0.4203392565250397, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 437256192}
{'loss': 3.8173, 'grad_norm': 0.4046834707260132, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 438304768}
{'loss': 3.8183, 'grad_norm': 0.3998134136199951, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 439353344}
{'loss': 3.8477, 'grad_norm': 0.4120945632457733, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 440401920}
{'loss': 3.8486, 'grad_norm': 0.39726078510284424, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 441450496}
{'loss': 3.942, 'grad_norm': 0.399142861366272, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 442499072}
{'loss': 3.9038, 'grad_norm': 0.41262856125831604, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 443547648}
{'loss': 3.8447, 'grad_norm': 0.4645870327949524, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 444596224}
{'loss': 3.9215, 'grad_norm': 0.49330976605415344, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 445644800}
{'loss': 4.5329, 'grad_norm': 4.813076972961426, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 446693376}
{'loss': 3.763, 'grad_norm': 1.0100675821304321, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 447741952}
{'loss': 3.9888, 'grad_norm': 1.2422761917114258, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 448790528}
{'loss': 3.9209, 'grad_norm': 1.1251254081726074, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 449839104}
{'loss': 4.1438, 'grad_norm': 1.926529049873352, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 450887680}
{'loss': 4.0952, 'grad_norm': 1.2948275804519653, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 451936256}
{'loss': 3.9411, 'grad_norm': 1.1000643968582153, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 452984832}
{'loss': 3.988, 'grad_norm': 1.3160468339920044, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 454033408}
{'loss': 4.0241, 'grad_norm': 1.0201517343521118, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 455081984}
{'loss': 3.9875, 'grad_norm': 0.9689710140228271, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 456130560}
{'loss': 3.8684, 'grad_norm': 1.045577049255371, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 457179136}
{'loss': 3.865, 'grad_norm': 0.931566059589386, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 458227712}
{'loss': 3.728, 'grad_norm': 0.945274293422699, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 459276288}
{'loss': 3.955, 'grad_norm': 0.7679930925369263, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 460324864}
{'loss': 4.4113, 'grad_norm': 0.889451801776886, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 461373440}
{'loss': 3.8928, 'grad_norm': 0.9069199562072754, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 462422016}
{'loss': 3.9624, 'grad_norm': 0.8945743441581726, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 463470592}
{'loss': 3.9698, 'grad_norm': 0.7373656630516052, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 464519168}
{'loss': 3.921, 'grad_norm': 0.6688440442085266, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 465567744}
{'loss': 3.8908, 'grad_norm': 0.5442579984664917, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 466616320}
{'loss': 3.9138, 'grad_norm': 0.5583804845809937, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 467664896}
{'loss': 3.8731, 'grad_norm': 0.504666268825531, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 468713472}
{'loss': 3.7961, 'grad_norm': 0.4965992867946625, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 469762048}
{'eval_loss': 3.7728981971740723, 'eval_runtime': 616.374, 'eval_samples_per_second': 26.581, 'eval_steps_per_second': 0.208, 'epoch': 0.05, 'num_input_tokens_seen': 469762048}
{'loss': 3.8829, 'grad_norm': 0.44414225220680237, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 470810624}
{'loss': 3.6939, 'grad_norm': 0.5276159644126892, 'learning_rate': 0.001, 'epoch': 0.05, 'num_input_tokens_seen': 471859200}
{'loss': 3.8173, 'grad_norm': 0.4666613042354584, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 472907776}
{'loss': 3.6884, 'grad_norm': 0.4581243097782135, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 473956352}
{'loss': 3.789, 'grad_norm': 0.4697781205177307, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 475004928}
{'loss': 3.8791, 'grad_norm': 0.5336131453514099, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 476053504}
{'loss': 3.8077, 'grad_norm': 0.5709654092788696, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 477102080}
{'loss': 3.8421, 'grad_norm': 0.5592761039733887, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 478150656}
{'loss': 3.8135, 'grad_norm': 0.4490680694580078, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 479199232}
{'loss': 3.7535, 'grad_norm': 0.3931736648082733, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 480247808}
{'loss': 3.7885, 'grad_norm': 0.41578060388565063, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 481296384}
{'loss': 3.6255, 'grad_norm': 0.429817795753479, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 482344960}
{'loss': 3.7202, 'grad_norm': 0.49616578221321106, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 483393536}
  9%|▊         | 704/8192 [9:33:48<79:08:04, 38.05s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 0faae356-e828-4cff-9a49-42b397431927)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_185.jsonl.zst
  9%|▊         | 704/8192 [9:38:28<79:08:04, 38.05s/it]Retrying in 1s [Retry 1/5].
  9%|▊         | 704/8192 [9:38:28<79:08:04, 38.05s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 9557423f-6937-4f70-b50f-05b0c01f5bf3)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_4035.jsonl.zst
  9%|▊         | 704/8192 [9:44:58<79:08:04, 38.05s/it]Retrying in 1s [Retry 1/5].
 10%|█         | 832/8192 [11:28:20<80:32:25, 39.39s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 939d1d36-c607-4d3c-a0a0-8e447579340b)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_165.jsonl.zst
 10%|█         | 832/8192 [11:30:25<80:32:25, 39.39s/it]Retrying in 1s [Retry 1/5].
 10%|█         | 832/8192 [11:30:25<80:32:25, 39.39s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 0b99bfd1-07ae-46db-81fa-fc6ef0eabdbc)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_1529.jsonl.zst
 10%|█         | 832/8192 [11:38:24<80:32:25, 39.39s/it]Retrying in 1s [Retry 1/5].
 10%|█         | 832/8192 [11:38:24<80:32:25, 39.39s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: c208d1bb-5d13-45d2-9a01-1d5a2defa598)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk5/example_holdout_4562.jsonl.zst
 10%|█         | 832/8192 [11:39:58<80:32:25, 39.39s/it]Retrying in 1s [Retry 1/5].
 10%|█         | 832/8192 [11:39:58<80:32:25, 39.39s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 2bf98b5c-473b-4e00-aca2-b152efddb992)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_4414.jsonl.zst
 10%|█         | 832/8192 [11:41:00<80:32:25, 39.39s/it]Retrying in 1s [Retry 1/5].
 11%|█         | 896/8192 [12:24:54<77:09:28, 38.07s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 3b8321b9-2d88-4bfa-9eca-b201c444cba3)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk5/example_holdout_405.jsonl.zst
 11%|█         | 896/8192 [12:25:55<77:09:28, 38.07s/it]Retrying in 1s [Retry 1/5].
 11%|█         | 896/8192 [12:25:55<77:09:28, 38.07s/it]'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: a98a238a-c0a4-4295-8502-316a89a7ae29)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk1/example_holdout_2524.jsonl.zst
 11%|█         | 896/8192 [12:33:14<77:09:28, 38.07s/it]Retrying in 1s [Retry 1/5].
 11%|█▏        | 922/8192 [12:52:49<76:09:46, 37.71s/it]'(ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), '(Request ID: 36a7cc72-4605-416a-8742-59488d719150)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk1/example_train_5267.jsonl.zst
 11%|█▏        | 922/8192 [12:52:59<76:09:46, 37.71s/it]Retrying in 1s [Retry 1/5].
 12%|█▏        | 943/8192 [13:06:07<76:15:57, 37.88s/it]
{'loss': 3.7796, 'grad_norm': 0.4774172008037567, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 484442112}
{'loss': 3.7779, 'grad_norm': 0.45830512046813965, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 485490688}
{'loss': 3.6516, 'grad_norm': 0.4130597710609436, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 486539264}
{'loss': 3.7018, 'grad_norm': 0.3804127275943756, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 487587840}
{'loss': 3.6893, 'grad_norm': 0.36560356616973877, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 488636416}
{'loss': 3.6362, 'grad_norm': 0.3827981948852539, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 489684992}
{'loss': 3.5987, 'grad_norm': 0.37492236495018005, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 490733568}
{'loss': 3.7165, 'grad_norm': 0.46995237469673157, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 491782144}
{'loss': 3.6097, 'grad_norm': 0.4908960461616516, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 492830720}
{'loss': 3.6035, 'grad_norm': 0.5318525433540344, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 493879296}
{'loss': 3.6643, 'grad_norm': 0.4848596453666687, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 494927872}
{'loss': 3.6586, 'grad_norm': 0.4421922266483307, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 495976448}
{'loss': 3.5902, 'grad_norm': 0.4107126295566559, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 497025024}
{'loss': 3.6937, 'grad_norm': 0.3975088894367218, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 498073600}
{'loss': 3.6496, 'grad_norm': 0.4559416174888611, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 499122176}
{'loss': 3.66, 'grad_norm': 0.41401296854019165, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 500170752}
{'loss': 3.5551, 'grad_norm': 0.45235902070999146, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 501219328}
{'loss': 3.4794, 'grad_norm': 0.427593857049942, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 502267904}
{'loss': 3.5345, 'grad_norm': 0.4024144411087036, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 503316480}
{'loss': 3.5784, 'grad_norm': 0.410284161567688, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 504365056}
{'loss': 3.6177, 'grad_norm': 0.37683290243148804, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 505413632}
{'loss': 3.5883, 'grad_norm': 0.417323499917984, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 506462208}
{'loss': 3.5888, 'grad_norm': 0.4327872693538666, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 507510784}
{'loss': 3.5891, 'grad_norm': 0.5366392731666565, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 508559360}
{'loss': 3.3725, 'grad_norm': 0.45735156536102295, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 509607936}
{'loss': 3.5674, 'grad_norm': 0.4255360960960388, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 510656512}
{'loss': 3.3523, 'grad_norm': 0.6517689824104309, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 511705088}
{'loss': 3.5901, 'grad_norm': 0.5713740587234497, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 512753664}
{'loss': 3.542, 'grad_norm': 0.5570502281188965, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 513802240}
{'loss': 3.4246, 'grad_norm': 0.6477808356285095, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 514850816}
{'loss': 3.4954, 'grad_norm': 0.5195346474647522, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 515899392}
{'loss': 3.6516, 'grad_norm': 0.5446246862411499, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 516947968}
{'loss': 3.5955, 'grad_norm': 0.5475099086761475, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 517996544}
{'loss': 3.5516, 'grad_norm': 0.4719395041465759, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 519045120}
{'loss': 3.5439, 'grad_norm': 0.43647533655166626, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 520093696}
{'loss': 3.579, 'grad_norm': 0.5048384070396423, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 521142272}
{'loss': 3.4742, 'grad_norm': 0.4902295172214508, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 522190848}
{'loss': 3.4363, 'grad_norm': 0.525496244430542, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 523239424}
{'loss': 3.3658, 'grad_norm': 0.5224571824073792, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 524288000}
{'loss': 3.4816, 'grad_norm': 0.45781856775283813, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 525336576}
{'loss': 3.4612, 'grad_norm': 0.3764704763889313, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 526385152}
{'loss': 3.5172, 'grad_norm': 0.3994409143924713, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 527433728}
{'loss': 3.5462, 'grad_norm': 0.45144984126091003, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 528482304}
{'loss': 3.5079, 'grad_norm': 0.4901409149169922, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 529530880}
{'loss': 3.5187, 'grad_norm': 0.45689818263053894, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 530579456}
{'loss': 3.4408, 'grad_norm': 0.4650699198246002, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 531628032}
{'loss': 3.4019, 'grad_norm': 0.40419647097587585, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 532676608}
{'loss': 3.5255, 'grad_norm': 0.3895981013774872, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 533725184}
{'loss': 3.312, 'grad_norm': 0.46533191204071045, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 534773760}
{'loss': 3.4233, 'grad_norm': 0.5021492838859558, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 535822336}
{'loss': 3.4211, 'grad_norm': 0.6763796806335449, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 536870912}
{'eval_loss': 3.38647198677063, 'eval_runtime': 681.5531, 'eval_samples_per_second': 24.039, 'eval_steps_per_second': 0.188, 'epoch': 0.06, 'num_input_tokens_seen': 536870912}
{'loss': 3.2825, 'grad_norm': 0.75739586353302, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 537919488}
{'loss': 3.4758, 'grad_norm': 0.49962809681892395, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 538968064}
{'loss': 3.4105, 'grad_norm': 0.47640085220336914, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 540016640}
{'loss': 3.4393, 'grad_norm': 0.4722411632537842, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 541065216}
{'loss': 3.4254, 'grad_norm': 0.4715781807899475, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 542113792}
{'loss': 3.3992, 'grad_norm': 0.474001407623291, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 543162368}
{'loss': 3.4274, 'grad_norm': 0.48976385593414307, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 544210944}
{'loss': 3.3255, 'grad_norm': 0.4819697141647339, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 545259520}
{'loss': 3.3679, 'grad_norm': 0.37490880489349365, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 546308096}
{'loss': 3.377, 'grad_norm': 0.4356544315814972, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 547356672}
{'loss': 3.4294, 'grad_norm': 0.3786229193210602, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 548405248}
{'loss': 3.2323, 'grad_norm': 0.4364008605480194, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 549453824}
{'loss': 3.4615, 'grad_norm': 0.39242950081825256, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 550502400}
{'loss': 3.3589, 'grad_norm': 0.4270903766155243, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 551550976}
{'loss': 3.4366, 'grad_norm': 0.4204763174057007, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 552599552}
{'loss': 3.3859, 'grad_norm': 0.554025411605835, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 553648128}
{'loss': 3.2353, 'grad_norm': 0.5719075798988342, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 554696704}
{'loss': 3.3798, 'grad_norm': 0.4803822338581085, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 555745280}
{'loss': 3.1191, 'grad_norm': 0.5494056344032288, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 556793856}
{'loss': 3.424, 'grad_norm': 0.4569101333618164, 'learning_rate': 0.001, 'epoch': 0.06, 'num_input_tokens_seen': 557842432}
{'loss': 3.4299, 'grad_norm': 0.48103874921798706, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 558891008}
{'loss': 3.3483, 'grad_norm': 0.44187718629837036, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 559939584}
{'loss': 3.3196, 'grad_norm': 0.4359618127346039, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 560988160}
{'loss': 3.4479, 'grad_norm': 0.37653473019599915, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 562036736}
{'loss': 3.2509, 'grad_norm': 0.4397211968898773, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 563085312}
{'loss': 3.4193, 'grad_norm': 0.5013746619224548, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 564133888}
{'loss': 3.3391, 'grad_norm': 0.5044407844543457, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 565182464}
{'loss': 3.3223, 'grad_norm': 0.45118412375450134, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 566231040}
{'loss': 3.3041, 'grad_norm': 0.5617747902870178, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 567279616}
{'loss': 3.3436, 'grad_norm': 0.5154598355293274, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 568328192}
{'loss': 3.3739, 'grad_norm': 0.4647876024246216, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 569376768}
{'loss': 3.3366, 'grad_norm': 0.3766598701477051, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 570425344}
{'loss': 3.3098, 'grad_norm': 0.40857356786727905, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 571473920}
{'loss': 3.0331, 'grad_norm': 0.4163903594017029, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 572522496}
{'loss': 3.3184, 'grad_norm': 0.38519713282585144, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 573571072}
{'loss': 3.3886, 'grad_norm': 0.38155344128608704, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 574619648}
{'loss': 3.2855, 'grad_norm': 0.3684964179992676, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 575668224}
{'loss': 3.0484, 'grad_norm': 0.3504279553890228, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 576716800}
{'loss': 3.2702, 'grad_norm': 0.42653048038482666, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 577765376}
{'loss': 3.312, 'grad_norm': 0.4263192415237427, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 578813952}
{'loss': 3.3355, 'grad_norm': 0.4272316098213196, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 579862528}
{'loss': 3.2806, 'grad_norm': 0.40996676683425903, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 580911104}
{'loss': 3.2504, 'grad_norm': 0.403242826461792, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 581959680}
{'loss': 3.2924, 'grad_norm': 0.46690869331359863, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 583008256}
{'loss': 3.1466, 'grad_norm': 0.515250027179718, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 584056832}
{'loss': 3.2898, 'grad_norm': 0.4872475266456604, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 585105408}
{'loss': 3.3699, 'grad_norm': 0.43510228395462036, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 586153984}
{'loss': 3.1568, 'grad_norm': 0.4732394814491272, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 587202560}
{'loss': 3.2145, 'grad_norm': 0.49767330288887024, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 588251136}
{'loss': 3.2966, 'grad_norm': 0.4968816936016083, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 589299712}
{'loss': 3.2249, 'grad_norm': 0.4123048782348633, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 590348288}
{'loss': 3.3819, 'grad_norm': 0.4349605143070221, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 591396864}
{'loss': 3.3477, 'grad_norm': 0.47485488653182983, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 592445440}
{'loss': 3.3202, 'grad_norm': 0.46784669160842896, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 593494016}
{'loss': 3.2231, 'grad_norm': 0.42318931221961975, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 594542592}
{'loss': 3.2901, 'grad_norm': 0.40393564105033875, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 595591168}
{'loss': 3.2065, 'grad_norm': 0.4144214391708374, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 596639744}
{'loss': 2.8698, 'grad_norm': 0.40921372175216675, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 597688320}
{'loss': 3.2242, 'grad_norm': 0.35226207971572876, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 598736896}
{'loss': 3.2125, 'grad_norm': 0.43364742398262024, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 599785472}
{'loss': 3.2296, 'grad_norm': 0.4272080361843109, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 600834048}
{'loss': 2.9346, 'grad_norm': 0.4155097007751465, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 601882624}
{'loss': 3.2706, 'grad_norm': 0.4263918697834015, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 602931200}
{'loss': 3.3124, 'grad_norm': 0.43336594104766846, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 603979776}
{'eval_loss': 3.1686322689056396, 'eval_runtime': 664.0006, 'eval_samples_per_second': 24.675, 'eval_steps_per_second': 0.193, 'epoch': 0.07, 'num_input_tokens_seen': 603979776}
{'loss': 3.349, 'grad_norm': 0.4504219889640808, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 605028352}
{'loss': 3.3015, 'grad_norm': 0.5899333953857422, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 606076928}
{'loss': 3.2036, 'grad_norm': 0.5814825892448425, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 607125504}
{'loss': 3.2786, 'grad_norm': 0.3971703350543976, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 608174080}
{'loss': 3.0979, 'grad_norm': 0.5669280290603638, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 609222656}
{'loss': 3.0683, 'grad_norm': 0.4786263406276703, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 610271232}
{'loss': 3.1731, 'grad_norm': 0.46415817737579346, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 611319808}
{'loss': 3.2282, 'grad_norm': 0.4295870363712311, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 612368384}
{'loss': 3.2196, 'grad_norm': 0.4184265732765198, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 613416960}
{'loss': 3.2445, 'grad_norm': 0.4624122381210327, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 614465536}
{'loss': 3.1135, 'grad_norm': 0.3681364059448242, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 615514112}
{'loss': 3.1877, 'grad_norm': 0.3612712621688843, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 616562688}
{'loss': 3.308, 'grad_norm': 0.34696292877197266, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 617611264}
{'loss': 3.4995, 'grad_norm': 0.5025363564491272, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 618659840}
{'loss': 3.1853, 'grad_norm': 0.6652331352233887, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 619708416}
{'loss': 3.1844, 'grad_norm': 0.7156277894973755, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 620756992}
{'loss': 3.2325, 'grad_norm': 0.5241081118583679, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 621805568}
{'loss': 2.972, 'grad_norm': 0.5001779198646545, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 622854144}
{'loss': 3.1742, 'grad_norm': 0.4062795341014862, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 623902720}
{'loss': 3.2539, 'grad_norm': 0.4671201705932617, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 624951296}
{'loss': 3.1948, 'grad_norm': 0.3894169330596924, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 625999872}
{'loss': 3.2469, 'grad_norm': 0.4665684998035431, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 627048448}
{'loss': 3.2742, 'grad_norm': 0.43211206793785095, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 628097024}
{'loss': 3.1195, 'grad_norm': 0.4476025700569153, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 629145600}
{'loss': 3.2127, 'grad_norm': 0.3596750795841217, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 630194176}
{'loss': 3.1741, 'grad_norm': 0.40869519114494324, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 631242752}
{'loss': 3.1708, 'grad_norm': 0.36658936738967896, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 632291328}
{'loss': 3.0925, 'grad_norm': 0.35227081179618835, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 633339904}
{'loss': 3.171, 'grad_norm': 0.3942136764526367, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 634388480}
{'loss': 3.1729, 'grad_norm': 0.3163004219532013, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 635437056}
{'loss': 3.1683, 'grad_norm': 0.35835322737693787, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 636485632}
{'loss': 3.1118, 'grad_norm': 0.3395129144191742, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 637534208}
{'loss': 3.2123, 'grad_norm': 0.38003110885620117, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 638582784}
{'loss': 3.167, 'grad_norm': 0.4000258445739746, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 639631360}
{'loss': 3.0668, 'grad_norm': 0.38393035531044006, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 640679936}
{'loss': 2.9125, 'grad_norm': 0.38961607217788696, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 641728512}
{'loss': 3.1024, 'grad_norm': 0.3406165540218353, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 642777088}
{'loss': 3.1262, 'grad_norm': 0.4859096109867096, 'learning_rate': 0.001, 'epoch': 0.07, 'num_input_tokens_seen': 643825664}
{'loss': 3.1155, 'grad_norm': 0.5454179048538208, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 644874240}
{'loss': 3.1594, 'grad_norm': 0.46631914377212524, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 645922816}
{'loss': 3.1164, 'grad_norm': 0.4049534797668457, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 646971392}
{'loss': 2.9272, 'grad_norm': 0.32954707741737366, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 648019968}
{'loss': 3.0888, 'grad_norm': 0.409853458404541, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 649068544}
{'loss': 3.2185, 'grad_norm': 0.43080267310142517, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 650117120}
{'loss': 3.1871, 'grad_norm': 0.4323279857635498, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 651165696}
{'loss': 2.9759, 'grad_norm': 0.3696155846118927, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 652214272}
{'loss': 3.1058, 'grad_norm': 0.3963398337364197, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 653262848}
{'loss': 3.1214, 'grad_norm': 0.4020082652568817, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 654311424}
{'loss': 3.0678, 'grad_norm': 0.4210987091064453, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 655360000}
{'loss': 2.9177, 'grad_norm': 0.44535601139068604, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 656408576}
{'loss': 3.1005, 'grad_norm': 0.363700807094574, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 657457152}
{'loss': 3.0285, 'grad_norm': 0.393673837184906, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 658505728}
{'loss': 3.031, 'grad_norm': 0.3472498059272766, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 659554304}
{'loss': 3.1837, 'grad_norm': 0.45663976669311523, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 660602880}
{'loss': 3.1636, 'grad_norm': 0.44765880703926086, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 661651456}
{'loss': 3.0421, 'grad_norm': 0.5289708375930786, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 662700032}
{'loss': 2.9394, 'grad_norm': 0.5272406339645386, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 663748608}
{'loss': 3.2419, 'grad_norm': 0.5471237301826477, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 664797184}
{'loss': 3.1506, 'grad_norm': 0.5762659311294556, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 665845760}
{'loss': 3.1258, 'grad_norm': 0.5486758351325989, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 666894336}
{'loss': 3.1686, 'grad_norm': 0.4877275228500366, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 667942912}
{'loss': 3.1062, 'grad_norm': 0.35992035269737244, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 668991488}
{'loss': 3.1655, 'grad_norm': 0.39184319972991943, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 670040064}
{'loss': 3.1455, 'grad_norm': 0.46003854274749756, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 671088640}
{'eval_loss': 3.036459445953369, 'eval_runtime': 676.6057, 'eval_samples_per_second': 24.215, 'eval_steps_per_second': 0.189, 'epoch': 0.08, 'num_input_tokens_seen': 671088640}
{'loss': 3.1058, 'grad_norm': 0.45958808064460754, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 672137216}
{'loss': 3.0861, 'grad_norm': 0.41562288999557495, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 673185792}
{'loss': 3.1135, 'grad_norm': 0.38576263189315796, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 674234368}
{'loss': 2.9998, 'grad_norm': 0.3936232924461365, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 675282944}
{'loss': 3.1349, 'grad_norm': 0.3888678252696991, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 676331520}
{'loss': 2.9192, 'grad_norm': 0.31759846210479736, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 677380096}
{'loss': 3.1324, 'grad_norm': 0.3801535964012146, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 678428672}
{'loss': 3.1064, 'grad_norm': 0.36299699544906616, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 679477248}
{'loss': 3.2258, 'grad_norm': 0.36732324957847595, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 680525824}
{'loss': 3.2162, 'grad_norm': 0.42108356952667236, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 681574400}
{'loss': 3.2189, 'grad_norm': 0.4113474190235138, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 682622976}
{'loss': 3.0585, 'grad_norm': 0.39936116337776184, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 683671552}
{'loss': 3.0693, 'grad_norm': 0.35424771904945374, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 684720128}
{'loss': 3.1134, 'grad_norm': 0.3333597183227539, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 685768704}
{'loss': 3.0536, 'grad_norm': 0.37569180130958557, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 686817280}
{'loss': 3.1396, 'grad_norm': 0.33836638927459717, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 687865856}
{'loss': 3.1353, 'grad_norm': 0.31407052278518677, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 688914432}
{'loss': 2.9977, 'grad_norm': 0.34316036105155945, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 689963008}
{'loss': 3.1683, 'grad_norm': 0.3779186010360718, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 691011584}
{'loss': 2.9567, 'grad_norm': 0.3414095342159271, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 692060160}
{'loss': 3.0806, 'grad_norm': 0.31614938378334045, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 693108736}
{'loss': 3.0975, 'grad_norm': 0.35552725195884705, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 694157312}
{'loss': 3.0241, 'grad_norm': 0.38724133372306824, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 695205888}
{'loss': 3.0701, 'grad_norm': 0.3581823408603668, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 696254464}
{'loss': 3.0222, 'grad_norm': 0.3632317781448364, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 697303040}
{'loss': 3.0188, 'grad_norm': 0.40560677647590637, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 698351616}
{'loss': 3.106, 'grad_norm': 0.3953804075717926, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 699400192}
{'loss': 3.1552, 'grad_norm': 0.40652376413345337, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 700448768}
{'loss': 2.8893, 'grad_norm': 0.3625616133213043, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 701497344}
{'loss': 2.9183, 'grad_norm': 0.3450768291950226, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 702545920}
{'loss': 2.9828, 'grad_norm': 0.36742398142814636, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 703594496}
{'loss': 3.0327, 'grad_norm': 0.3611394762992859, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 704643072}
{'loss': 3.1466, 'grad_norm': 0.3593210279941559, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 705691648}
{'loss': 3.0163, 'grad_norm': 0.3994838297367096, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 706740224}
{'loss': 3.0563, 'grad_norm': 0.41202738881111145, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 707788800}
{'loss': 3.0912, 'grad_norm': 0.3404449224472046, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 708837376}
{'loss': 3.0108, 'grad_norm': 0.3745224177837372, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 709885952}
{'loss': 3.0864, 'grad_norm': 0.4320204555988312, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 710934528}
{'loss': 3.0387, 'grad_norm': 0.34649956226348877, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 711983104}
{'loss': 3.013, 'grad_norm': 0.34744057059288025, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 713031680}
{'loss': 3.0985, 'grad_norm': 0.3638330101966858, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 714080256}
{'loss': 3.1498, 'grad_norm': 0.43823716044425964, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 715128832}
{'loss': 3.0366, 'grad_norm': 0.6364668011665344, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 716177408}
{'loss': 2.9614, 'grad_norm': 0.6294976472854614, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 717225984}
{'loss': 3.0619, 'grad_norm': 0.5871465802192688, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 718274560}
{'loss': 3.1489, 'grad_norm': 0.7779986262321472, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 719323136}
{'loss': 3.1331, 'grad_norm': 1.102079153060913, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 720371712}
{'loss': 3.1423, 'grad_norm': 0.6352481245994568, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 721420288}
{'loss': 3.1509, 'grad_norm': 0.5698557496070862, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 722468864}
{'loss': 2.6683, 'grad_norm': 0.501290500164032, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 723517440}
{'loss': 3.0334, 'grad_norm': 0.4512772560119629, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 724566016}
{'loss': 3.0485, 'grad_norm': 0.4409146308898926, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 725614592}
{'loss': 3.0154, 'grad_norm': 0.3902524411678314, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 726663168}
{'loss': 3.0742, 'grad_norm': 0.3692473769187927, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 727711744}
{'loss': 2.8306, 'grad_norm': 0.385005384683609, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 728760320}
{'loss': 2.9258, 'grad_norm': 0.37514418363571167, 'learning_rate': 0.001, 'epoch': 0.08, 'num_input_tokens_seen': 729808896}
{'loss': 3.0061, 'grad_norm': 0.42038342356681824, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 730857472}
{'loss': 3.0588, 'grad_norm': 0.40415653586387634, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 731906048}
{'loss': 2.9542, 'grad_norm': 0.38514354825019836, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 732954624}
{'loss': 2.9252, 'grad_norm': 0.3861909806728363, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 734003200}
{'loss': 2.8432, 'grad_norm': 0.40519189834594727, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 735051776}
{'loss': 2.9779, 'grad_norm': 0.37011685967445374, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 736100352}
{'loss': 2.9908, 'grad_norm': 0.34850460290908813, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 737148928}
{'loss': 2.9589, 'grad_norm': 0.371500700712204, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 738197504}
[2025-03-11 00:58:41 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 0faae356-e828-4cff-9a49-42b397431927)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_185.jsonl.zst
[2025-03-11 00:58:41 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-11 01:05:12 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 9557423f-6937-4f70-b50f-05b0c01f5bf3)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_4035.jsonl.zst
[2025-03-11 01:05:12 WARNING] Retrying in 1s [Retry 1/5].
{'eval_loss': 2.9496541023254395, 'eval_runtime': 714.5105, 'eval_samples_per_second': 22.93, 'eval_steps_per_second': 0.179, 'epoch': 0.09, 'num_input_tokens_seen': 738197504}
{'loss': 2.9029, 'grad_norm': 0.3044391870498657, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 739246080}
{'loss': 2.8536, 'grad_norm': 0.34875407814979553, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 740294656}
{'loss': 2.8478, 'grad_norm': 0.4568244516849518, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 741343232}
{'loss': 3.1164, 'grad_norm': 0.44005003571510315, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 742391808}
{'loss': 2.8584, 'grad_norm': 0.39490336179733276, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 743440384}
{'loss': 3.0681, 'grad_norm': 0.4427798092365265, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 744488960}
{'loss': 3.0315, 'grad_norm': 0.4771106243133545, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 745537536}
{'loss': 2.8794, 'grad_norm': 0.4624035656452179, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 746586112}
{'loss': 2.9624, 'grad_norm': 0.4244724214076996, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 747634688}
{'loss': 2.9925, 'grad_norm': 0.39176708459854126, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 748683264}
{'loss': 2.9753, 'grad_norm': 0.43686383962631226, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 749731840}
{'loss': 3.0718, 'grad_norm': 0.4536241590976715, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 750780416}
{'loss': 3.0065, 'grad_norm': 0.3421417772769928, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 751828992}
{'loss': 2.8965, 'grad_norm': 0.30937010049819946, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 752877568}
{'loss': 3.0347, 'grad_norm': 0.33371758460998535, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 753926144}
{'loss': 3.0133, 'grad_norm': 0.3285418450832367, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 754974720}
{'loss': 3.1219, 'grad_norm': 0.33177846670150757, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 756023296}
{'loss': 2.9354, 'grad_norm': 0.36487525701522827, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 757071872}
{'loss': 3.133, 'grad_norm': 0.35576146841049194, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 758120448}
{'loss': 2.9771, 'grad_norm': 0.4217855930328369, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 759169024}
{'loss': 2.9906, 'grad_norm': 0.4007001519203186, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 760217600}
{'loss': 3.0219, 'grad_norm': 0.36323100328445435, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 761266176}
{'loss': 2.89, 'grad_norm': 0.323297381401062, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 762314752}
{'loss': 2.8566, 'grad_norm': 0.3450233042240143, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 763363328}
{'loss': 3.0536, 'grad_norm': 0.36228489875793457, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 764411904}
{'loss': 2.9259, 'grad_norm': 0.3553276062011719, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 765460480}
{'loss': 2.8431, 'grad_norm': 0.37074941396713257, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 766509056}
{'loss': 3.0549, 'grad_norm': 0.4105451703071594, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 767557632}
{'loss': 2.8431, 'grad_norm': 0.4433744549751282, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 768606208}
{'loss': 2.9545, 'grad_norm': 0.4024113416671753, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 769654784}
{'loss': 2.9237, 'grad_norm': 0.3534025549888611, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 770703360}
{'loss': 2.9306, 'grad_norm': 0.3788505792617798, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 771751936}
{'loss': 2.9218, 'grad_norm': 0.3302527666091919, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 772800512}
{'loss': 3.0647, 'grad_norm': 0.36651748418807983, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 773849088}
{'loss': 3.0289, 'grad_norm': 0.35838624835014343, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 774897664}
{'loss': 2.9157, 'grad_norm': 0.34652525186538696, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 775946240}
{'loss': 2.9358, 'grad_norm': 0.37369009852409363, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 776994816}
{'loss': 3.0725, 'grad_norm': 0.37748783826828003, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 778043392}
{'loss': 2.8444, 'grad_norm': 0.339287132024765, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 779091968}
{'loss': 2.859, 'grad_norm': 0.3415367305278778, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 780140544}
{'loss': 2.9334, 'grad_norm': 0.3661401569843292, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 781189120}
{'loss': 3.0287, 'grad_norm': 0.3512025773525238, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 782237696}
{'loss': 2.8093, 'grad_norm': 0.3412944972515106, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 783286272}
{'loss': 2.9112, 'grad_norm': 0.35280412435531616, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 784334848}
{'loss': 2.8939, 'grad_norm': 0.3652521073818207, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 785383424}
{'loss': 2.961, 'grad_norm': 0.3336659371852875, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 786432000}
{'loss': 2.9547, 'grad_norm': 0.3242711126804352, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 787480576}
{'loss': 2.8035, 'grad_norm': 0.3276830017566681, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 788529152}
{'loss': 2.9639, 'grad_norm': 0.32558611035346985, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 789577728}
{'loss': 2.9981, 'grad_norm': 0.32141759991645813, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 790626304}
{'loss': 2.8053, 'grad_norm': 0.33697575330734253, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 791674880}
{'loss': 2.9265, 'grad_norm': 0.3305177092552185, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 792723456}
{'loss': 2.9357, 'grad_norm': 0.3303467035293579, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 793772032}
{'loss': 2.9209, 'grad_norm': 0.33826348185539246, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 794820608}
{'loss': 3.0134, 'grad_norm': 0.3682444393634796, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 795869184}
{'loss': 2.8786, 'grad_norm': 0.364545613527298, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 796917760}
{'loss': 3.0202, 'grad_norm': 0.4031524360179901, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 797966336}
{'loss': 2.4912, 'grad_norm': 0.40752920508384705, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 799014912}
{'loss': 2.9311, 'grad_norm': 0.36912065744400024, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 800063488}
{'loss': 2.8768, 'grad_norm': 0.3906254172325134, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 801112064}
{'loss': 2.8677, 'grad_norm': 0.3680756092071533, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 802160640}
{'loss': 2.967, 'grad_norm': 0.42479801177978516, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 803209216}
{'loss': 3.0138, 'grad_norm': 0.4966808259487152, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 804257792}
{'loss': 2.9186, 'grad_norm': 0.413562536239624, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 805306368}
{'eval_loss': 2.8718671798706055, 'eval_runtime': 1149.5487, 'eval_samples_per_second': 14.253, 'eval_steps_per_second': 0.111, 'epoch': 0.09, 'num_input_tokens_seen': 805306368}
{'loss': 2.8717, 'grad_norm': 0.3343268632888794, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 806354944}
{'loss': 3.0123, 'grad_norm': 0.42326104640960693, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 807403520}
{'loss': 2.9691, 'grad_norm': 0.35408785939216614, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 808452096}
{'loss': 2.8862, 'grad_norm': 0.35168665647506714, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 809500672}
{'loss': 2.9754, 'grad_norm': 0.3385300934314728, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 810549248}
{'loss': 2.751, 'grad_norm': 0.36974239349365234, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 811597824}
{'loss': 2.8481, 'grad_norm': 0.3535187244415283, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 812646400}
{'loss': 2.9605, 'grad_norm': 0.39851564168930054, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 813694976}
{'loss': 2.9251, 'grad_norm': 0.35983574390411377, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 814743552}
{'loss': 2.8766, 'grad_norm': 0.34153202176094055, 'learning_rate': 0.001, 'epoch': 0.09, 'num_input_tokens_seen': 815792128}
{'loss': 2.9205, 'grad_norm': 0.3700859546661377, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 816840704}
{'loss': 2.7621, 'grad_norm': 0.3954067528247833, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 817889280}
{'loss': 2.886, 'grad_norm': 0.4191531538963318, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 818937856}
{'loss': 2.9203, 'grad_norm': 0.3315434157848358, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 819986432}
{'loss': 2.9563, 'grad_norm': 0.3308311700820923, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 821035008}
{'loss': 2.9391, 'grad_norm': 0.3073643445968628, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 822083584}
{'loss': 2.7197, 'grad_norm': 0.3343094289302826, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 823132160}
{'loss': 2.909, 'grad_norm': 0.31464704871177673, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 824180736}
{'loss': 2.8581, 'grad_norm': 0.40213140845298767, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 825229312}
{'loss': 2.9224, 'grad_norm': 0.36158621311187744, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 826277888}
{'loss': 2.985, 'grad_norm': 0.3831183910369873, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 827326464}
{'loss': 2.8964, 'grad_norm': 0.3219353258609772, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 828375040}
{'loss': 3.0832, 'grad_norm': 0.31743234395980835, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 829423616}
{'loss': 2.9602, 'grad_norm': 0.3629371225833893, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 830472192}
{'loss': 2.8327, 'grad_norm': 0.3800980746746063, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 831520768}
{'loss': 2.8298, 'grad_norm': 0.3349006772041321, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 832569344}
{'loss': 2.9633, 'grad_norm': 0.3282972276210785, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 833617920}
{'loss': 2.9234, 'grad_norm': 0.3283899128437042, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 834666496}
{'loss': 2.9754, 'grad_norm': 0.33885031938552856, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 835715072}
{'loss': 2.8825, 'grad_norm': 0.3113347589969635, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 836763648}
{'loss': 2.9483, 'grad_norm': 0.3759271204471588, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 837812224}
{'loss': 2.8577, 'grad_norm': 0.38608986139297485, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 838860800}
{'loss': 2.6639, 'grad_norm': 0.3253604471683502, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 839909376}
{'loss': 2.8295, 'grad_norm': 0.31234994530677795, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 840957952}
{'loss': 2.9323, 'grad_norm': 0.37187162041664124, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 842006528}
{'loss': 3.2357, 'grad_norm': 0.5417175889015198, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 843055104}
{'loss': 2.8982, 'grad_norm': 0.6133915781974792, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 844103680}
{'loss': 2.928, 'grad_norm': 0.7637872099876404, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 845152256}
{'loss': 2.9283, 'grad_norm': 0.7322977781295776, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 846200832}
{'loss': 2.8209, 'grad_norm': 0.5112255215644836, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 847249408}
{'loss': 2.8696, 'grad_norm': 0.49990609288215637, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 848297984}
{'loss': 2.9193, 'grad_norm': 0.4511178135871887, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 849346560}
{'loss': 2.9658, 'grad_norm': 0.4653412997722626, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 850395136}
{'loss': 2.889, 'grad_norm': 0.3913695812225342, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 851443712}
{'loss': 2.9534, 'grad_norm': 0.39285045862197876, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 852492288}
{'loss': 2.8341, 'grad_norm': 0.5052099227905273, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 853540864}
{'loss': 3.0436, 'grad_norm': 0.5978823900222778, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 854589440}
{'loss': 2.9484, 'grad_norm': 0.4584784507751465, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 855638016}
{'loss': 2.8786, 'grad_norm': 0.40823692083358765, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 856686592}
{'loss': 2.942, 'grad_norm': 0.4448293447494507, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 857735168}
{'loss': 2.9347, 'grad_norm': 0.4112764596939087, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 858783744}
{'loss': 2.8359, 'grad_norm': 0.3826068341732025, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 859832320}
{'loss': 2.9277, 'grad_norm': 0.37165558338165283, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 860880896}
{'loss': 2.6527, 'grad_norm': 0.4285834729671478, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 861929472}
{'loss': 2.8451, 'grad_norm': 0.36497727036476135, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 862978048}
{'loss': 2.9039, 'grad_norm': 0.35966625809669495, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 864026624}
{'loss': 2.9268, 'grad_norm': 0.3529391586780548, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 865075200}
{'loss': 2.9953, 'grad_norm': 0.3455546498298645, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 866123776}
{'loss': 2.9307, 'grad_norm': 0.3788530230522156, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 867172352}
{'loss': 2.9448, 'grad_norm': 0.35837656259536743, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 868220928}
{'loss': 2.9937, 'grad_norm': 0.3842633366584778, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 869269504}
{'loss': 2.8324, 'grad_norm': 0.32774215936660767, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 870318080}
{'loss': 2.8613, 'grad_norm': 0.327158659696579, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 871366656}
{'loss': 2.7653, 'grad_norm': 0.3515920639038086, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 872415232}
[2025-03-11 02:50:38 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 939d1d36-c607-4d3c-a0a0-8e447579340b)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_165.jsonl.zst
[2025-03-11 02:50:39 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-11 02:58:37 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 0b99bfd1-07ae-46db-81fa-fc6ef0eabdbc)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_1529.jsonl.zst
[2025-03-11 02:58:37 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-11 03:00:11 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: c208d1bb-5d13-45d2-9a01-1d5a2defa598)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk5/example_holdout_4562.jsonl.zst
[2025-03-11 03:00:11 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-11 03:01:14 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 2bf98b5c-473b-4e00-aca2-b152efddb992)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk3/example_holdout_4414.jsonl.zst
[2025-03-11 03:01:14 WARNING] Retrying in 1s [Retry 1/5].
{'eval_loss': 2.816462278366089, 'eval_runtime': 954.8041, 'eval_samples_per_second': 17.16, 'eval_steps_per_second': 0.134, 'epoch': 0.1, 'num_input_tokens_seen': 872415232}
{'loss': 2.867, 'grad_norm': 0.3173666000366211, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 873463808}
{'loss': 2.8701, 'grad_norm': 0.3399354815483093, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 874512384}
{'loss': 2.8575, 'grad_norm': 0.36704689264297485, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 875560960}
{'loss': 2.9582, 'grad_norm': 0.33231136202812195, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 876609536}
{'loss': 2.7719, 'grad_norm': 0.34316956996917725, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 877658112}
{'loss': 2.8915, 'grad_norm': 0.3483976423740387, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 878706688}
{'loss': 2.7566, 'grad_norm': 0.3104913532733917, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 879755264}
{'loss': 3.0013, 'grad_norm': 0.38844239711761475, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 880803840}
{'loss': 2.5568, 'grad_norm': 0.40875244140625, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 881852416}
{'loss': 2.8336, 'grad_norm': 0.3538399934768677, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 882900992}
{'loss': 2.9391, 'grad_norm': 0.3494492471218109, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 883949568}
{'loss': 2.8535, 'grad_norm': 0.3472343981266022, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 884998144}
{'loss': 2.9836, 'grad_norm': 0.34867390990257263, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 886046720}
{'loss': 2.8416, 'grad_norm': 0.3527415096759796, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 887095296}
{'loss': 2.8756, 'grad_norm': 0.3338777422904968, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 888143872}
{'loss': 2.8428, 'grad_norm': 0.3345812261104584, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 889192448}
{'loss': 2.8977, 'grad_norm': 0.31487980484962463, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 890241024}
{'loss': 2.9543, 'grad_norm': 0.3655254542827606, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 891289600}
{'loss': 2.9423, 'grad_norm': 0.33075806498527527, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 892338176}
{'loss': 2.9001, 'grad_norm': 0.34644609689712524, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 893386752}
{'loss': 2.9029, 'grad_norm': 0.39070528745651245, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 894435328}
{'loss': 2.9101, 'grad_norm': 0.39556533098220825, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 895483904}
{'loss': 2.8119, 'grad_norm': 0.39002978801727295, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 896532480}
{'loss': 3.0102, 'grad_norm': 0.37797507643699646, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 897581056}
{'loss': 2.666, 'grad_norm': 0.4306756258010864, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 898629632}
{'loss': 2.9257, 'grad_norm': 0.4526049494743347, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 899678208}
{'loss': 2.8196, 'grad_norm': 0.3978416621685028, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 900726784}
{'loss': 2.9057, 'grad_norm': 0.3925896883010864, 'learning_rate': 0.001, 'epoch': 0.1, 'num_input_tokens_seen': 901775360}
{'loss': 3.0017, 'grad_norm': 0.45828214287757874, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 902823936}
{'loss': 2.89, 'grad_norm': 0.4745008647441864, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 903872512}
{'loss': 2.7335, 'grad_norm': 0.4270082116127014, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 904921088}
{'loss': 2.8234, 'grad_norm': 0.38832950592041016, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 905969664}
{'loss': 2.8618, 'grad_norm': 0.3907729387283325, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 907018240}
{'loss': 2.8703, 'grad_norm': 0.368655264377594, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 908066816}
{'loss': 2.8321, 'grad_norm': 0.41538506746292114, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 909115392}
{'loss': 2.886, 'grad_norm': 0.41877180337905884, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 910163968}
{'loss': 2.6224, 'grad_norm': 0.33238673210144043, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 911212544}
{'loss': 2.8617, 'grad_norm': 0.4095931351184845, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 912261120}
{'loss': 2.8172, 'grad_norm': 0.41708603501319885, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 913309696}
{'loss': 2.7658, 'grad_norm': 0.37449270486831665, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 914358272}
{'loss': 2.9042, 'grad_norm': 0.3935737609863281, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 915406848}
{'loss': 2.7612, 'grad_norm': 0.3586251735687256, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 916455424}
{'loss': 2.8785, 'grad_norm': 0.3712047338485718, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 917504000}
{'loss': 2.739, 'grad_norm': 0.37707045674324036, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 918552576}
{'loss': 2.8372, 'grad_norm': 0.3432702422142029, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 919601152}
{'loss': 2.5638, 'grad_norm': 0.3493041396141052, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 920649728}
{'loss': 2.8759, 'grad_norm': 0.3401539623737335, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 921698304}
{'loss': 3.0048, 'grad_norm': 0.4632040858268738, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 922746880}
{'loss': 2.9394, 'grad_norm': 0.4968065023422241, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 923795456}
{'loss': 2.8441, 'grad_norm': 0.5426673889160156, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 924844032}
{'loss': 2.9975, 'grad_norm': 0.4630672037601471, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 925892608}
{'loss': 2.9584, 'grad_norm': 0.38806748390197754, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 926941184}
{'loss': 2.8904, 'grad_norm': 0.39797642827033997, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 927989760}
{'loss': 2.5774, 'grad_norm': 0.4063512980937958, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 929038336}
{'loss': 2.812, 'grad_norm': 0.3161136209964752, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 930086912}
{'loss': 2.7483, 'grad_norm': 0.3628361225128174, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 931135488}
{'loss': 2.7916, 'grad_norm': 0.37376269698143005, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 932184064}
{'loss': 2.7985, 'grad_norm': 0.3399117887020111, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 933232640}
{'loss': 2.7107, 'grad_norm': 0.3453179597854614, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 934281216}
{'loss': 2.9254, 'grad_norm': 0.39461833238601685, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 935329792}
{'loss': 2.8487, 'grad_norm': 0.3668413460254669, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 936378368}
{'loss': 2.7928, 'grad_norm': 0.28304487466812134, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 937426944}
{'loss': 2.8503, 'grad_norm': 0.35816267132759094, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 938475520}
{'loss': 3.0328, 'grad_norm': 0.3540339469909668, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 939524096}
[2025-03-11 03:46:08 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 3b8321b9-2d88-4bfa-9eca-b201c444cba3)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk5/example_holdout_405.jsonl.zst
[2025-03-11 03:46:08 WARNING] Retrying in 1s [Retry 1/5].
[2025-03-11 03:53:27 WARNING] '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: a98a238a-c0a4-4295-8502-316a89a7ae29)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk1/example_holdout_2524.jsonl.zst
[2025-03-11 03:53:27 WARNING] Retrying in 1s [Retry 1/5].
{'eval_loss': 2.7651162147521973, 'eval_runtime': 687.962, 'eval_samples_per_second': 23.815, 'eval_steps_per_second': 0.186, 'epoch': 0.11, 'num_input_tokens_seen': 939524096}
{'loss': 2.9368, 'grad_norm': 0.34962671995162964, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 940572672}
{'loss': 2.3627, 'grad_norm': 0.37516310811042786, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 941621248}
{'loss': 2.8854, 'grad_norm': 0.3487492501735687, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 942669824}
{'loss': 2.7892, 'grad_norm': 0.37180987000465393, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 943718400}
{'loss': 2.8067, 'grad_norm': 0.3387952744960785, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 944766976}
{'loss': 2.841, 'grad_norm': 0.32076528668403625, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 945815552}
{'loss': 2.7965, 'grad_norm': 0.3348572552204132, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 946864128}
{'loss': 2.6788, 'grad_norm': 0.3531329929828644, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 947912704}
{'loss': 2.7276, 'grad_norm': 0.300353467464447, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 948961280}
{'loss': 2.8189, 'grad_norm': 0.3258875012397766, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 950009856}
{'loss': 2.8388, 'grad_norm': 0.3434987962245941, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 951058432}
{'loss': 2.856, 'grad_norm': 0.33045029640197754, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 952107008}
{'loss': 2.658, 'grad_norm': 0.34896957874298096, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 953155584}
{'loss': 2.8484, 'grad_norm': 0.3819083273410797, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 954204160}
{'loss': 2.8402, 'grad_norm': 0.39541998505592346, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 955252736}
{'loss': 2.8281, 'grad_norm': 0.3843367397785187, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 956301312}
{'loss': 2.8339, 'grad_norm': 0.4067714214324951, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 957349888}
{'loss': 2.8693, 'grad_norm': 0.3071018159389496, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 958398464}
{'loss': 2.6747, 'grad_norm': 0.3676702380180359, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 959447040}
{'loss': 2.6961, 'grad_norm': 0.357799232006073, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 960495616}
{'loss': 2.7944, 'grad_norm': 0.318391352891922, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 961544192}
{'loss': 2.8084, 'grad_norm': 0.32000190019607544, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 962592768}
{'loss': 2.8024, 'grad_norm': 0.3250137269496918, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 963641344}
{'loss': 2.7951, 'grad_norm': 0.33021438121795654, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 964689920}
{'loss': 2.8069, 'grad_norm': 0.3257495164871216, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 965738496}
{'loss': 2.8148, 'grad_norm': 0.3608018159866333, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 966787072}
[2025-03-11 04:13:12 WARNING] '(ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), '(Request ID: 36a7cc72-4605-416a-8742-59488d719150)')' thrown while requesting GET https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk1/example_train_5267.jsonl.zst
[2025-03-11 04:13:12 WARNING] Retrying in 1s [Retry 1/5].
{'loss': 2.8089, 'grad_norm': 0.3657573163509369, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 967835648}
{'loss': 2.8243, 'grad_norm': 0.3791966736316681, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 968884224}
{'loss': 2.6837, 'grad_norm': 0.4036826193332672, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 969932800}
{'loss': 2.6694, 'grad_norm': 0.34643635153770447, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 970981376}
{'loss': 2.8455, 'grad_norm': 0.35321497917175293, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 972029952}
{'loss': 2.5156, 'grad_norm': 0.3488744795322418, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 973078528}
{'loss': 2.7185, 'grad_norm': 0.33396172523498535, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 974127104}
{'loss': 2.856, 'grad_norm': 0.36425134539604187, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 975175680}
{'loss': 2.7639, 'grad_norm': 0.34361588954925537, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 976224256}
{'loss': 2.7777, 'grad_norm': 0.45501893758773804, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 977272832}
{'loss': 2.8692, 'grad_norm': 0.4391760230064392, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 978321408}
{'loss': 2.7885, 'grad_norm': 0.385729044675827, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 979369984}
{'loss': 2.8622, 'grad_norm': 0.4122815728187561, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 980418560}
{'loss': 2.674, 'grad_norm': 0.3223947584629059, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 981467136}
{'loss': 2.7148, 'grad_norm': 0.39820024371147156, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 982515712}
{'loss': 2.6975, 'grad_norm': 0.38311144709587097, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 983564288}
{'loss': 2.8515, 'grad_norm': 0.4324709177017212, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 984612864}
{'loss': 2.5684, 'grad_norm': 0.3579341471195221, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 985661440}
{'loss': 2.9478, 'grad_norm': 0.4081536531448364, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 986710016}
{'loss': 2.7375, 'grad_norm': 0.4332145154476166, 'learning_rate': 0.001, 'epoch': 0.11, 'num_input_tokens_seen': 987758592}
{'loss': 2.7773, 'grad_norm': 0.43510711193084717, 'learning_rate': 0.001, 'epoch': 0.12, 'num_input_tokens_seen': 988807168}
...
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 1378, in _iter_from_urlpaths
    raise FileNotFoundError(urlpath)
FileNotFoundError: zstd://example_train_1215.jsonl::hf://datasets/cerebras/SlimPajama-627B@2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk9/example_train_1215.jsonl.zst

[bauwenst]

Two more today:

FileNotFoundError: zstd://example_holdout_5012.jsonl::hf://datasets/cerebras/SlimPajama-627B@2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk4/example_holdout_5012.jsonl.zst

and

FileNotFoundError: zstd://example_holdout_3073.jsonl::hf://datasets/cerebras/SlimPajama-627B@2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/validation/chunk2/example_holdout_3073.jsonl.zst

both of which exist on the hub (here and here).

[Rabona17]

I also observe the same thing when using streaming with DCLM dataset with 64 GPUs. I have tried export HF_DATASETS_STREAMING_PARALLELISM=1 but doesn't help.

[bauwenst]

Another error today, this time a 504 gateway timeout HfHubHTTPError. I have no idea if this is related, but I suspect that it is considering the setup is identical. Notably though, the two errors I posted yesterday were for evaluation (hence the holdout in the URLs) whereas today there was no problem doing that first evaluation, but now the train split failed.

...
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 2226, in __iter__
    for key, example in ex_iterable:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1499, in __iter__
    for x in self.ex_iterable:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1067, in __iter__
    yield from self._iter()
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1231, in _iter
    for key, transformed_example in iter_outputs():
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1207, in iter_outputs
    for i, key_example in inputs_iterator:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1111, in iter_inputs
    for key, example in iterator:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 371, in __iter__
    for key, pa_table in self.generate_tables_fn(**gen_kwags):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/packaged_modules/json/json.py", line 114, in _generate_tables
    with open(file, "rb") as f:
         ^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/streaming.py", line 75, in wrapper
    return function(*args, download_config=download_config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 948, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/fsspec/core.py", line 147, in open
    return self.__enter__()
           ^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/fsspec/core.py", line 105, in __enter__
    f = self.fs.open(self.path, mode=mode)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/fsspec/spec.py", line 1301, in open
    f = self._open(
        ^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/filesystems/compression.py", line 85, in _open
    return self._open_with_fsspec().open()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/fsspec/core.py", line 147, in open
    return self.__enter__()
           ^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/fsspec/core.py", line 105, in __enter__
    f = self.fs.open(self.path, mode=mode)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/fsspec/spec.py", line 1301, in open
    f = self._open(
        ^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 234, in _open
    return HfFileSystemFile(self, path, mode=mode, revision=revision, block_size=block_size, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 691, in __init__
    self.details = fs.info(self.resolved_path.unresolve(), expand_info=False)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 524, in info
    self.ls(parent_path, expand_info=False)
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 284, in ls
    out = self._ls_tree(path, refresh=refresh, revision=revision, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 375, in _ls_tree
    for path_info in tree:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 3080, in list_repo_tree
    for path_info in paginate(path=tree_url, headers=headers, params={"recursive": recursive, "expand": expand}):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/utils/_pagination.py", line 46, in paginate
    hf_raise_for_status(r)
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 477, in hf_raise_for_status
    raise _format(HfHubHTTPError, str(e), response) from e
huggingface_hub.errors.HfHubHTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/cerebras/SlimPajama-627B/tree/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train%2Fchunk8?recursive=False&expand=False&cursor=ZXlKbWFXeGxYMjVoYldVaU9pSjBjbUZwYmk5amFIVnVhemd2WlhoaGJYQnNaVjkwY21GcGJsOHpOams0TG1wemIyNXNMbnB6ZENKOTozMDAw

[bauwenst]

Another one today:

FileNotFoundError: zstd://example_train_4985.jsonl::hf://datasets/cerebras/SlimPajama-627B@2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk5/example_train_4985.jsonl.zst

[Vectorrent]

This is a constant issue, and has been for six months, at least. Currently, half of my streaming datasets are failing with errors like this.

Muennighoff/natural-instructions:

  File "/home/crow/repos/praxis/.venv/lib/python3.13/site-packages/datasets/utils/file_utils.py", line 1379, in _iter_from_urlpaths
    raise FileNotFoundError(urlpath)
FileNotFoundError: hf://datasets/Muennighoff/natural-instructions@a29a9757125f4bb1c26445ad0d2ef7d9b2cc9c4c/train/task343_winomt_classification_profession_anti_train.jsonl

open-phi/textbooks:

  File "/home/crow/repos/praxis/.venv/lib/python3.13/site-packages/datasets/utils/file_utils.py", line 1379, in _iter_from_urlpaths
    raise FileNotFoundError(urlpath)
FileNotFoundError: hf://datasets/open-phi/textbooks@292aaae99cbecacad50f692d7327887f05dacaf2/data/train-00000-of-00001-b513d9e388d56453.parquet

HuggingFaceTB/smoltalk:

  File "/home/crow/repos/praxis/.venv/lib/python3.13/site-packages/datasets/utils/file_utils.py", line 1379, in _iter_from_urlpaths
    raise FileNotFoundError(urlpath)
FileNotFoundError: hf://datasets/HuggingFaceTB/smoltalk@5feaf2fd3ffca7c237fc38d1861bc30365d48ffa/data/all/train-00003-of-00009.parquet

[bauwenst]

This line of issues has now been going on since April of 2024. It is now August of 2025. I opened this particular issue almost five months ago. Our training runs are still failing. It is apparently too difficult for datasets to reliable fetch some text from some server. This is by far the biggest bottleneck in our research and the amount of time spent on setbacks caused by this is unimaginable.

A week ago:

  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 2361, in __iter__
    generator=generator, features=features, gen_kwargs=gen_kwargs, streaming=True, split=split
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1558, in __iter__
    )
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1107, in __iter__
    # If `batched`, first build the batch, if `batch_size` is None or <=0, then the batch is the whole dataset
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1286, in _iter
    iterator = _convert_to_arrow(
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1267, in iter_outputs
    num_examples_to_skip -= 1
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1156, in iter_inputs
    additional_args = ()
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 397, in __iter__
    shard_example_idx_start = self._state_dict["shard_example_idx"] if self._state_dict else 0
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/packaged_modules/json/json.py", line 99, in _generate_tables
    for file_idx, file in enumerate(itertools.chain.from_iterable(files)):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/track.py", line 49, in __iter__
    for x in self.generator(*self.args):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 1359, in _iter_from_urlpaths
    cls, urlpaths: Union[str, list[str]], download_config: Optional[DownloadConfig] = None
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: zstd://example_train_1820.jsonl::hf://datasets/cerebras/SlimPajama-627B@2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk2/example_train_1820.jsonl.zst

Today:

  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 2270, in __iter__
    for key, example in ex_iterable:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1535, in __iter__
    for x in self.ex_iterable:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1084, in __iter__
    yield from self._iter()
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1263, in _iter
    for key, transformed_example in outputs:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1244, in iter_outputs
    for i, key_example in inputs_iterator:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1133, in iter_inputs
    for key, example in iterator:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 374, in __iter__
    for key, pa_table in self.generate_tables_fn(**gen_kwags):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/packaged_modules/json/json.py", line 99, in _generate_tables
    for file_idx, file in enumerate(itertools.chain.from_iterable(files)):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/track.py", line 49, in __iter__
    for x in self.generator(*self.args):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 1379, in _iter_from_urlpaths
    raise FileNotFoundError(urlpath)
FileNotFoundError: zstd://example_train_5054.jsonl::hf://datasets/cerebras/SlimPajama-627B@2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk1/example_train_5054.jsonl.zst

Seriously?