Llama3 changes #793

ebsmothers · 2024-04-18T16:01:40Z

No description provided.

pytorch-bot · 2024-04-18T16:01:43Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/793

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit ff63068 with merge base 83785f9 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

kartikayk · 2024-04-18T16:15:37Z

docs/source/api_ref_models.rst

+
+.. code-block:: bash
+
+    tune download meta-llama/Llama-3-8b-hf --hf-token <ACCESS_TOKEN>


Need to update this

joecummings · 2024-04-18T16:22:58Z

docs/source/tutorials/llama3.rst

+
+.. code-block:: bash
+
+    tune download meta-llama/Llama-3-8b-hf \


joecummings · 2024-04-18T16:24:03Z

recipes/configs/llama3/8B_full.yaml

+# Tokenizer
+tokenizer:
+  _component_: torchtune.models.llama3.llama3_tokenizer
+  path: /tmp/Llama-3-8b-hf/tokenizer.model


Suggested change

path: /tmp/Llama-3-8b-hf/tokenizer.model

path: /tmp/Meta-Llama-3-8B/original/tokenizer.model

kartikayk · 2024-04-18T16:36:42Z

recipes/configs/llama3/8B_full_single_device.yaml

+checkpointer:
+  _component_: torchtune.utils.FullModelHFCheckpointer
+  checkpoint_dir: /tmp/Llama-3-8b-hf/
+  checkpoint_files: [
+    pytorch_model-00001-of-00003.bin,
+    pytorch_model-00002-of-00003.bin,
+    pytorch_model-00003-of-00003.bin,
+  ]
+  recipe_checkpoint: null
+  output_dir: /tmp/Llama-3-8b-hf/
+  model_type: LLAMA3
+resume_from_checkpoint: False


Tested this:

checkpointer: _component_: torchtune.utils.FullModelMetaCheckpointer checkpoint_dir: /tmp/Meta-Llama-3-8B checkpoint_files: [ consolidated.00.pth ] recipe_checkpoint: null output_dir: /tmp/Meta-Llama-3-8B model_type: LLAMA3 resume_from_checkpoint: False

kartikayk · 2024-04-18T16:39:06Z

docs/source/tutorials/llama3.rst

+.. code-block:: yaml
+
+    checkpointer:
+        _component_: torchtune.utils.FullModelHFCheckpointer


Suggested change

_component_: torchtune.utils.FullModelHFCheckpointer

_component_: torchtune.utils.FullModelMetaCheckpointer

kartikayk · 2024-04-18T16:43:33Z

recipes/configs/llama3/8B_full.yaml

+# Tokenizer
+tokenizer:
+  _component_: torchtune.models.llama3.llama3_tokenizer
+  path: /tmp/Llama-3-8b-hf/original/tokenizer.model


Suggested change

path: /tmp/Llama-3-8b-hf/original/tokenizer.model

path: /tmp/Meta-Llama-3-8B/original/tokenizer.model

kartikayk · 2024-04-18T16:43:45Z

recipes/configs/llama3/8B_full.yaml

+
+checkpointer:
+  _component_: torchtune.utils.FullModelMetaCheckpointer
+  checkpoint_dir: /tmp/Llama-3-8b-hf/original/


Suggested change

checkpoint_dir: /tmp/Llama-3-8b-hf/original/

checkpoint_dir: /tmp/Meta-Llama-3-8B/original/

kartikayk · 2024-04-18T16:43:55Z

recipes/configs/llama3/8B_full.yaml

+    consolidated.00.pth
+  ]
+  recipe_checkpoint: null
+  output_dir: /tmp/Llama-3-8b-hf/


Suggested change

output_dir: /tmp/Llama-3-8b-hf/

output_dir: /tmp/Meta-Llama-3-8B

kartikayk · 2024-04-18T16:44:11Z

recipes/configs/llama3/8B_full_single_device.yaml

+# Tokenizer
+tokenizer:
+  _component_: torchtune.models.llama3.llama3_tokenizer
+  path: /tmp/Llama-3-8b-hf/original/tokenizer.model


Suggested change

path: /tmp/Llama-3-8b-hf/original/tokenizer.model

path: /tmp/Meta-Llama-3-8B/original/tokenizer.model

kartikayk · 2024-04-18T16:44:34Z

recipes/configs/llama3/8B_full_single_device.yaml

+
+checkpointer:
+  _component_: torchtune.utils.FullModelMetaCheckpointer
+  checkpoint_dir: /tmp/Llama-3-8b-hf/original/


Suggested change

checkpoint_dir: /tmp/Llama-3-8b-hf/original/

checkpoint_dir: /tmp/Meta-Llama-3-8B/original/

kartikayk · 2024-04-18T16:44:44Z

recipes/configs/llama3/8B_full_single_device.yaml

+    consolidated.00.pth
+  ]
+  recipe_checkpoint: null
+  output_dir: /tmp/Llama-3-8b-hf/


Suggested change

output_dir: /tmp/Llama-3-8b-hf/

output_dir: /tmp/Meta-Llama-3-8B

kartikayk · 2024-04-18T16:45:38Z

recipes/configs/llama3/8B_lora.yaml

+
+checkpointer:
+  _component_: torchtune.utils.FullModelMetaCheckpointer
+  checkpoint_dir: /tmp/Llama-3-8b-hf/original/


Suggested change

checkpoint_dir: /tmp/Llama-3-8b-hf/original/

checkpoint_dir: /tmp/Meta-Llama-3-8B/original/

kartikayk · 2024-04-18T16:45:49Z

recipes/configs/llama3/8B_lora.yaml

+    consolidated.00.pth
+  ]
+  recipe_checkpoint: null
+  output_dir: /tmp/Llama-3-8b-hf/


Suggested change

output_dir: /tmp/Llama-3-8b-hf/

output_dir: /tmp/Meta-Llama-3-8B

kartikayk · 2024-04-18T16:46:03Z

recipes/configs/llama3/8B_lora_single_device.yaml

+# Tokenizer
+tokenizer:
+  _component_: torchtune.models.llama3.llama3_tokenizer
+  path: /tmp/Llama-3-8b-hf/original/tokenizer.model


Suggested change

path: /tmp/Llama-3-8b-hf/original/tokenizer.model

path: /tmp/Meta-Llama-3-8B/original/tokenizer.model

kartikayk · 2024-04-18T16:46:13Z

recipes/configs/llama3/8B_lora_single_device.yaml

+
+checkpointer:
+  _component_: torchtune.utils.FullModelMetaCheckpointer
+  checkpoint_dir: /tmp/Llama-3-8b-hf/original/


Suggested change

checkpoint_dir: /tmp/Llama-3-8b-hf/original/

checkpoint_dir: /tmp/Meta-Llama-3-8B/original/

kartikayk · 2024-04-18T16:46:28Z

recipes/configs/llama3/8B_qlora_single_device.yaml

+# Tokenizer
+tokenizer:
+  _component_: torchtune.models.llama3.llama3_tokenizer
+  path: /tmp/Llama-3-8b-hf/original/tokenizer.model


Suggested change

path: /tmp/Llama-3-8b-hf/original/tokenizer.model

path: /tmp/Meta-Llama-3-8B/original/tokenizer.model

kartikayk · 2024-04-18T16:46:37Z

recipes/configs/llama3/8B_qlora_single_device.yaml

+
+checkpointer:
+  _component_: torchtune.utils.FullModelMetaCheckpointer
+  checkpoint_dir: /tmp/Llama-3-8b-hf/original/


Suggested change

checkpoint_dir: /tmp/Llama-3-8b-hf/original/

checkpoint_dir: /tmp/Meta-Llama-3-8B/original/

kartikayk · 2024-04-18T16:48:25Z

docs/source/tutorials/llama3.rst

+.. code-block:: yaml
+
+    checkpointer:
+        _component_: torchtune.utils.FullModelHFCheckpointer


Suggested change

_component_: torchtune.utils.FullModelHFCheckpointer

_component_: torchtune.utils.FullModelMetaCheckpointer

joecummings · 2024-04-18T16:57:22Z

docs/source/tutorials/llama3.rst

+---------------------------
+
+First, let's download the model from Hugging Face. You will need to follow the instructions
+on the `official Meta page <https://github.com/meta-llama/llama3/blob/main/README.md>`_ to gain access to the model.


Link to HF page for access? Much easier and straightforward.

Llama3 changes

90f6268

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 18, 2024

merge

8e9c8e1

kartikayk reviewed Apr 18, 2024

View reviewed changes

joecummings reviewed Apr 18, 2024

View reviewed changes

docs/source/tutorials/llama3.rst Outdated

.. code-block:: bash

tune download meta-llama/Llama-3-8b-hf \

Copy link

Contributor

joecummings Apr 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update

joecummings reviewed Apr 18, 2024

View reviewed changes

update paths

645551f

kartikayk reviewed Apr 18, 2024

View reviewed changes

_00 -> .00

939fa66

kartikayk reviewed Apr 18, 2024

View reviewed changes

ebsmothers added 2 commits April 18, 2024 09:40

remove max batch size

2dbf7d7

small fixes

5b2eaf3

kartikayk reviewed Apr 18, 2024

View reviewed changes

fix config

a93368e

joecummings reviewed Apr 18, 2024

View reviewed changes

address comments

ff63068

kartikayk approved these changes Apr 18, 2024

View reviewed changes

ebsmothers merged commit 20747cd into pytorch:main Apr 18, 2024
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Llama3 changes #793

Llama3 changes #793

ebsmothers commented Apr 18, 2024

pytorch-bot bot commented Apr 18, 2024 •

edited

kartikayk Apr 18, 2024

joecummings Apr 18, 2024

joecummings Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

kartikayk Apr 18, 2024

joecummings Apr 18, 2024


		.. code-block:: bash

		tune download meta-llama/Llama-3-8b-hf --hf-token <ACCESS_TOKEN>


		.. code-block:: bash

		tune download meta-llama/Llama-3-8b-hf \

	path: /tmp/Llama-3-8b-hf/tokenizer.model
	path: /tmp/Meta-Llama-3-8B/original/tokenizer.model

	_component_: torchtune.utils.FullModelHFCheckpointer
	_component_: torchtune.utils.FullModelMetaCheckpointer

	path: /tmp/Llama-3-8b-hf/original/tokenizer.model
	path: /tmp/Meta-Llama-3-8B/original/tokenizer.model

	checkpoint_dir: /tmp/Llama-3-8b-hf/original/
	checkpoint_dir: /tmp/Meta-Llama-3-8B/original/

	output_dir: /tmp/Llama-3-8b-hf/
	output_dir: /tmp/Meta-Llama-3-8B

Llama3 changes #793

Llama3 changes #793

Conversation

ebsmothers commented Apr 18, 2024

pytorch-bot bot commented Apr 18, 2024 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/793

✅ No Failures

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pytorch-bot bot commented Apr 18, 2024 •

edited