From d5c9151584f2398c1c0219b6c82fb5f36681db76 Mon Sep 17 00:00:00 2001
From: Michael Gschwind <61328285+mikekgfb@users.noreply.github.com>
Date: Mon, 9 Dec 2024 20:53:59 -0800
Subject: [PATCH 1/4] Update evaluation.md to include AOTI

Update evaluation.md to include running tests with AOTI
---
 torchchat/utils/docs/evaluation.md | 31 +++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/torchchat/utils/docs/evaluation.md b/torchchat/utils/docs/evaluation.md
index 8bc995ca7..3234df11d 100644
--- a/torchchat/utils/docs/evaluation.md
+++ b/torchchat/utils/docs/evaluation.md
@@ -21,24 +21,49 @@ library.
 
 The evaluation mode of `torchchat.py` script can be used to evaluate your language model on various tasks available in the `lm_eval` library such as "wikitext". You can specify the task(s) you want to evaluate using the `--tasks` option, and limit the evaluation using the `--limit` option. If no task is specified, the task will default to evaluating on "wikitext".
 
-**Examples**
+## Examples
+
+**Python**
 
 Running wikitext for 10 iterations
 ```
 python3 torchchat.py eval stories15M --tasks wikitext --limit 10
 ```
 
-Running an exported model
+Running wikitext with torch.compile for 10 iterations
+```
+python3 torchchat.py eval stories15M --compile --tasks wikitext --limit 10
+```
+
+Running multiple tasks and calling eval.py directly (with torch.compile):
+```
+python3 torchchat.py eval stories15M --compile --tasks wikitext hellaswag
+```
+
+**exported PTE**
+Running an exported model as PTE
 ```
 python3 torchchat.py export stories15M --output-pte-path stories15M.pte
 python3 torchchat.py eval stories15M --pte-path stories15M.pte
 ```
 
-Running multiple tasks and calling eval.py directly:
+Running multiple tasks and calling eval.py directly (with PTE):
 ```
 python3 torchchat.py eval stories15M --pte-path stories15M.pte --tasks wikitext hellaswag
 ```
 
+**exported AOTI**
+Running an exported model as PTE
+```
+python3 torchchat.py export stories15M --dtype fast16 --output-dso-path stories15M.so
+python3 torchchat.py eval stories15M --dtype fast16 --pte-path stories15M.so
+```
+
+Running multiple tasks and calling eval.py directly (with AOTI):
+```
+python3 torchchat.py eval stories15M --dso-path stories15M.so --tasks wikitext hellaswag
+```
+
 For more information and a list of tasks/metrics see [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).
 
 [end default]: end

From 70bd5dffc3ac64198034b5213b017a9294b696d0 Mon Sep 17 00:00:00 2001
From: Michael Gschwind <61328285+mikekgfb@users.noreply.github.com>
Date: Mon, 9 Dec 2024 20:56:36 -0800
Subject: [PATCH 2/4] Update evaluation.md

Fix typo
---
 torchchat/utils/docs/evaluation.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/torchchat/utils/docs/evaluation.md b/torchchat/utils/docs/evaluation.md
index 3234df11d..e429c36c9 100644
--- a/torchchat/utils/docs/evaluation.md
+++ b/torchchat/utils/docs/evaluation.md
@@ -41,7 +41,7 @@ python3 torchchat.py eval stories15M --compile --tasks wikitext hellaswag
 ```
 
 **exported PTE**
-Running an exported model as PTE
+Running an exported model with ExecuTorch (as PTE)
 ```
 python3 torchchat.py export stories15M --output-pte-path stories15M.pte
 python3 torchchat.py eval stories15M --pte-path stories15M.pte
@@ -53,10 +53,10 @@ python3 torchchat.py eval stories15M --pte-path stories15M.pte --tasks wikitext
 ```
 
 **exported AOTI**
-Running an exported model as PTE
+Running an exported model with AOT Inductor (DSO model)
 ```
 python3 torchchat.py export stories15M --dtype fast16 --output-dso-path stories15M.so
-python3 torchchat.py eval stories15M --dtype fast16 --pte-path stories15M.so
+python3 torchchat.py eval stories15M --dtype fast16 --dso-path stories15M.so
 ```
 
 Running multiple tasks and calling eval.py directly (with AOTI):

From 36d2765fb8f9041b8b9931435986c7f7a68217fa Mon Sep 17 00:00:00 2001
From: Michael Gschwind <61328285+mikekgfb@users.noreply.github.com>
Date: Mon, 9 Dec 2024 20:57:56 -0800
Subject: [PATCH 3/4] Formatting evaluation.md

Fix formatting
---
 torchchat/utils/docs/evaluation.md | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/torchchat/utils/docs/evaluation.md b/torchchat/utils/docs/evaluation.md
index e429c36c9..1ebe24248 100644
--- a/torchchat/utils/docs/evaluation.md
+++ b/torchchat/utils/docs/evaluation.md
@@ -23,7 +23,7 @@ The evaluation mode of `torchchat.py` script can be used to evaluate your langua
 
 ## Examples
 
-**Python**
+### Python
 
 Running wikitext for 10 iterations
 ```
@@ -40,7 +40,8 @@ Running multiple tasks and calling eval.py directly (with torch.compile):
 python3 torchchat.py eval stories15M --compile --tasks wikitext hellaswag
 ```
 
-**exported PTE**
+### exported PTE
+
 Running an exported model with ExecuTorch (as PTE)
 ```
 python3 torchchat.py export stories15M --output-pte-path stories15M.pte
@@ -52,7 +53,8 @@ Running multiple tasks and calling eval.py directly (with PTE):
 python3 torchchat.py eval stories15M --pte-path stories15M.pte --tasks wikitext hellaswag
 ```
 
-**exported AOTI**
+### exported AOTI
+
 Running an exported model with AOT Inductor (DSO model)
 ```
 python3 torchchat.py export stories15M --dtype fast16 --output-dso-path stories15M.so

From 4368bdbcfe731e2dab482a262541aac2f7728c8b Mon Sep 17 00:00:00 2001
From: Michael Gschwind <61328285+mikekgfb@users.noreply.github.com>
Date: Mon, 9 Dec 2024 20:59:56 -0800
Subject: [PATCH 4/4] Formatting evaluation.md

---
 torchchat/utils/docs/evaluation.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/torchchat/utils/docs/evaluation.md b/torchchat/utils/docs/evaluation.md
index 1ebe24248..ac2aa54d3 100644
--- a/torchchat/utils/docs/evaluation.md
+++ b/torchchat/utils/docs/evaluation.md
@@ -23,7 +23,7 @@ The evaluation mode of `torchchat.py` script can be used to evaluate your langua
 
 ## Examples
 
-### Python
+### Evaluation example with model in Python
 
 Running wikitext for 10 iterations
 ```
@@ -40,7 +40,7 @@ Running multiple tasks and calling eval.py directly (with torch.compile):
 python3 torchchat.py eval stories15M --compile --tasks wikitext hellaswag
 ```
 
-### exported PTE
+### Evaluation with model exported to PTE with ExecuTorch
 
 Running an exported model with ExecuTorch (as PTE)
 ```
@@ -53,7 +53,7 @@ Running multiple tasks and calling eval.py directly (with PTE):
 python3 torchchat.py eval stories15M --pte-path stories15M.pte --tasks wikitext hellaswag
 ```
 
-### exported AOTI
+### Evaluation with model exported to DSO with AOT Inductor (AOTI)
 
 Running an exported model with AOT Inductor (DSO model)
 ```