Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add evaluate to Readme #1528

Merged
merged 3 commits into from
Jun 25, 2024
Merged

Add evaluate to Readme #1528

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b3a5548
add `evaluate` to Readme
rasbt Jun 25, 2024
140ef84
add inline links and standardize whitespace
rasbt Jun 25, 2024
977ff41
fix links
rasbt Jun 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 23 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ After installing LitGPT, select the model and action you want to take on that mo
# ligpt [action] [model]
litgpt download meta-llama/Meta-Llama-3-8B-Instruct
litgpt chat meta-llama/Meta-Llama-3-8B-Instruct
litgpt evaluate meta-llama/Meta-Llama-3-8B-Instruct
litgpt finetune meta-llama/Meta-Llama-3-8B-Instruct
litgpt pretrain meta-llama/Meta-Llama-3-8B-Instruct
litgpt serve meta-llama/Meta-Llama-3-8B-Instruct
Expand Down Expand Up @@ -201,7 +202,8 @@ litgpt chat out/custom-model/final
 

### Pretrain an LLM
Train an LLM from scratch on your own data via pretraining:

[Train an LLM from scratch](tutorials/pretrain.md) on your own data via pretraining:

<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-pretrain">
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg"; alt="Open In Studio"/>
Expand Down Expand Up @@ -233,7 +235,8 @@ litgpt chat out/custom-model/final
&nbsp;

### Continue pretraining an LLM
This is another way of finetuning that specializes an already pretrained model by training on custom data:

[Continued pretraining](tutorials/pretrain.md#continued-pretraining-on-custom-data) is another way of finetuning that specializes an already pretrained model by training on custom data:


<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-continue-pretraining">
Expand Down Expand Up @@ -265,8 +268,21 @@ litgpt chat out/custom-model/final

&nbsp;

### Evaluate an LLM

If you want to [evaluate](tutorials/evaluation.md) a downloaded, finetuned, or pretrained LLM on popular benchmark tasks, such as MMLU and Truthful QA, run the following command:

```bash
litgpt evaluate microsoft/phi-2 --tasks 'truthfulqa_mc2,mmlu'
```

> [!NOTE]
> **[Read the evaluation docs](tutorials/evaluation.md)** for more options.

&nbsp;

### Deploy an LLM
Once you're ready to deploy a finetuned LLM, run this command:
Once you're ready to [deploy](tutorials/deploy.md) a finetuned LLM, run this command:

<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-serve">
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
Expand Down Expand Up @@ -294,17 +310,17 @@ response = requests.post(
print(response.json()["output"])
```

&nbsp;

> [!NOTE]
> **[Read the full docs](tutorials/0_to_litgpt.md)**.
> **[Read the full docs](tutorials/deploy.md)**.

&nbsp;

----

### Use an LLM for inference
Use LLMs for inference to test its chatting capabilities, run evaluations, or extract embeddings, etc.

Use LLMs for [inference](tutorials/deploy.md) to test its chatting capabilities, run evaluations, or extract embeddings, etc.
Here's an example showing how to use the Phi-2 LLM.

<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-chat">
Expand Down Expand Up @@ -333,6 +349,7 @@ For more information on the different inference options, refer to the [inference
&nbsp;

# State-of-the-art features

✅ &nbsp;State-of-the-art optimizations: Flash Attention v2, multi-GPU support via fully-sharded data parallelism, [optional CPU offloading](tutorials/oom.md#do-sharding-across-multiple-gpus), and [TPU and XLA support](extensions/xla).

✅ &nbsp;[Pretrain](tutorials/pretrain.md), [finetune](tutorials/finetune.md), and [deploy](tutorials/inference.md)
Expand Down