Skip to content

Commit

Permalink
local llm guide - pytest
Browse files Browse the repository at this point in the history
  • Loading branch information
pchalasani committed Jun 10, 2024
1 parent 8d2cab6 commit 9b7f1d2
Showing 1 changed file with 16 additions and 2 deletions.
18 changes: 16 additions & 2 deletions docs/tutorials/local-llm-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,12 @@
although the latter repo may contain some examples that are not in the `langroid` repo.
Most of these example scripts allow you to specify an LLM in the format `-m <model>`,
where the specification of `<model>` is described in the quide below for local/open LLMs,
or in the [Non-OpenAI LLM](https://langroid.github.io/langroid/tutorials/non-openai-llms/) guide.
or in the [Non-OpenAI LLM](https://langroid.github.io/langroid/tutorials/non-openai-llms/) guide. Scripts
that have the string `local` in their name have been especially designed to work with
certain local LLMs, as described in the respective scripts.
If you want a pointer to a specific script that illustrates a 2-agent chat, have a look
at [`chat-search-assistant.py`](https://github.com/langroid/langroid/blob/main/examples/basic/chat-search-assistant.py).
at [`chat-search-assistant.py`](https://github.com/langroid/langroid/blob/main/examples/basic/chat-search-assistant.py). This specific script, originally designed
for GPT-4/GPT-4o, runs well without any changes using `llama3-70b` (tested via Groq, mentioned below).

## Easiest: with Ollama

Expand Down Expand Up @@ -164,3 +167,14 @@ llm_config = lm.OpenAIGPTConfig(
)
```

## Quick testing with local LLMs
As mentioned [here](https://langroid.github.io/langroid/tutorials/non-openai-llms/#quick-testing-with-non-openai-models),
you can run many of the [tests](https://github.com/langroid/langroid/tree/main/tests/main) in the main langroid repo against a local LLM
(which by default run against an OpenAI model),
by specifying the model as `--m <model>`,
where `<model>` follows the syntax described in the previous sections. Here's an example:

```bash
pytest tests/main/test_chat_agent.py --m ollama/mixtral
```
Of course, bear in mind that the tests may not pass due to weaknesses of the local LLM.

0 comments on commit 9b7f1d2

Please sign in to comment.