Skip to content

Small improvements to documentation custom LLMs #410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 29, 2023
Merged

Small improvements to documentation custom LLMs #410

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions docs/howtos/customisations/llms.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@
"source": [
"# Bring your own LLMs\n",
"\n",
"Ragas uses langchain under the hood for connecting to LLMs for metrices that require them. This means you can swap out the default LLM we use (`gpt-3.5-turbo-16k`) to use any 100s of API supported out of the box with langchain.\n",
"Ragas uses langchain under the hood for connecting to LLMs for metrics that require them. This means you can swap out the default LLM we use (`gpt-3.5-turbo-16k`) with any 100s of API supported out of the box by langchain:\n",
"\n",
"- [Completion LLMs Supported](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.llms)\n",
"- [Chat based LLMs Supported](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.chat_models)\n",
"\n",
"This guide will show you how to use another or LLM API for evaluation."
"This guide will show you how to use another LLM API for evaluation."
]
},
{
Expand All @@ -32,9 +32,9 @@
"source": [
"## Evaluating with GPT4\n",
"\n",
"Ragas uses gpt3.5 by default but using gpt4 for evaluation can improve the results so lets use that for the `Faithfulness` metric\n",
"Ragas uses gpt3.5 by default but using gpt4 for evaluation can improve the results so lets use that for the `Faithfulness` metric.\n",
"\n",
"To start-off, we initialise the gpt4 `chat_model` from langchain"
"To start-off, we initialise the gpt4 `chat_model` from langchain."
]
},
{
Expand Down Expand Up @@ -67,12 +67,14 @@
"id": "f1fdb48b",
"metadata": {},
"source": [
"In order to you the Langchain LLM you have to use the `RagasLLM` wrapper. This help the Ragas library specify the interfaces that will be used internally by the metrics and what is exposed via the Langchain library. You can also use other LLM APIs in tools like LlamaIndex and LiteLLM but creating your own implementation of `RagasLLM` that supports it."
"`RagasLLM` wrapper is required to use the Langchain LLM. This helps the Ragas library specify the interfaces that will be used internally by the metrics and what is exposed via the Langchain library. \n",
"\n",
"You can also use other LLM APIs from tools like LlamaIndex or LiteLLM but that requires creating your own implementation of `RagasLLM` that supports it."
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"id": "25c72521-3372-4663-81e4-c159b0edde40",
"metadata": {},
"outputs": [],
Expand All @@ -87,7 +89,7 @@
"id": "62645da8-6a52-4cbb-bec7-59f7e153cd38",
"metadata": {},
"source": [
"Substitute the llm in `Metric` instance with the newly create GPT4 model."
"Substitute the `llm` in `Metric` instance with the newly create GPT4 model.\n"
]
},
{
Expand All @@ -107,7 +109,7 @@
"id": "1930dd49",
"metadata": {},
"source": [
"That's it! faithfulness will now be using GPT-4 under the hood for evaluations.\n",
"That's it! `Faithfulness` will now be using GPT-4 under the hood for evaluations.\n",
"\n",
"Now lets run the evaluations using the example from [quickstart](../quickstart.ipnb)."
]
Expand Down Expand Up @@ -213,7 +215,7 @@
"source": [
"## Evaluating with Open-Source LLMs\n",
"\n",
"You can also use any of the Open-Source LLM for evaluating. Ragas support most the the deployment methods like [HuggingFace TGI](https://python.langchain.com/docs/integrations/llms/huggingface_textgen_inference), [Anyscale](https://python.langchain.com/docs/integrations/llms/anyscale), [vLLM](https://python.langchain.com/docs/integrations/llms/vllm) and many [more](https://python.langchain.com/docs/integrations/llms/) through Langchain. \n",
"You can also use any of the Open-Source LLM for evaluation. Ragas support most the the deployment methods like [HuggingFace TGI](https://python.langchain.com/docs/integrations/llms/huggingface_textgen_inference), [Anyscale](https://python.langchain.com/docs/integrations/llms/anyscale), [vLLM](https://python.langchain.com/docs/integrations/llms/vllm) and many [more](https://python.langchain.com/docs/integrations/llms/) through Langchain. \n",
"\n",
"When it comes to selecting open-source language models, there are some rules of thumb to follow, given that the quality of evaluation metrics depends heavily on the model's quality:\n",
"\n",
Expand Down Expand Up @@ -310,7 +312,7 @@
"id": "58a610f2-19e5-40ec-bb7d-760c1d608a85",
"metadata": {},
"source": [
"Now you can run the evaluations with and analyse the results."
"Now you can run the evaluations with `HuggingFaceH4/zephyr-7b-alpha` and analyse the results."
]
},
{
Expand Down