Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update QA and RAG evaluation notebooks #10782

Merged
merged 5 commits into from Jan 5, 2024
Merged

Update QA and RAG evaluation notebooks #10782

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9f8fa5c
modify guidance for setting openai key
annzhang-db Jan 4, 2024
e678b77
re-run qa notebook
annzhang-db Jan 4, 2024
742c188
Remove reference to 2.6.1.dev0
annzhang-db Jan 4, 2024
07883cd
also update question-answering-evaluation.ipynb
annzhang-db Jan 4, 2024
0fac76c
also update rag notebooks
annzhang-db Jan 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
48 changes: 6 additions & 42 deletions docs/source/llms/llm-evaluate/notebooks/question-answering-evaluation.ipynb
View file
Open in desktop
Expand Up @@ -14,7 +14,7 @@
"source": [
"# LLM Evaluation with MLflow Example Notebook\n",
"\n",
"In this notebook, we will demonstrate how to evaluate various LLMs and RAG systems with MLflow, leveraging simple metrics such as perplexity and toxicity, as well as LLM-judged metrics such as relevance, and even custom LLM-judged metrics such as professionalism"
"In this notebook, we will demonstrate how to evaluate various LLMs and RAG systems with MLflow, leveraging simple metrics such as toxicity, as well as LLM-judged metrics such as relevance, and even custom LLM-judged metrics such as professionalism"
]
},
{
Expand All @@ -29,47 +29,11 @@
}
},
"source": [
"Set OpenAI Key"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "fb946228-62fb-4d68-9732-75935c9cb401",
"showTitle": false,
"title": ""
}
},
"outputs": [],
"source": [
"import os"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "bec25067-224d-4ee8-9b5d-0beeb6cde684",
"showTitle": false,
"title": ""
}
},
"outputs": [],
"source": [
"os.environ[\"OPENAI_API_KEY\"] = \"redacted\""
"We need to set our OpenAI API key, since we will be using GPT-4 for our LLM-judged metrics.\n",
"\n",
"In order to set your private key safely, please be sure to either export your key through a command-line terminal for your current instance, or, for a permanent addition to all user-based sessions, configure your favored environment management configuration file (i.e., .bashrc, .zshrc) to have the following entry:\n",
"\n",
"`OPENAI_API_KEY=<your openai API key>`"
]
},
{
Expand Down
17 changes: 6 additions & 11 deletions docs/source/llms/llm-evaluate/notebooks/rag-evaluation-llama2.ipynb
View file
Open in desktop
Expand Up @@ -28,17 +28,13 @@
},
{
"cell_type": "markdown",
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {},
"inputWidgets": {},
"nuid": "bdff35e3-0e09-48b8-87ce-78759de88998",
"showTitle": false,
"title": ""
}
},
"metadata": {},
"source": [
"Set OpenAI Key"
"We need to set our OpenAI API key.\n",
"\n",
"In order to set your private key safely, please be sure to either export your key through a command-line terminal for your current instance, or, for a permanent addition to all user-based sessions, configure your favored environment management configuration file (i.e., .bashrc, .zshrc) to have the following entry:\n",
"\n",
"`OPENAI_API_KEY=<your openai API key>`"
]
},
{
Expand All @@ -58,7 +54,6 @@
},
"outputs": [],
"source": [
"os.environ[\"OPENAI_API_KEY\"] = \"REDACTED\"\n",
"os.environ[\"DATABRICKS_HOST\"] = \"REDACTED\"\n",
"os.environ[\"DATABRICKS_TOKEN\"] = \"REDACTED\""
]
Expand Down
59 changes: 15 additions & 44 deletions docs/source/llms/llm-evaluate/notebooks/rag-evaluation.ipynb
View file
Open in desktop
Expand Up @@ -17,54 +17,25 @@
"In this notebook, we will demonstrate how to evaluate various a RAG system with MLflow."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import os"
]
},
{
"cell_type": "markdown",
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {},
"inputWidgets": {},
"nuid": "bdff35e3-0e09-48b8-87ce-78759de88998",
"showTitle": false,
"title": ""
}
},
"source": [
"Set OpenAI (or Azure OpenAI) environment variables"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "bec25067-224d-4ee8-9b5d-0beeb6cde684",
"showTitle": false,
"title": ""
}
},
"outputs": [],
"metadata": {},
"source": [
"os.environ[\"OPENAI_API_KEY\"] = \"REDACTED\"\n",
"We need to set our OpenAI API key.\n",
"\n",
"In order to set your private key safely, please be sure to either export your key through a command-line terminal for your current instance, or, for a permanent addition to all user-based sessions, configure your favored environment management configuration file (i.e., .bashrc, .zshrc) to have the following entry:\n",
"\n",
"`OPENAI_API_KEY=<your openai API key>`\n",
"\n",
"If using Azure OpenAI, you will instead need to set\n",
"\n",
"`OPENAI_API_TYPE=\"azure\"`\n",
"\n",
"`OPENAI_API_VERSION=<YYYY-MM-DD>`\n",
"\n",
"`OPENAI_API_KEY=<https://<>.<>.<>.com>`\n",
"\n",
"# If using Azure OpenAI\n",
"# os.environ[\"OPENAI_API_TYPE\"] = \"azure\"\n",
"# os.environ[\"OPENAI_API_VERSION\"] = \"2023-05-15\"\n",
"# os.environ[\"OPENAI_API_KEY\"] = \"https://<>.<>.<>.com\"\n",
"# os.environ[\"OPENAI_DEPLOYMENT_NAME\"] = \"deployment-name\""
"`OPENAI_API_DEPLOYMENT_NAME=<deployment name>`\n"
]
},
{
Expand Down