Prompt Flow Bath and Evaluation

README.md

@@ -6,6 +6,7 @@ The repo provides a way to upload your own data so it's ready to try end to end.
 
 ## Updates
 
+* 9/15/2023 - Modified the azure search package to 11.4.0b9 and langchain to latest version.  Added capability to perform [evaluation](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-develop-an-evaluation-flow?view=azureml-api-2) on PromptFlow for both QnA and Chat.  [Bert PDF](./Workshop/Data/PDF/Bert.pdf) and [Evaluation Data](./api/PromptFlow/QuestionAnswering/bert.jsonl) can be used to perform Batch and Evaluation in Prompt Flow.  [Sample Notebook](./Workshop/12_PromptFlowQa.ipynb) showcasing the flow and E2E process is available.  [Bert Chat](./Workshop/promptflow/BertChat/) folder allows you to test E2E Prompt Flow, Batch Run and Evaluation in form of Notebook.
 * 9/3/2023 - Added [API](./api/PromptFlow/Chat/) for [Chat](./assets/ChatPf.png) using the Prompt Flow.  Allow end-user to select between Azure Functions as API (`ApiType` Configuration in Web App) or using Prompt Flow Managed endpoint.
 * 9/2/2023 - Added [API](./api/PromptFlow/QuestionAnswering/) for [Question Answering](./assets/QaPf.png) using the Prompt Flow.  Allow end-user to select between Azure Functions as API (`ApiType` Configuration in Web App) or using Prompt Flow Managed endpoint.
 * 8/31/2023 - Added example for [LLMOps](LLMOps.md) using [Prompt Flow](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/overview-what-is-prompt-flow?view=azureml-api-2).  The repo will be adding the flexibility to use the Prompt Flow Deployed Model as an alternative to current Azure Functions.

Workshop/0A_PromptEngg.ipynb

@@ -13,7 +13,19 @@
    "cell_type": "code",
    "execution_count": 1,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "ename": "AssertionError",
+     "evalue": "ERROR: Azure OpenAI Endpoint should be in the form: \n\n\t<your unique endpoint identifier>.openai.azure.com",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[1;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
+      "\u001b[1;32md:\\repos\\chatpdf\\Workshop\\0A_PromptEngg.ipynb Cell 2\u001b[0m line \u001b[0;36m1\n\u001b[0;32m     <a href='vscode-notebook-cell:/d%3A/repos/chatpdf/Workshop/0A_PromptEngg.ipynb#W1sZmlsZQ%3D%3D?line=11'>12</a>\u001b[0m openAiEndPoint \u001b[39m=\u001b[39m \u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mOpenAiEndPoint\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m\n\u001b[0;32m     <a href='vscode-notebook-cell:/d%3A/repos/chatpdf/Workshop/0A_PromptEngg.ipynb#W1sZmlsZQ%3D%3D?line=12'>13</a>\u001b[0m \u001b[39massert\u001b[39;00m openAiEndPoint, \u001b[39m\"\u001b[39m\u001b[39mERROR: Azure OpenAI Endpoint is missing\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m---> <a href='vscode-notebook-cell:/d%3A/repos/chatpdf/Workshop/0A_PromptEngg.ipynb#W1sZmlsZQ%3D%3D?line=13'>14</a>\u001b[0m \u001b[39massert\u001b[39;00m \u001b[39m\"\u001b[39m\u001b[39mopenai.azure.com\u001b[39m\u001b[39m\"\u001b[39m \u001b[39min\u001b[39;00m openAiEndPoint\u001b[39m.\u001b[39mlower(), \u001b[39m\"\u001b[39m\u001b[39mERROR: Azure OpenAI Endpoint should be in the form: \u001b[39m\u001b[39m\\n\u001b[39;00m\u001b[39m\\n\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m<your unique endpoint identifier>.openai.azure.com\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m     <a href='vscode-notebook-cell:/d%3A/repos/chatpdf/Workshop/0A_PromptEngg.ipynb#W1sZmlsZQ%3D%3D?line=14'>15</a>\u001b[0m openai\u001b[39m.\u001b[39mapi_base \u001b[39m=\u001b[39m openAiEndPoint\n",
+      "\u001b[1;31mAssertionError\u001b[0m: ERROR: Azure OpenAI Endpoint should be in the form: \n\n\t<your unique endpoint identifier>.openai.azure.com"
+     ]
+    }
+   ],
    "source": [
     "# Import Python libraries\n",
     "import os\n",

Workshop/0_SetupandConfig.ipynb

@@ -128,7 +128,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -144,8 +144,6 @@
     "assert openai_api_key, \"ERROR: Azure OpenAI Key is missing\"\n",
     "openai.api_key = openai_api_key\n",
     "openAiEndPoint = f\"{OpenAiEndPoint}\"\n",
-    "assert openAiEndPoint, \"ERROR: Azure OpenAI Endpoint is missing\"\n",
-    "assert \"openai.azure.com\" in openAiEndPoint.lower(), \"ERROR: Azure OpenAI Endpoint should be in the form: \\n\\n\\t<your unique endpoint identifier>.openai.azure.com\"\n",
     "openai.api_base = openAiEndPoint"
    ]
   },
@@ -581,7 +579,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -599,7 +597,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [
     {
@@ -632,29 +630,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "subjective choice and varies from person to person. Some may prefer dogs due to their loyalty and companionship, while others may prefer cats for their independence and low-maintenance nature. Others may prefer smaller pets like rabbits, hamsters, or guinea pigs, while some may even prefer reptiles or\n",
-      "subjective topic and can vary depending on individual preferences and lifestyles. However, some popular choices for pet animals include dogs, cats, and fish.\n",
-      "\n",
-      "Dogs are often considered to be the best pets because of their loyalty, affectionate nature, and ability to provide companionship. They are also known for\n",
-      "subjective choice and depends on an individual's personal preferences and lifestyle. Some people may argue that dogs make the best pets because they are loyal, affectionate, and can be trained to perform various tasks. Others may argue that cats make the best pets because they are more independent, low maintenance, and\n",
-      "matter of personal preference, as different pets have different qualities that may appeal to different individuals. Some may prefer a dog for its loyalty and companionship, while others may prefer a cat for its independence and low maintenance. Others may prefer smaller pets such as guinea pigs or hamsters for their ease of\n",
-      "matter of personal preference. Some people may argue that a dog is the best pet because they are loyal, protective, and make great companions. Others may argue that a cat is the best pet because they are independent, low maintenance, and provide comfort. Some people may prefer different types of pets such as\n",
-      "personal preference and depends on individual circumstances such as living situation, lifestyle, and personal preferences.\n",
-      "\n",
-      "Some people may prefer dogs as pets because they are loyal, affectionate and can provide companionship and protection. Dogs are also known to be trainable, making it easier to teach them commands and tricks. However,\n",
-      "subjective topic and can vary depending on personal preferences and lifestyles. Some people may prefer dogs as pets because they are loyal, loving, and can be trained to perform various tasks. Others may prefer cats because they are independent, low maintenance, and provide companionship. Other popular pets include birds, fish\n",
-      "subjective question because it depends on individual preferences and lifestyles. Some people may prefer dogs as pets because they are loyal, playful, and can be trained easily. Others may prefer cats because they are independent, low-maintenance, and can provide comfort and companionship. Some people may even prefer unconventional pets\n",
-      "subjective matter, as different people have different preferences and needs. However, some popular pets that are often considered great choices include:\n",
-      "\n",
-      "- Dogs: Known for their loyalty, companionship, and ability to form deep bonds with their owners, dogs are a popular choice for many people. They come in a\n",
-      "subjective topic and can vary depending on each individual's preferences and lifestyle. Some people may prefer dogs as they are loyal, affectionate, and can provide protection. Others may prefer cats as they are independent, low maintenance, and can be great companions. Additionally, some people may prefer smaller pets such\n"
+      "subjective question and can vary depending on personal preferences and circumstances. Some people may argue that a dog is the best pet because they are loyal, loving, and can provide companionship and protection. Others may prefer a cat because they are independent, low-maintenance, and can still provide companionship.\n",
+      "subjective topic, as it depends on personal preference and lifestyle. Some people may prefer a cat as a pet, as they are independent, low maintenance, and can provide companionship. Others may prefer a dog, as they are loyal, loving, and can be trained to perform various tasks. Some\n",
+      "subjective opinion and can vary depending on individual preferences and lifestyle. Some people may argue that dogs make the best pets due to their loyalty, companionship, and trainability. Others may argue that cats are the best pets because of their independence, low maintenance, and ability to cuddle. Other popular\n",
+      "matter of personal preference and lifestyle. Some people may prefer dogs as they are loyal, trainable, and can be great companions for exercise and outdoor activities. Others may prefer cats for their independence, low maintenance, and ability to provide comfort and companionship. Other popular pets include fish, birds, rabbits,\n",
+      "subjective question as what makes a pet the \"best\" can vary greatly depending on individual preferences, lifestyle, and circumstances. However, some commonly mentioned pets that are often considered to be good options for many people include dogs, cats, and small animals such as guinea pigs, hamsters, or\n",
+      "subjective topic, as what makes a pet \"the best\" can vary depending on personal preferences and individual circumstances. Some people may argue that dogs make the best pets because of their loyalty, companionship, and ability to be trained. Others may prefer cats for their independence, low maintenance, and ability\n",
+      "subjective matter, as different people have different preferences and lifestyles. Some may prefer dogs, as they are loyal, protective, and can be trained to perform various tasks. Others may prefer cats, as they are independent, low maintenance, and can provide comfort and companionship. Some may prefer smaller pets\n",
+      "subjective topic, as what is considered the best pet varies from person to person. However, some popular choices for pets include dogs, cats, fish, birds, and hamsters. Ultimately, the best pet for someone depends on their lifestyle, living situation, and personal preferences.\n",
+      "matter of personal preference, as different people have different lifestyles and preferences when it comes to their pets. However, some popular choices for pets include dogs, cats, fish, and birds. Dogs are often referred to as man's best friend due to their loyalty and companionship. They require regular exercise and\n",
+      "subjective topic, as it depends on individual preferences and lifestyles. Some people may prefer dogs because of their loyalty and companionship, while others may prefer cats for their independence and low maintenance. Some may prefer smaller pets like hamsters or guinea pigs, while others may prefer larger pets like horses or\n"
      ]
     }
    ],