Merge pull request #89 from microsoft/integration

Integration

Author: nicoleserafino

docs/02_setup/README.md

@@ -99,7 +99,7 @@ You can run the project in your local VS Code Dev Container using the [Dev Conta
    ```
 
 1. When prompted provided the following names:
-   1. Environment name (used as the name of the new resource group to deploy into)
+   1. Environment name (used as the name of the new or existing resource group to deploy into)
    1. Azure Subscription to use
    1. Azure location to use
 

docs/03_observability/README.md

@@ -30,6 +30,22 @@ The bicep infrastructure deploys an Application Insights resource where you can
 
 ## Observe Application Behavior
 
+### Local Configuration
+
+You can see telemetry data if you are running the app locally, in a local dev container, or in Azure via `azd up`. It will not work if you're running in Github Codespaces. 
+
+If running locally or in a dev container, there are configuration steps to wire up the Aspire Dashboard:
+1. Update the `.env` file with this value: `TELEMETRY_SCENARIO=console,aspire_dashboard`
+1. Run this command to host the Aspire Dashboard in a local container:
+
+   ```bash
+   docker run --rm -it -d -p 18888:18888 -p 4317:18889 -e DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true --name aspire-dashboard mcr.microsoft.com/dotnet/aspire-dashboard:9.0
+   ```
+
+If running locally (not in a dev container), update the `.env` file with this value: `ASPIRE_DASHBOARD_ENDPOINT=http://localhost:4317`
+
+No additional steps are needed if you are running the app that was deployed by `azd up`.
+
 ### Conduct a conversation with the customer service agents
 
 Conduct a conversation with the customer service agents following this sequence:

docs/04_explore/README.md

@@ -125,7 +125,7 @@
 
 5. Let's add data source for insurance policy / rental policies, etc. with AI Search
 
-    Create a JSON file for car rental policies.  Name the file car_rental_policy.json and store it in the data folder along with the flight & hotel policies:
+   a)  Create a JSON file for car rental policies.  Name the file car_rental_policy.json and store it in the data folder along with the flight & hotel policies:
 
     <details>
     <summary> Click to expand/collapse</summary>
@@ -153,7 +153,7 @@
     </details>
 
 
-    Update the plugin to setup Azure OpenAI client for embeddings, create get_embedding method to generate text embeddings.  Define the SearchClient class with semantic search functionality and finally initialize the search client.
+    b) Update the plugin (car_rental_plugin.py) to setup Azure OpenAI client for embeddings, create get_embedding method to generate text embeddings.  Define the SearchClient class with semantic search functionality and finally initialize the search client.
 
     <details>
     <summary> Click to expand/collapse</summary>
@@ -196,7 +196,7 @@
 
     </details>
 
-    Update the Car_Rental_Tools class to include policy search:
+    c) Update the Car_Rental_Tools class in the plugin (car_rental_plugin.py) to include policy search:
 
     <details>
     <summary> Click to expand/collapse</summary>
@@ -222,7 +222,7 @@
 
     </details>
 
-    Create an embedding generation script & store it in  ./scripts/generate_car_rental_policy_embeddings.py :
+    d) Create an embedding generation script called generate_car_rental_policy_embeddings.py & in  app/backend/scripts folder:
 
     <details>
     <summary> Click to expand/collapse</summary>
@@ -269,7 +269,7 @@
     </details>
 
 
-    Run the embedding generation script in the terminal 
+    e) Run the embedding generation script in the terminal 
 
     <details>
     <summary> Click to expand/collapse</summary>
@@ -278,29 +278,140 @@
     cd c:\Code\multi-modal-customer-service-agent\voice_agent\app\backend
     python scripts/generate_car_rental_policy_embeddings.py
     ```
-
     </details>
 
-    The car rental policy search uses the same semantic search approach as the hotel policies.  The policies are stored in JSON format with embeddings for efficient semantic search.
+    f) After implementation, test with queries about:
+      - Insurance coverage options and costs
+      - Rental requirements and age restrictions
+      - Deductibles and additional coverage options
+
+    ### Expand to see the complete car_rental_plugins.py file
+    <details>
+    <summary> Click to expand/collapse</summary>
+
+    ```python
+    from typing import Annotated, Any  
+    from semantic_kernel.functions import kernel_function  
+    import os
+    from openai import AzureOpenAI
+    import json
+    from scipy import spatial
+
+    # Constants for Azure OpenAI  
+    AZURE_OPENAI_API_KEY = os.getenv("AZURE_OPENAI_API_KEY")  
+    AZURE_OPENAI_ENDPOINT = os.getenv("AZURE_OPENAI_ENDPOINT")  
+    AZURE_OPENAI_EMB_DEPLOYMENT = os.getenv("AZURE_OPENAI_EMB_DEPLOYMENT")
+    AZURE_OPENAI_EMB_ENDPOINT = os.getenv("AZURE_OPENAI_EMB_ENDPOINT", AZURE_OPENAI_ENDPOINT)
+    AZURE_OPENAI_EMB_API_KEY = os.getenv("AZURE_OPENAI_EMB_API_KEY", AZURE_OPENAI_API_KEY)  
+    AZURE_OPENAI_CHAT_DEPLOYMENT = os.getenv("AZURE_OPENAI_CHAT_DEPLOYMENT")  
+
+    # Azure OpenAI client setup  
+    embedding_client = AzureOpenAI(  
+        api_key=AZURE_OPENAI_EMB_API_KEY,  
+        azure_endpoint=AZURE_OPENAI_EMB_ENDPOINT,  
+        api_version="2023-12-01-preview"  
+    )  
+
+    def get_embedding(text: str, model: str = AZURE_OPENAI_EMB_DEPLOYMENT) -> list[float]:  
+        """Generate text embeddings using Azure OpenAI."""  
+        text = text.replace("\n", " ")  
+        return embedding_client.embeddings.create(input=[text], model=model).data[0].embedding  
 
-    The Car_Rental_Tools class now includes a method to search policies.
+    class SearchClient:  
+        """Client for performing semantic search on a knowledge base."""  
+        def __init__(self, emb_map_file_path: str):  
+            with open(emb_map_file_path) as file:  
+                self.chunks_emb = json.load(file)  
+    
+        def find_article(self, question: str, topk: int = 3) -> str:  
+            """Find relevant articles based on cosine similarity."""  
+            input_vector = get_embedding(question)  
+            cosine_list = [  
+                (item['id'], item['policy_text'], 1 - spatial.distance.cosine(input_vector, item['policy_text_embedding']))  
+                for item in self.chunks_emb  
+            ]  
+            cosine_list.sort(key=lambda x: x[2], reverse=True)  
+            cosine_list = cosine_list[:topk]  
+    
+            return "\n".join(f"{chunk_id}\n{content}" for chunk_id, content, _ in cosine_list)  
 
-    The embedding generation script needs to be run once to prepare the policy data.
+    # Initialize the search client
+    search_client = SearchClient("./data/car_rental_policy.json")
 
-    You can expand the policy database by adding more entries to the JSON file.
+    # Kernel functions  
+    class Car_Rental_Tools:
+        """Tools for car rental agent to perform various rental operations"""
+    
+        agent_name = "car_rental_agent"  # Name of the agent
 
-    After implementing these changes:
+        @kernel_function(
+            description="Search for available rental cars",
+            name="search_cars"
+        )
+        async def search_cars(self, location: str, pickup_date: str, return_date: str) -> str:
+            # Mock implementation - replace with actual car rental API
+            return f"Found available cars in {location} from {pickup_date} to {return_date}: \n" \
+                "1. Economy - Toyota Corolla ($45/day)\n" \
+                "2. SUV - Honda CR-V ($65/day)\n" \
+                "3. Luxury - BMW 5 Series ($95/day)"
 
-    - The car rental agent will be available alongside the hotel and flight agents
-    - The agent can be triggered through intent detection
-    - The agent will have access to car rental specific tools through its plugin
-    - The agent will maintain its own context and persona while handling customer interactions
+        @kernel_function(
+            description="Get rental car details",
+            name="get_car_details"
+        )
+        async def get_car_details(self, car_id: str) -> str:
+            # Mock implementation
+            car_details = {
+                "1": "Toyota Corolla: 4 doors, 5 seats, automatic transmission, GPS, bluetooth",
+                "2": "Honda CR-V: 5 doors, 5 seats, automatic transmission, GPS, bluetooth, roof rack",
+                "3": "BMW 5 Series: 4 doors, 5 seats, automatic transmission, GPS, bluetooth, leather seats"
+            }
+            return car_details.get(car_id, "Car not found")
 
-    Test the new agent with various car rental scenarios.
 
-    Future steps to try on your own.
-    - Expand the Car_Rental_Tools class with additional functions as needed.  
-    - Replace mock implementations with actual car rental API integrations
+        @kernel_function(
+            name="search_rental_policies",
+            description="Search car rental and insurance policies for relevant information."
+        )
+        async def search_rental_policies(self, 
+            search_query: Annotated[str, "The search query about rental or insurance policies."]
+        ) -> str:
+            return search_client.find_article(search_query)
+    ```
+
+    </details>
+
+### Summarizing Exercise 1 - Adding a New Car Rental Agent with Semantic Search
+
+This exercise demonstrates how to extend the voice agent system by adding a completely new domain-specific agent for car rentals, complete with semantic search capabilities for policy information.
+
+#### Files Created:
+- `car_rental_agent_profile.yaml` - Defines the agent's persona and responsibilities
+- `car_rental_plugins.py` - Contains tools for car searching, details, and policy lookups
+- `car_rental_policy.json` - Stores policy data with embeddings for semantic search
+- `scripts/generate_car_rental_policy_embeddings.py` - Script to generate policy embeddings
+
+#### Files Updated:
+- `rtmt.py` - Modified to register and initialize the new car rental agent
+- `utility.py` - Updated to include car rental in intent detection system messages
+
+#### Key Components:
+1. **Agent Profile**: Defines the car rental agent's persona and scope
+2. **Tool Integration**: Custom tools for searching cars, viewing details, and accessing policies
+3. **SearchClient Class**: Implements vector similarity search using cosine distance
+4. **Intent Detection**: Updates to route car rental queries to the specialized agent
+
+#### Integration Benefits:
+- Demonstrates the extensibility of the multi-agent architecture
+- Adds a completely new domain (car rentals) to the existing hotel and flight agents
+- Semantic search provides accurate policy information retrieval
+- Maintains consistent agent experience while expanding capabilities
+
+#### Future Enhancements:
+- Expand the policy database with more detailed entries
+- Add domain-specific tools to the Car_Rental_Tools class
+- Implement actual API integrations to replace mock implementations
+- Create more sophisticated embedding strategies for better matching
 
 ### Exercise 2: debug/fix/commit flow
   - scenario which 'fails' from a functionality standpoint -> use observability (tracing in AI Foundry), figure out the problem, change system-prompt/etc., run test suite (local run of Evaluations), then PR

voice_agent/app/backend/.env.sample

@@ -18,3 +18,5 @@ TURN_DETECTION_PREFIX_PADDING_MS=300
 TURN_DETECTION_SILENCE_DURATION_MS=200
 AZURE_REDIS_ENDPOINT=#optional, if you want to use redis for caching which support distributed caching for high
 AZURE_REDIS_KEY=#optional, if you want to use redis for caching which support distributed caching for high
+ASPIRE_DASHBOARD_ENDPOINT=http://host.docker.internal:4317
+TELEMETRY_SCENARIO=console

voice_agent/app/infra/main.bicep

@@ -120,7 +120,7 @@ param containerAppsSubnetPrefix string = '10.0.2.0/23'
 
 // Organize resources in a resource group
 resource resourceGroup 'Microsoft.Resources/resourceGroups@2021-04-01' = {
-  name: 'rg-${environmentName}'
+  name: environmentName
   location: location
   tags: tags
 }

voice_agent/app/start.sh

@@ -1,61 +1,91 @@
-#!/bin/sh  
-# This script sets up and starts both the backend and the frontend for local development.  
-# The backend is run with a Python virtual environment and the frontend uses Vite’s dev server.  
+#!/usr/bin/env bash  
+#  
+# dev.sh  –  start backend (FastAPI) and frontend (Vite) in watch mode  
+#            while taking care of Python venv / npm install, plus  
+#            graceful shutdown on Ctrl-C.  
+  
+set -euo pipefail  
+  
+###############################################################################  
+# Helper functions  
+###############################################################################  
+die () { echo "ERROR: $*" >&2; exit 1; }  
+  
+cleanup () {  
+  echo                    # blank line  
+  echo "Stopping services…"  
+  [[ -n "${backend_pid:-}"  ]] && kill "$backend_pid"  2>/dev/null || true  
+  [[ -n "${frontend_pid:-}" ]] && kill "$frontend_pid" 2>/dev/null || true  
+  exit 0  
+}  
+  
+trap cleanup INT TERM  
 
-# --------------------------- Start the Backend Service ---------------------------  
-echo ""  
+###############################################################################  
+# Backend  
+###############################################################################  
+echo  
 echo "Setting up the Python virtual environment for the backend..."  
 
-cd backend || { echo "backend folder not found."; exit 1; }  
+[[ -d backend ]] || die "backend folder not found."  
+pushd backend > /dev/null  
 
-# Create virtual environment if it doesn't exist  
-if [ ! -d ".venv" ]; then  
-    python3 -m venv .venv  
+# ------------------------------------------------------------------  
+# Create / upgrade the virtual-env  
+# ------------------------------------------------------------------  
+if [[ ! -d .venv ]]; then  
+  # --upgrade-deps guarantees fresh pip / setuptools / wheel  
+  python -m venv --upgrade-deps .venv  
+else  
+  # env exists – still make sure the essentials are up to date  
+  .venv/bin/python -m pip install --upgrade pip setuptools wheel packaging  
 fi  
 
-echo ""  
-echo "Installing backend Python packages..."  
-./.venv/bin/python -m pip install -r requirements.txt || { echo "Backend dependency installation failed."; exit 1; }  
-  
-echo ""  
-echo "Starting backend service (listening on port 8765)..."  
-# Use --reload for auto-restarting during development  
-./.venv/bin/python app.py --reload &  
-BACKEND_PID=$!  
-  
-cd ..  
-  
-# --------------------------- Start the Frontend Service ---------------------------  
-echo ""  
-echo "Installing frontend npm packages..."  
-  
-cd frontend || { echo "frontend folder not found."; exit 1; }  
-npm install || { echo "Frontend npm installation failed."; exit 1; }  
-
-echo ""
-echo "Building frontend application..."
-npm run build || { echo "Frontend build failed."; exit 1; }
-# Note:  
-# • The .env file for the frontend should be placed in the frontend directory (i.e., ./frontend/.env)  
-# • This file can contain environment variables such as BACKEND_WS_URL that Vite will load.  
-#  
-# Example .env file:  
-export VITE_BACKEND_WS_URL="ws://localhost:8765"  
+echo  
+echo "Installing backend Python packages…"  
+  
+venv_py=".venv/bin/python"  
+[[ -x "$venv_py" ]] || { popd >/dev/null; die "Could not find the virtual-env Python executable."; }  
+  
+"$venv_py" -m pip install -r requirements.txt || { popd >/dev/null; die "Backend dependency installation failed."; }  
 
-echo ""  
-echo "Starting frontend dev server..."  
-node server.cjs &
-FRONTEND_PID=$!
-
-echo "Frontend proxy started at: http://localhost:3000"
-
-cd ..  
+echo  
+echo "Starting backend service (listening on port 8765)…"  
+  
+# Launch FastAPI (or whatever app.py holds) in the background  
+"$venv_py" app.py --reload &  
+backend_pid=$!  
+  
+popd > /dev/null  
+  
+###############################################################################  
+# Frontend  
+###############################################################################  
+echo  
+echo "Installing frontend npm packages…"  
+  
+[[ -d frontend ]] || { kill "$backend_pid"; die "frontend folder not found."; }  
+pushd frontend > /dev/null  
+  
+npm install || { popd >/dev/null; kill "$backend_pid"; die "Frontend npm installation failed."; }  
+  
+echo  
+echo "Starting frontend dev server…"  
+  
+export VITE_BACKEND_WS_URL="ws://localhost:8765"  
+npm run dev &  
+frontend_pid=$!  
 
-# --------------------------- Clean-up on Exit ---------------------------  
-trap "echo 'Stopping services...'; kill $BACKEND_PID; kill $FRONTEND_PID; exit 0" INT TERM  
+popd > /dev/null  
 
-echo ""  
+###############################################################################  
+# Wait forever until user presses Ctrl-C  
+###############################################################################  
+echo  
 echo "Both backend and frontend services are running."  
 echo "Press Ctrl+C to stop."  
 
-wait  
+# Keep script alive  
+while true; do  
+  sleep 1  
+done