Merge pull request #84 from microsoft/integration

Integration

Author: james-tn

README.md

@@ -27,17 +27,11 @@ Visit this section to get an overview of the overall architecture and patterns,
 
 Proceed to this section if you'd like to setup your local development environment. You will install the required tools and libraries, clone the prebuilt app repository, and set up your development environment.
 
-You will also set up the Azure Landing Zone to host the application once deployed.
+### [Explore application observability](docs/03_observability/README.md)
 
-### [Build and Run the App Locally](docs/03_build/README.md)
+Visit this section to observe application behavior and performance through various tools. See logs, traces, and metrics to understand what your application was doing during the realtime conversation.
 
-In this section, you will get the instructions to build and run the backend (Python) and frontend (React.js) of the app locally. You will understand how to test the app locally to ensure everything is working as expected.
-
-### [Deploy to Azure](docs/04_deploy/README.md)
-
-Visit this section to get the detailed instructions to deploy the application front end and backend services to Azure and proceed with some basic tests to ensure everything is working as expected.
-
-### [Explore this Solution Accelerator with Guided Exercises](docs/05_explore/README.md)
+### [Explore this Solution Accelerator with Guided Exercises](docs/04_explore/README.md)
 
 The purpose of this section is to help you learn how to best leverage this solution by walking you through some key scenarios related to different personas involved in the lifecycle of a complex Agentic AI solution. For instance, learn how to add new features to the app, scale it, secure it, monitor and troubleshoot it.
 

docs/01_architecture/README.md

@@ -155,4 +155,7 @@ Every request—regardless of channel—is tied to a unique session ID (UUID for
 
 ---  
 
-**This architecture combines the best of cloud-native, real-time interaction, secure AI orchestration, and practical scaling, making it ready for both quick prototyping and robust, enterprise deployments.**  
+**This architecture combines the best of cloud-native, real-time interaction, secure AI orchestration, and practical scaling, making it ready for both quick prototyping and robust, enterprise deployments.**  
+
+---
+#### Navigation: [Home](../../README.md) | [Previous Section](../../README.md) | [Next Section](../02_setup/README.md) 

docs/02_setup/README.md

@@ -1,12 +1,107 @@
-# Setup your Environment (skip is using a Lab environment)
+# Setup your Environment 
 
-## 1. Local Developer env Setup
-- Install required tools and libraries
-- Clone the prebuilt app repository
-- Set up your development environment
+## Running this sample
 
-## 2. Provision Azure Infrastructure
-- Setting up Azure Landing Zones for infrastructure
+We'll follow 4 steps to get this example running in your own environment: pre-requisites, creating an index, setting up the environment, and running the app.
+
+### Pre-requisites
+
+You'll need instances of the following Azure services. You can re-use service instances you have already or create new ones.
+
+1. [Azure OpenAI](https://ms.portal.azure.com/#create/Microsoft.CognitiveServicesOpenAI), with 2 model deployments, one of the **gpt-4o-realtime-preview** model, a regular **gpt-4o-mini** model.
+1. [Optional] Train an intent_detection model with a SLM using Azure AI Studio. Check [the training data](./intent_detection_model)
+
+### Setting up the environment
+
+The app needs to know which service endpoints to use for the Azure OpenAI and Azure AI Search. The following variables can be set as environment variables, or you can create a ".env" file in the "app/backend/" directory with this content.
+
+The voice agent can use a fine-tuned SLM deployment to classify intent to minimize latency. If you do not have this deployment available then you can use the Azure OpenAI **gpt-4o-mini** deployment, which is fast enough to classify intent with minimal impact on latency. To use **gpt-4o-mini** leave the `INTENT_SHIFT_API_*` env variables empty and supply `AZURE_OPENAI_4O_MINI_DEPLOYMENT`.
+
+```bash
+AZURE_OPENAI_ENDPOINT="https://.openai.azure.com/"
+AZURE_OPENAI_API_KEY=
+AZURE_OPENAI_API_VERSION=2024-10-01-preview
+AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-4o-mini
+AZURE_OPENAI_EMB_DEPLOYMENT="text-embedding-ada-002"
+AZURE_OPENAI_EMB_ENDPOINT= [Optional] if different from your realtime endpoint
+AZURE_OPENAI_EMB_API_KEY= [Optional] if providing an embedding endpoint
+AZURE_OPENAI_4O_MINI_DEPLOYMENT=YOUR_AZURE_OPENAI_4O_MINI_DEPLOYMENT_NAME
+INTENT_SHIFT_API_KEY=
+INTENT_SHIFT_API_URL=https://YOUR_ML_DEPLOYMENT.westus2.inference.ml.azure.com/score
+INTENT_SHIFT_API_DEPLOYMENT=YOUR_ML_DEPLOYMENT_NAME
+AZURE_OPENAI_API_VERSION=2024-10-01-preview
+AZURE_OPENAI_REALTIME_DEPLOYMENT_NAME=gpt-4o-realtime-preview
+```
+
+### Running the app
+
+#### GitHub Codespaces
+
+You can run this repo virtually by using GitHub Codespaces, which will open a web-based VS Code in your browser:
+
+[![Open in GitHub Codespaces](https://img.shields.io/static/v1?style=for-the-badge&label=GitHub+Codespaces&message=Open&color=brightgreen&logo=github)](https://github.com/codespaces/new?hide_repo_select=true&ref=main&skip_quickstart=true&machine=basicLinux32gb&repo=840462613&devcontainer_path=.devcontainer%2Fdevcontainer.json&geo=WestUs2)
+
+Once the codespace opens (this may take several minutes), open a new terminal.
+
+#### VS Code Dev Containers
+
+You can run the project in your local VS Code Dev Container using the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers):
+
+1. Start Docker Desktop (install it if not already installed)
+2. Open the project:
+
+    [![Open in Dev Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/microsoft/multi-modal-customer-service-agent)
+3. In the VS Code window that opens, once the project files show up (this may take several minutes), open a new terminal.
+
+#### Local environment
+
+1. Install the required tools:
+   - [Node.js](https://nodejs.org/en)
+   - [Python >=3.11](https://www.python.org/downloads/)
+      - **Important**: Python and the pip package manager must be in the path in Windows for the setup scripts to work.
+      - **Important**: Ensure you can run `python --version` from console. On Ubuntu, you might need to run `sudo apt install python-is-python3` to link `python` to `python3`.
+   - [Powershell](https://learn.microsoft.com/powershell/scripting/install/installing-powershell)
+
+1. Clone the repo (`git clone https://github.com/microsoft/multi-modal-customer-service-agent`)
+1. Ensure env variables are set per [Setting up the environment](#2-setting-up-the-environment)
+1. Run this command to start the app:
+
+   Windows:
+
+   ```pwsh
+   cd voice_agent\app
+   pwsh .\start.ps1
+   ```
+
+   Linux/Mac:
+
+   ```bash
+   cd voice_agent/app
+   ./start.sh
+   ```
+
+1. The app is available on http://localhost:8765
+
+#### Deploy to Azure using azd
+
+1. Download and install [azd](https://aka.ms/azd/install) (Azure Developer CLI)
+
+1. Change directory to:
+
+   ```bash
+   cd voice_agent/app
+   ```
+
+1. Execute this command:
+
+   ```bash
+   azd up
+   ```
+
+1. When prompted provided the following names:
+   1. Environment name (used as the name of the new resource group to deploy into)
+   1. Azure Subscription to use
+   1. Azure location to use
 
 ---
-#### Navigation: [Home](../../README.md) | [Previous Section](../01_architecture/README.md) | [Next Section](../03_build/README.md)
+#### Navigation: [Home](../../README.md) | [Previous Section](../01_architecture/README.md) | [Next Section](../03_observability/README.md)

docs/03_build/README.md

@@ -0,0 +1,67 @@
+# Observe application behavior
+
+## Semantic Kernel and Observability
+
+It's worth taking a minute to read these two pages about observability with semantic kernel:
+[Observability in Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability)
+
+## Observability Architecture
+
+This application uses the [Open Telemetry](https://opentelemetry.io/) standard for shipping application tracing data to three telemetry destinations:
+- [.NET Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/overview?tabs=bash), an OTEL-based dashboard for viewing application behavior.
+- [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview), Azure's application observability service.
+- Console log
+
+## How does the app send telemetry data to observability destinations?
+
+The application uses the python `opentelemetry` sdk to send data to various destinations. The sdk uses base classes `LogProvider`, `SpanProvider` and `MetricProvider` to send data to various endpoints. In [utility.py]('..\..\voice_agent\app\backend\utility.py) the application uses specific implementations of those classes to send telemetry data to the destinations.
+
+## What observability resources are deployed to my resource group?
+
+### Aspire Dashboard
+
+The bicep infrastructure deploys Aspire Dashboard as a standalone ACA container called `aspire-dashboard`. You can find it in the list of resources in your resource group. Browse the dashboard by clicking the container link.
+
+![Logical architecture](../../media/aspire_dashboard.png)
+
+### Application Insights Resource
+
+The bicep infrastructure deploys an Application Insights resource where you can observe your telemetry data. Use the following features to observe semantic kernel behavior in App Insights:
+
+## Observe Application Behavior
+
+### Conduct a conversation with the customer service agents
+
+Conduct a conversation with the customer service agents following this sequence:
+- Ask about your upcoming hotel stay
+- Upgrade the room to a suite
+- Ask about your upcoming flights
+- Upgrade your seat to Business class
+
+### Observe application behavior in Aspire Dashboard
+
+- Go to your resource group in Azure Portal
+- Click on the `aspire-dashboard` resource
+- Click on the `Application URL` in the top right corner to navigate to the dashboard
+![Logs](../../media/aspire_dashboard_logs.png)
+- Observe the application log showing the conversation you conducted. Locate an entry: "Function hotel_tools-load_user_reservation_info invoking." and click on the Trace link. This navigates you to the Trace page for that function call.
+- Observe the details of the trace, such as the duration and other metadata.
+![Trace Details](../../media/aspire_dashboard_trace_details.png)
+- Click on the Traces navigation on the left.
+- Observe the sequence of function calls and confirm it matches the conversation you had. Drill into traces to see information as desired.
+![Traces](../../media/aspire_dashboard_traces.png)
+- Click on the Metrics navigation on the left.
+- Click on the function duration metric to see the duration on the most recent function calls.
+![Metrics](../../media/aspire_dashboard_metrics.png)
+
+
+## Application Environment Variables
+
+- You can disable specific OTEL destinations by adjusting the `TELEMETRY_SCENARIO` variable. By default it is `console,application_insights,aspire_dashboard`. You can remove any destination from the list by and re-deploy by this sequence:
+    - Go to the `backend` ACA app
+    - Click on containers
+    - Click on Environment Variables
+    - Adjust the value as desired
+    - Click `Deploy as new revision`. This will deploy a new revision of the backend app with your new value.
+---
+#### Navigation: [Home](../../README.md) | [Previous Section](../02_setup/README.md) | [Next Section](../04_explore/README.md)

docs/03_observability/README.md

@@ -5,7 +5,7 @@
 1.  First, create a new car rental tools plugin.  Create file car_rental_plugins.py in voice_agent\app\backend\agents\tools and add the following code:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
         from typing import List
@@ -44,7 +44,7 @@
 2. Create a car rental agent profile.  Create car_rental_agent_profile.yaml in voice_agent\app\backend\agents\agent_profiles and add the following:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
       name: car_rental_agent
@@ -67,7 +67,7 @@
 3. Modify the RTMiddleTier class to include the new car rental agent.  Add the following to the _load_agents method of voice_agent\app\backend\rtmt.py:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
         from agents.tools.car_rental_plugins import Car_Rental_Tools
@@ -113,7 +113,7 @@
 4.  Next steps are to update the system message to include the car rental agent along with the intent detection functionality.  Update the utility.py detect_intent method to now include car rental agent as part of the system message.  Add the below code to the messages block.
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
     - **car_rental_agent**: Deal with car rentals, vehicle reservations, changes, and general car rental policy questions.
@@ -128,7 +128,7 @@
     Create a JSON file for car rental policies.  Name the file car_rental_policy.json and store it in the data folder along with the flight & hotel policies:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
     [
@@ -156,7 +156,7 @@
     Update the plugin to setup Azure OpenAI client for embeddings, create get_embedding method to generate text embeddings.  Define the SearchClient class with semantic search functionality and finally initialize the search client.
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
     # Azure OpenAI client setup  
@@ -199,7 +199,7 @@
     Update the Car_Rental_Tools class to include policy search:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
         from openai import AzureOpenAI
@@ -225,7 +225,7 @@
     Create an embedding generation script & store it in  ./scripts/generate_car_rental_policy_embeddings.py :
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
     import json
@@ -272,7 +272,7 @@
     Run the embedding generation script in the terminal 
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```
     cd c:\Code\multi-modal-customer-service-agent\voice_agent\app\backend
@@ -739,4 +739,4 @@ This approach provides quantitative metrics to measure the impact of upgrading y
 - (phase-2 / consider once this functionality is integrated into the AI Foundry Agent Service post Build 2025): Exercise 4: real-time monitoring around generative quality & safety (RAI)
 
 ---
-#### Navigation: [Home](../../README.md) | [Previous Section](../04_deploy/README.md)
+#### Navigation: [Home](../../README.md) | [Previous Section](../03_observability/README.md)