docs: add observability docs

Author: Adrian Wright

README.md

@@ -43,4 +43,8 @@ The purpose of this section is to help you learn how to best leverage this solut
 
 You will experience the benefits of a production-grade AI application when it comes to speed of innovation, maintainability and scalability.
 
-This is a great way to get familiar with the solution and understand how to best leverage it for your own use cases.
+This is a great way to get familiar with the solution and understand how to best leverage it for your own use cases.
+
+### [Explore application observability](docs/06_observability/README.md)
+
+Visit this section to observe application behavior and performance through various tools. See logs, traces, and metrics to understand what your application was doing during the realtime conversation.

docs/05_explore/README.md

@@ -5,7 +5,7 @@
 1.  First, create a new car rental tools plugin.  Create file car_rental_plugins.py in voice_agent\app\backend\agents\tools and add the following code:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
         from typing import List
@@ -44,7 +44,7 @@
 2. Create a car rental agent profile.  Create car_rental_agent_profile.yaml in voice_agent\app\backend\agents\agent_profiles and add the following:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
       name: car_rental_agent
@@ -67,7 +67,7 @@
 3. Modify the RTMiddleTier class to include the new car rental agent.  Add the following to the _load_agents method of voice_agent\app\backend\rtmt.py:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
         from agents.tools.car_rental_plugins import Car_Rental_Tools
@@ -113,7 +113,7 @@
 4.  Next steps are to update the system message to include the car rental agent along with the intent detection functionality.  Update the utility.py detect_intent method to now include car rental agent as part of the system message.  Add the below code to the messages block.
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
     - **car_rental_agent**: Deal with car rentals, vehicle reservations, changes, and general car rental policy questions.
@@ -128,7 +128,7 @@
     Create a JSON file for car rental policies.  Name the file car_rental_policy.json and store it in the data folder along with the flight & hotel policies:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
     [
@@ -156,7 +156,7 @@
     Update the plugin to setup Azure OpenAI client for embeddings, create get_embedding method to generate text embeddings.  Define the SearchClient class with semantic search functionality and finally initialize the search client.
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
     # Azure OpenAI client setup  
@@ -199,7 +199,7 @@
     Update the Car_Rental_Tools class to include policy search:
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
         from openai import AzureOpenAI
@@ -225,7 +225,7 @@
     Create an embedding generation script & store it in  ./scripts/generate_car_rental_policy_embeddings.py :
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```python
     import json
@@ -272,7 +272,7 @@
     Run the embedding generation script in the terminal 
 
     <details>
-    <summary> Click to expand/collaspse</summary>
+    <summary> Click to expand/collapse</summary>
 
     ```
     cd c:\Code\multi-modal-customer-service-agent\voice_agent\app\backend
@@ -739,4 +739,4 @@ This approach provides quantitative metrics to measure the impact of upgrading y
 - (phase-2 / consider once this functionality is integrated into the AI Foundry Agent Service post Build 2025): Exercise 4: real-time monitoring around generative quality & safety (RAI)
 
 ---
-#### Navigation: [Home](../../README.md) | [Previous Section](../04_deploy/README.md)
+#### Navigation: [Home](../../README.md) | [Previous Section](../04_deploy/README.md) | [Next Section](../06_observability/README.md)

docs/06_observability/README.md

@@ -0,0 +1,62 @@
+# Observe application behavior
+
+## Semantic Kernel and Observability
+
+It's worth taking a minute to read these two pages about observability with semantic kernel:
+[Observability in Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability)
+
+## Observability Architecture
+
+This application uses the [Open Telemetry](https://opentelemetry.io/) standard for shipping application tracing data to three telemetry destinations:
+- [.NET Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/overview?tabs=bash), an OTEL-based dashboard for viewing application behavior.
+- [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview), Azure's application observability service.
+- Console log
+
+## How does the app send telemetry data to observability destinations?
+
+The application uses the python `opentelemetry` sdk to send data to various destinations. The sdk uses base classes `LogProvider`, `SpanProvider` and `MetricProvider` to send data to various endpoints. In [utility.py]('..\..\voice_agent\app\backend\utility.py) the application uses specific implementations of those classes to send telemetry data to the destinations.
+
+## What observability resources are deployed to my resource group?
+
+### Aspire Dashboard
+
+The bicep infrastructure deploys Aspire Dashboard as a standalone ACA container called `aspire-dashboard`. You can find it in the list of resources in your resource group. Browse the dashboard by clicking the container link.
+
+![Logical architecture](../../media/aspire_dashboard.png)
+
+### Application Insights Resource
+
+The bicep infrastructure deploys an Application Insights resource where you can observe your telemetry data. Use the following features to observe semantic kernel behavior in App Insights:
+
+## Observe Application Behavior
+
+### Conduct a conversation with the customer service agents
+
+Conduct a conversation with the customer service agents following this sequence:
+- Ask about your upcoming hotel stay
+- Upgrade the room to a suite
+- Ask about your upcoming flights
+- Upgrade your seat to Business class
+
+### Observe application behavior in Aspire Dashboard
+
+- Go to your resource group in Azure Portal
+- Click on the `aspire-dashboard` resource
+- Click on the `Application URL` in the top right corner to navigate to the dashboard
+- Observe the application log showing the conversation you conducted. Locate an entry: "Function hotel_tools-load_user_reservation_info invoking." and click on the Trace link. This navigates you to the Trace page for that function call.
+- Observe the details of the trace, such as the duration and other metadata.
+- Click on the Traces navigation on the left.
+- Observe the sequence of function calls and confirm it matches the conversation you had. Drill into traces to see information as desired.
+- Click on the Metrics navigation on the left.
+- Click on the function duration metric to see the duration on the most recent function calls.
+
+## Application Environment Variables
+
+- You can disable specific OTEL destinations by adjusting the `TELEMETRY_SCENARIO` variable. By default it is `console,application_insights,aspire_dashboard`. You can remove any destination from the list by and re-deploy by this sequence:
+    - Go to the `backend` ACA app
+    - Click on containers
+    - Click on Environment Variables
+    - Adjust the value as desired
+    - Click `Deploy as new revision`. This will deploy a new revision of the backend app with your new value.
+---
+#### Navigation: [Home](../../README.md) | [Previous Section](../05_explore/README.md)