As the use of AI-based solutions grows, developers need to monitor the performance and costs of these applications. This README demonstrates how to use OpenTelemetry and Elastic to monitor Azure OpenAI applications.
The OpenAI API returns token counts and other useful information in each API response. Use this data can to calculate the cost of each API call based on Azure OpenAI's pricing.
OpenTelemetry is a powerful and widely adopted observability tool that you can use to monitor Azure OpenAI applications. This example uses Flask, Azure OpenAI API, and OpenTelemetry to instrument external calls and monitor the performance of Azure OpenAI API calls.
- Set up an Elastic Cloud Account and create a deployment.
- Install the required Python libraries.
pip3 install openai flask opentelemetry-distro[otlp] opentelemetry-instrumentation
- Set the following environment variable using your Azure OpenAI and Elastic credentials:
export AZURE_OPENAI_API_KEY="your-Azure-OpenAI-API-key"
export AZURE_OPENAI_ENDPOINT="your-Azure-OpenAI-endpoint"
export OPENAI_API_VERSION="your_api_version"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<your-otel-exporter-auth-header>"
export OTEL_EXPORTER_OTLP_ENDPOINT="your-otel-exporter-endpoint"
export OTEL_RESOURCE_ATTRIBUTES=service.name=your-service-name
-
Run
opentelemetry-instrument flask --app counter run -p 8000 -
Go to
http://localhost:8000/completion.
The example application demonstrates how to use OpenTelemetry to instrument a Flask application that makes Azure OpenAI API calls. The magic happens inside the monitor code that you can use freely to instrument your Azure OpenAI code.
The app uses monkey patching to modify the behavior of the chat.completions call at runtime so that the response metrics can be captured and sent to the OpenTelemetry OTLP endpoint (Elastic).
The calculate_cost function calculates the cost of a single request to the Azure OpenAI APIs based on the token counts and the model used.
Once the data is captured, you can view and analyze it in Elastic. You can use the captured data to build dashboards, monitor transactions and latency, and assess the performance of your Azure OpenAI service.