This article provides instructions for how to leverage and validate the use cases you deployed as part of the Enterprise Azure OpenAI Hub. The table of content is aligned with the high-level deployment options provided in the reference implementation, so jump straight to the relevant section based on your deployment choice.
When selecting the production deployment option, you have the option to deploy the 'On Your Data' use case, and the Image and Video recognition use case into a single Azure region, with all security capabilities enabled by default, such as usage of Private Endpoint deploying into an existing Virtual Network that is already created with the required address space. This deployment option is recommended for - of course - production, to conform with the security and compliance requirements for highly regulated Enterprise customers.
The following sections provide instructions for how to leverage and validate these use cases.
When using Enterprise Azure OpenAI in a secure context, you may already have experienced that curated tools such as Azure OpenAI Studio will not be fully available, due to the following reasons:
- Private Endpoints are being used for the Azure OpenAI service (and dependent services such as Azure AI Search and Azure Storage), and AI Studio cannot access the service over a private endpoint.
- Storage account is deployed using Entra ID for centralized RBAC, and SAS tokens are disabled for the storage account, which is a recommended security best practice.
- For both AI Search, and Azure OpenAI, local authentication is disabled, and similar to the storage account, RBAC is granted via Role Based Access Control (RBAC) and Entra ID. This means usage of API keys to access the services is not possible.
- Restricted oubound traffic from the Azure services to the internet, and the Azure services are not accessible from the internet.
So, how do you get started with the 'On Your Data' use case, to first ingest your data so it can be chunked and indexed, while Azure OpenAI generate embeddings for you to chat with your own Enterprise data?
If we assume you will take care of the connectivity aspect, such as accessing the Azure OpenAI instance over the Private Endpoint since you are either 1) connecting from a client that can access the service via your virtual network, or 2) have deployed Azure Bastion (and a VM) into a dedicated subnet that can access the endpoint, you can use the following PowerShell script and instructions to get started with the 'On Your Data' use case, as we will simply validate the end to end flow before you start building more sophisiacted applications on top of the Azure OpenAI instance.
Note: If you think the above assumptions aren't fair, and you think we should make it easier for you to also validate and get started even on a private endpoint using Azure Bastion from the get go - please let us know and create an Issue so we can triage and prioritize accordingly.
-
Upload files, such as text, images, and videos, to the storage account that you have created as part of the deployment, subject to the configuration you have selected (e.g., the setup highly recommends using Azure RBAC, disabling SAS tokens, and using customer-managed keys for encryption at rest, but if you have selected anything differently, you need to cater for that while getting the data into your storage account).
-
Use the Azure Open AI ingestion API to create one or more indexes in Azure AI Search, to start indexing the data that you have uploaded to the storage account, and make it available for the Azure Open AI instance to enable typical RAG use cases.
-
Use the Azure Open AI API to interact with the Azure Open AI instance, and start generating content based on the data that you have uploaded to the storage account, and indexed in Azure AI Search.
The following PowerShell scripts can be used to 1) start an ingestion job on Azure Open AI to ingest the data from the storage account into Azure AI Search, and 2) access the Azure Open AI API to start generating content based on the data that you have ingested.
Modify this script to provide the necessary values for the Azure Open AI endpoint, the embedding deployment name, the ingestion job name, the storage account endpoint, the storage container name, the storage resource ID, and the Azure AI Search endpoint.
# Ingestion job using Azure Open AI, AI Search, and Storage Account. The following snippet assumes Managed Identity is properly configured and has the necessary permissions to access the resources, and that the user has Open AI contributor role on the Azure Open AI resource.
# Azure Open AI configuration
$AzureOpenAIEndpoint = ""
$EmbeddingDeploymentName = ""
$IngestionJobName = ""
# Storage Configuration
$StorageAccountEndpoint = ""
$StorageContainerName = ""
$StorageResourceId = ""
# Azure AI search configuraton
$AzureAiSearchEndpoint = ""
# Get Token
$TokenRequest = Get-AzAccessToken -ResourceUrl "https://cognitiveservices.azure.com"
$MyToken = $TokenRequest.token
# Set Body (body must be present but empty for the request)
$Body = @'
{
}
'@
# AI Ingestion Request
$AzureOAIRequest = @{
Uri = "https://$($AzureOpenAIEndpoint)/openai/extensions/on-your-data/ingestion-jobs/$($IngestionJobName)?api-version=2023-10-01-preview"
Headers = @{
Authorization = "Bearer $($MyToken)"
'Content-Type' = 'application/json'
'storageEndpoint' = "https://$($StorageAccountEndpoint)"
'storageConnectionString' = "ResourceId=$($StorageResourceId)"
'storageContainer' = $StorageContainerName
'searchServiceEndpoint' = "https://$($AzureAiSearchEndpoint)"
'embeddingDeploymentName' = $EmbeddingDeploymentName
}
Body = $Body
Method = 'PUT'
}
$Response = Invoke-WebRequest @AzureOAIRequest
[Newtonsoft.Json.Linq.JObject]::Parse($Response.Content).ToString()
# Get Status on the ingestion job
$GetStatusRequest = @{
Uri = "https://$($AzureOpenAIEndpoint)/openai/extensions/on-your-data/ingestion-jobs/$($IngestionJob)?api-version=2023-10-01-preview"
Headers = @{
Authorization = "Bearer $($MyToken)"
}
Method = 'GET'
}
$GetResponse = Invoke-WebRequest @GetStatusRequest
[Newtonsoft.Json.Linq.JObject]::Parse($GetResponse.Content).ToString()
Modify this script to provide the necessary values for the Azure Open AI endpoint, the embedding deployment name, and the model name.
#T he following snippet assumes Managed Identity is properly configured and has the necessary permissions to access the resources, and that the user has Open AI reader role on the Azure Open AI resource.
# Azure Open AI configuration
$AzureOpenAIEndpoint = ""
$DeploymentName = ""
$EmbeddingDeploymentName = ""
$Prompt = ""
# Azure AI search configuraton
$AzureAiSearchEndpoint = ""
$IndexName = ""
# Get Token
$TokenRequest = Get-AzAccessToken -ResourceUrl "https://cognitiveservices.azure.com"
$MyToken = $TokenRequest.token
# Form the request body towards the Azure Open AI API endpoint, with AzureCognitiveSearch added as dataSource for RAG
$Body = @"
{
"dataSources": [
{
"type": "AzureCognitiveSearch",
"parameters": {
"endpoint": "https://$($AzureAiSearchEndpoint)",
"indexName": "$($IndexName)",
"embeddingDeploymentName": "$($EmbeddingDeploymentName)"
}
}
],
"messages": [
{
"role": "system",
"content": "You are an AI assistant that helps people find information."
},
{
"role": "user",
"content": "$($Prompt)"
}
]
}
"@
# AI Request
$AzureOAIRequest = @{
Uri = "https://$($AzureOpenAIEndpoint)/openai/deployments/$($DeploymentName)/extensions/chat/completions?api-version=2023-10-01-preview"
Headers = @{
Authorization = "Bearer $($MyToken)"
'Content-Type' = 'application/json'
}
Method = 'POST'
Body = $Body
#UseBasicParsing = $true
}
$Response = Invoke-WebRequest @AzureOAIRequest
[Newtonsoft.Json.Linq.JObject]::Parse($Response.Content).ToString()
If you quickly need to explore what this is all about without conforming to all the networking pre-requisites. Perhaps you need spin up an internal demo to showcase core capabilities and relevance of the use cases, or even doing a larger hackathon with other engineers and architects to enumerate viable architectural patterns and best practices for both the front-end and the back-end, this is the deployment option for you.
Today, you have the following options to choose from when deploying the Proof of Concept:
-
'On Your Data' with sample Web Application.
-
Multi region deployment with APIM.
Note: Before proceeding with the steps below, ensure you have followed the same instructions in the 'On Your Data' section in the Production Deployment section, to get the data into the storage account, and ingested into Azure AI Search.
When selecting the "Proof of Concept" deployment intent, you have the option to deploy the 'On Your Data' use case, and a sample Web Application into a single Azure region. This deployment option is recommended for proof of concept, to validate the power of Azure OpenAI's RAG capabilities, and to validate the end to end flow of ingesting data, generating embeddings, and decoding the embeddings to generate text via a sample Web Application.
Deploying the Sample App will in additon to creating the Resource Group containing the Azure OpenAI related services, create a dedicated Resource Group in your Azure subscription containing an App Service Plan, and the Web App as seen in the following image.
When you select the option to deploy the Sample App, you are asked if you want to use existing - or create a new Application Registration in Entra ID. If using an existing App Registration, you can select the App Registration directly in the portal and the clientId will be provided to the Web App configuration, but you must also provide the client secret, which is only visible when you create the App Registration. If you create a new App Registration, you will be provided with the clientId and client secret, and you can use these values to configure the Web App.
If you selected to not create an App Registration or use an existing one, you can at any time navigate to the App Registration in the Azure portal, and select the "Certificates & secrets" section to create a new client secret, and use the value to configure the Web App.
Once this is completed, there's one final step that must be completed that we are currently not able to automate entirely during deployment.
- Navigate to the App Registration in Entra ID, and search for the display name of the App Registration.
- Select the App Registration, and in the new window in the "Overview" section, you can see an option for "Redirect URIs". Select this option, and add the following redirect URI to the list of redirect URIs, where the Web App name is in the following format: "https://--wa.azurewebsites.net/", and add the ".auth/login/aad/callback" to the end of the URL like below.
https://<deploymentPrefi-swedencentral-wa.azurewebsites.net/.auth/login/aad/callback
- Once done, navigate to the Web App in the Azure portal and restart the Web App to ensure the changes are picked up.
- You can now access the Web App, and you should see the following page.
When selecting the "Proof of Concept" deployment intent, you have the option to deploy Enterprise Azure OpenAI Hub into multiple Azure regions where Azure API Management (APIM) is used to provide a single endpoint for the use cases, and the use cases are deployed into a single Azure region. This deployment option is recommended for proof of concept, to validate the power of API Mangement's policy logic to distribute traffic, provide retry logic, and to do error handling.
When you have deployed Enterprise Azure OpenAI into multiple regions, you will see two Resource Groups in your Azure subscription, one Resource Group for the APIM instance together with the rest of the Azure services in the first region, and a second Resource Group in the second region. Assuming you have enabled RBAC and Managed Identity for the Azure services, you can follow the following steps to validate you can access the Azure OpenAI instances from APIM connected via Managed Identity.
In the following example, the deployments are made into "Sweden Central" and "Switzerland North" regions, and the APIM instance is deployed into the "West Europe" region.
And checking the "Rag-rg-swedencentral" Resource Group, you will see the following resources deployed, including APIM in the "West Europe" region.
- Navigate to the APIM instance in the "West Europe" region, and select the "Named Values" section, and you will see the "backend1" and "backend2" named values. These are the Azure OpenAI instances deployed in the "Sweden Central" and "Switzerland North" regions, and you can use these named values to access the Azure OpenAI instances from APIM.
- Access the APIM instance in the "West Europe" region, and navigate to the "APIs" section, and you will see the APIs that are deployed. You should see the "Azure OpenAi Service API".
- Open the "Azure OpenAi Service API", and navigate to the "Test" section, and you can test the API to validate you can access the Azure OpenAI instances from APIM.
- Select the "POST - Creates a completion for the chat message" API operation and provide the following input parameters to test the API:
- API-version: 2023-12-01-preview
- Deployment-id: The deployment name of the GPT4 you created during the deployment
Note: If you did not deploy any GPT4 model, or did not select the 'On Your Data' use case, you will not have a deployment name to provide. To deploy a model to your Azure OpenAI instances, navigate to https://oai.azure.com and use the same deployment name for both instances, or APIM will not be able to access the Azure OpenAI model deployments in a consistent way.
- Copy and paste the following JSON into the request body (you can obviously change the prompt to whatever you like):
{
"messages": [
{
"role": "system",
"content": "You are an AI assistant that helps people find information."
},
{
"role": "user",
"content": "Who is the CEO of Microsoft?"
}
]
}Hit the "Send" button, and you should see a response from the Azure OpenAI instance in the "Sweden Central" region.
And finally, the response from the Azure OpenAI instance in the "Sweden Central" region.
You have now successfully validated that you can access the Azure OpenAI instances from APIM.