Azure Operator Insights data ingestion samples

Code samples and guidance demonstrating how to ingest data into an Azure Operator Insights Data Product.

Azure Operator Insights offers a range of options for ingesting data into Data Products:

• The Azure Operator Insights ingestion agent, which runs on-premises or on an Azure VM. The agent can consume data from different sources and upload the data to an Azure Operator Insights Data Product. The agent currently supports ingestion by:
  • Pulling data from an SFTP server
  • Terminating a TCP stream of enhanced data records (EDRs) from the Affirmed MCC.
• Ingestion via other methods:
  • Existing Microsoft or third-party tools which upload data to Azure. E.g. AzCopy or Azure Data Factory.
  • Custom code, e.g. using the Azure CLI or Azure SDK.

The Azure Operator Insights ingestion agent is the recommended ingestion method for the use cases it supports.

For other use cases, this repository provides samples of ingestion using common Microsoft tools or custom code.

Getting Started

Read this README then explore the samples directory: samples. When you want more in-depth guidance on implementing ingestion, read INGESTION-OVERVIEW.md.

Prerequisites

For most of the samples, you need:

• An Azure Operator Insights Data Product
• A Microsoft Entra identity to use to run the sample. The identity must have the following roles:
  • Reader role on your Data Product
  • Key Vault Secrets User role on the managed key vault associated with your Data Product.

Some samples have different prerequisites, which are listed in the code or README files for each sample (e.g. for Azure Databricks).

Quickstart

1. git clone https://github.com/Azure-Samples/operator-insights-data-ingestion.git
2. cd operator-insights-data-ingestion/samples
3. Choose a sample, update the required parameters to work with your Data Product and data source, and run the sample.

Next steps

The samples in this repository demonstrate how to implement basic ingestion into a Data Product, and they are provided for prototyping and educational purposes only.

Before using these samples as part of a production system, you should consider what additional logging, security, performance and resilience you need to add to make the code suitable for production use.

Samples

End-to-end samples

• Azure CLI
• AzCopy:
  • Upload from local directory
  • Copy from an Azure Storage Account
• Python, using Azure SDK for Python
• Azure Databricks

Most end-to-end samples follow three steps:

1. Find the name of the managed key vault associated with the Data Product, by querying the Data Product to find the Data Product's unique ID
2. Authenticate with the managed key vault and fetch the secret containing the ingestion SAS URL for the Data Product.
3. Using the ingestion URL/storage account name and ingestion SAS token, upload data to the ingestion endpoint of the Data Product.

Helper code snippets

• Setup script: Use the Azure CLI to perform steps 1 and 2 above, saving the parameters as environment variables: samples/snippets/az-cli-set-ingestion-params.sh

Common parameters

Most samples require the same parameters:

• resourceGroup: the resource group where the Data Product is deployed
• dataProductName: the name of the Data Product to upload data to
• dataType: the data type name for the data to upload, e.g. edr or pmstat. The valid data types for a Data Product are listed on the Data Management > Data types pane of the Azure Portal for the Data Product.
• sourceDataFilePath: the path to a local directory containing the files you want to upload to the Data Product

All samples must authenticate with the managed key vault that is associated with the Data Product. This authentication is handled differently depending on the sample.

Resources

• To learn about Azure Operator Insights, visit Azure Operator Insights documentation.
• To ingest EDRs from an Affirmed MCC, or to ingest files from an SFTP server, you can use the Azure Operator Insights Ingestion Agent.
• For details of the Azure Operator Insights (network-analytics) Azure CLI extension, see Azure CLI reference: az network-analytics.
• For guidance on using AzCopy to copy files to an Azure Storage Account (including Azure Operator Insights Data Product ingestion storage), see Get started with AzCopy.
• To learn more about developing your own Azure Operator Insights Data Product, visit Azure Operator Insights Data Product factory.