Azure DCR JSON Generator using Docker

A dockerized solution using Logstash (v8.8.0) to generate Azure Data Collection Rule (DCR) JSON Sample Files from unstructured logs.
An Apache2 Access log is used in this example. Limited testing has been done with single line log entries (Bind9 DNS & Apache2).

This is inspired from using a Logstash container to ship unstructured logs to Log Analytics using the Custom Table (Log) DCR API.
See cited resources below for a more detailed explanation.

Cited Resources:

Microsoft: Azure DCR API Tutorial
Medium: Azure DCR API Tutorial

Assumptions:

1. Docker is installed
2. You're connected to the Internet
3. You have the required access to build and run Docker containers
4. You can clone or download and unzip this repository

Instructions:

git clone https://github.com/dcodev1702/azure_dcr_json_generator.git

cd azure_dcr_json_generator

docker build -t logstash_azure_dcr .

chmod 755 generate_json_for_dcr_docker.sh

./generate_json_for_dcr_docker.sh

If you want to delete existing sample JSON files stored in ./tmp do the following

./generate_json_for_dcr_docker.sh 1

Post Condition:

The sample file: e.g. sampleFilexxxxxxxxxx.json will be in $(pwd)/tmp
You can use this file when creating a Custom Log in Log Analytics via DCR API at the KQL transformation stage

Below is the JSON from the sample file stored in ./tmp. Once uploaded to Log Analytrics - Custom Table (DCR)
you can write your KQL to normalize/transform the unstructured data to a structured format to your liking.
See KQL_Apache2_AccessLog_Transformation.txt for an example.



The Apache2 Access Log is now parsed (transformed) at ingestion time into several different distinct and queryable fields.