Document the required info for dynamic routing

[bryanklewis]

The Event Hub ingest for Azure Data Explorer mentions that "you can also use dynamic routing, where your data includes the necessary routing information". What is the information required in an event to use dynamic routing? Example?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: d5808ed2-5d09-4c56-dc0b-4e3650f8fc16
• Version Independent ID: 10ebaf9d-0cc0-b59b-ae95-63bc7b1be4b5
• Content: Quickstart: Ingest data from Event Hub into Azure Data Explorer
• Content Source: articles/data-explorer/ingest-data-event-hub.md
• Service: data-explorer
• GitHub Login: @orspod
• Microsoft Alias: v-orspod

[angoyal-msft]

@bryanklewis Thanks for your feedback. As per the document, "you can also use dynamic routing, where your data includes the necessary routing information" signifies the information like table name, file format, and mapping which we need to provide while creating New Data Connection with static routing, is not needed in case of dynamic routing.

Please refer below screenshots in case of static and dynamic routing which might give you clear picture, what this document is referring to.

Static Routing:

Dynamic Routing:

@orspod Could you please review the document and update it appropriately, So that there is no confusion further.

@bryanklewis Please let us know if you still have any concerns.

[bryanklewis]

Understand the options in the portal, but that immediately invokes the question around what is required or rather, what data is Azure Data Explorer looking for inside a given event to fulfill those data elements? I assume some key in a key=value pair named a special way will allow ADX to find the table name, file format, and mapping. What is that key? Do those keys need to be at the root level? Anything special for Avro vs json vs csv? Looks like a very powerful feature to be able to dynamically route my event hub, just not enough info in order to model my events.

[bryanklewis]

Or does the ingest service use the event hub name as the table name, reads the format, and uses X to determine the mapping? Not sure...

[orspod]

@bryanklewis, thanks for your feedback. I am working on updating the documentation to answer your questions. In the meantime, @radennis, will you address the question above?

[JasonWHowell]

@bryanklewis I found the following information from preview timeframe, but I haven't tested it first hand, so the experience could vary in the current version. See if this helps for now:

Dynamic routing means that events read from a single Event Hub can land in different tables in your Kusto cluster.

This requires the following properties to be added to the EventData.Properties bag:

• Table - name (case sensitive) of the target Data Explorer table
• Format - payload format (might be any of supported data formats). Compressed payloads are not currently supported
• IngestionMappingReference - name of the ingestion mapping object (precreated on the database) to be used Kusto provides a control command to create/edit/delete an ingestion mapping.

[JasonWHowell]

#in-progress

[angoyal-msft]

@bryanklewis I have assigned this issue to @JasonWHowell to help you further.

[JasonWHowell]

@bryanklewis I don't have any extra detail yet. Did that bit I offered unblock you?

We'll explain more in the docs in our next revision, but it might be a few weeks.

[bryanklewis]

@JasonWHowell Yes, the EventData.Properties bag was the enabler. Adding the keys "Table", "Format", "IngestionMappingReference" with the appropriate values routed the events correctly from Event Hubs to Azure Data Explorer.

Example sending events from .net core in json format

• I added 2 tables to ADX, log1 and log2 with a json mapping and schema of {timestamp, message}. This aligns to the send events tutorial with .net core.
• I modified the async Task SendMessagesToEventHub to add properties designed to send all the odd events to ADX table log1 and all the even events to log2.

// Creates an event hub client and sends messages to the event hub.
private static async Task SendMessagesToEventHub(int numMessagesToSend)
{
    for (var i = 0; i < numMessagesToSend; i++)
    {
        try
        {
            // send every other table
            int tableId = ((i % 2) == 0) ? 2 : 1;
            string tableName = "log" + tableId.ToString();

            // serialize as json
            Dictionary<string, string> newMessage = new Dictionary<string, string>
            {
                { "timestamp", DateTime.UtcNow.ToString() },
                { "message", $"Message {i} for {tableName}" }
            };
            string newMessageJson = JsonConvert.SerializeObject(newMessage, Formatting.None);

            // set properties to route
            var newEvent = new EventData(Encoding.UTF8.GetBytes(newMessageJson));
            newEvent.Properties.Add("Table", tableName);
            newEvent.Properties.Add("Format", "json");
            newEvent.Properties.Add("IngestionMappingReference", "basicmsg");

            Console.WriteLine($"Sending message: {newMessage["message"]}");
            await eventHubClient.SendAsync(newEvent);
        }
        catch (Exception exception)
        {
            Console.WriteLine($"{DateTime.Now} > Exception: {exception.Message}");
        }

        await Task.Delay(10);
    }

    Console.WriteLine($"{numMessagesToSend} messages sent.");
}

[bryanklewis]

Schema:
[{"column":"timestamp","path":"$.timestamp","datatype":null,"transform":0},{"column":"message","path":"$.message","datatype":null,"transform":0}]

Result:

[JasonWHowell]

@bryanklewis 🥇 nice work.
Thanks for sharing your example to pay it forward to others.
CC @orspod we'll need to capture this request and make a doc soon.
Thanks, Jason

#please-close

[BryanTrach-MSFT]

@bryanklewis We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

[vjrantal]

@BryanTrach-MSFT @JasonWHowell @orspod The comments under this GitHub issue are useful and answers to the original question. I was wondering if this information is also captured in the documentation and if yes, could you add the link here as comment so that it is easy to find.

[JasonWHowell]

@vjrantal We have it in the plan, but I cannot add it inline in this doc. Quickstarts are meant to be short, single path forward to get started with a topic. We'll have to add a separate doc for the depths of the feature since this isn't the most "basic" scenario. Thanks, Jason cc: @orspod

[chaosmail]

It might be interesting for someone that EventData.Properties from the .NET SDK is also available in the Python SDK under the EventData.application_properties property. Hence in the Python SDK one needs to write something like the following for dynamic routing.

data = {}
tableName = 'TestTable'
tableMapping = 'TestMapping'

event = EventData(json.dumps(data).encode('UTF-8'))
event.application_properties = {
    'Table': tableName,
    'Format': "json",
    'IngestionMappingReference': tableMapping,
}
sender.send(event)

[arsenvlad]

@JasonWHowell is there now a doc that describes the more complex scenario?

[JasonWHowell]

@arsenvlad Let's check with @orspod since I'm no longer on this project.

[orspod]

@arsenvlad , a sample with dynamic routing was added as a link in the note in the Event Hub ingestion doc.
LMK if you need more info. Thanks, Ornat

[MikeWilcoxMicrosoft]

If my data is coming from IOT Hub where do I change the settings in IOT Hub so that below works ?

Dynamic routing means that events read from a single Event Hub can land in different tables in your Kusto cluster.

This requires the following properties to be added to the EventData.Properties bag:

Table - name (case sensitive) of the target Data Explorer table
Format - payload format (might be any of supported data formats). Compressed payloads are not currently supported
IngestionMappingReference - name of the ingestion mapping object (precreated on the database) to be used Kusto provides a control command to create/edit/delete an ingestion mapping.

[orspod]

@MikeWilcoxMicrosoft - thanks for your question. Best practice is to open a new request/issue but we are happy to assist you. @kerend - can you assist?