Add support for azure-eventhub input in integrations

[narph]

What does this PR do?

Add support for azure-eventhub input in integrations

• we allowed some consistency to the azure-eventhub fields in the other azure logs data streams
• the option Parse azure message will do limited json parsing of the message including the most generic fields like subscriptionId and resourceId (frequently asked by azure users). I have added below the 2 examples of events.

Limitations:

• the reason the azure-eventhub input is in the Azure Logs package is because it shares configuration options with the rest of the azure log data streams. This could cause some confusion here as the azure-eventhub input is not limited to collecting logs only, can collect any type events.
• the data stream for the input has the type logs , I assume the allowed types are logs, metrics, traces. What do we do with inputs as this or kafka etc which can retrieve any type of messages users are streaming to their eventhubs. Should we add an additional type messages, events or decide on one of the existing ones?

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

Related issues

Screenshots

overlapping error is fixed in future release.

Ex of event not parsed:

   {
        "_index" : ".ds-logs-azure.eventhub-default-2021.10.18-000001",
        "_type" : "_doc",
        "_id" : "WExik3wBs4hpITkzfNnM",
        "_score" : null,
        "_source" : {
          "agent" : {
            "hostname" : "docker-fleet-agent",
            "name" : "docker-fleet-agent",
            "id" : "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
            "type" : "filebeat",
            "ephemeral_id" : "42a183d5-df19-4008-8776-d9765bc01d50",
            "version" : "7.15.0"
          },
          "elastic_agent" : {
            "id" : "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
            "version" : "7.15.0",
            "snapshot" : true
          },
          "message" : """{"ReleaseVersion":"6.2021.39.11+d1f0e29.release_2021w39_az","RoleLocation":"North Europe","callerIpAddress":"77.170.179.229","category":"Administrative","correlationId":"a75a0752-ebbb-42bf-831f-43788a8c1193","durationMs":"0","identity":.....,"eventCategory":"Administrative","hierarchy":"4fa94b7d-a743-486f-abcc-6c276c44cf4b/5341238b-665c-4eb4-b259-b250371ae430/7657426d-c4c3-44ac-88a2-3b2cd59e6dba","message":"Microsoft.ServiceBus/namespaces/delete"},"resourceId":"/SUBSCRIPTIONS/7657426D-C4C3-44AC-88A2-3B2CD59E6DBA/RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.SERVICEBUS/NAMESPACES/TESTOBS","resultSignature":"Started.","resultType":"Start","tenantId":"4fa94b7d-a743-486f-abcc-6c276c44cf4b","time":"2021-10-15T09:08:29.9268177Z"}""",
          "azure-eventhub" : {
            "sequence_number" : 1215,
            "consumer_group" : "$Default",
            "offset" : 274878093752,
            "eventhub" : "insights-activity-logs",
            "enqueued_time" : "2021-10-15T09:14:25.419Z"
          },
          "tags" : [
            "azure-eventhub"
          ],
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2021-10-18T12:31:17.027Z",
          "ecs" : {
            "version" : "1.12.0"
          },
          "data_stream" : {
            "namespace" : "default",
            "type" : "logs",
            "dataset" : "azure.eventhub"
          },
          "host" : {
            "hostname" : "docker-fleet-agent",
            "os" : {
              "kernel" : "4.19.128-microsoft-standard",
              "codename" : "Core",
              "name" : "CentOS Linux",
              "family" : "redhat",
              "type" : "linux",
              "version" : "7 (Core)",
              "platform" : "centos"
            },
            "containerized" : true,
            "ip" : [
              "172.27.0.7"
            ],
            "name" : "docker-fleet-agent",
            "id" : "6505f7ca36739e7eb909bdb52bf3ec18",
            "mac" : [
              "02:42:ac:1b:00:07"
            ],
            "architecture" : "x86_64"
          },
          "event" : {
            "agent_id_status" : "verified",
            "ingested" : "2021-10-18T12:31:17Z",
            "kind" : "event",
            "dataset" : "azure.eventhub"
          }
        },
        "sort" : [
          1634560277027
        ]
      }

if parse_message is enabled:

 {
        "_index" : ".ds-logs-azure.eventhub-default-2021.10.18-000001",
        "_type" : "_doc",
        "_id" : "l0xZk3wBs4hpITkz-syS",
        "_score" : null,
        "_source" : {
          "agent" : {
            "hostname" : "docker-fleet-agent",
            "name" : "docker-fleet-agent",
            "id" : "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
            "ephemeral_id" : "6580f640-c603-4f59-93fe-b4a546a54630",
            "type" : "filebeat",
            "version" : "7.15.0"
          },
          "elastic_agent" : {
            "id" : "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
            "version" : "7.15.0",
            "snapshot" : true
          },
          "azure-eventhub" : {
            "sequence_number" : 1189,
            "consumer_group" : "$Default",
            "offset" : 261993165400,
            "eventhub" : "insights-activity-logs",
            "enqueued_time" : "2021-10-18T03:17:49.601Z"
          },
          "tags" : [
            "parse_message",
            "azure-eventhub"
          ],
          "cloud" : {
            "provider" : "azure"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2021-10-18T03:14:41.682Z",
          "ecs" : {
            "version" : "1.12.0"
          },
          "data_stream" : {
            "namespace" : "default",
            "type" : "logs",
            "dataset" : "azure.eventhub"
          },
          "host" : {
            "hostname" : "docker-fleet-agent",
            "os" : {
              "kernel" : "4.19.128-microsoft-standard",
              "codename" : "Core",
              "name" : "CentOS Linux",
              "type" : "linux",
              "family" : "redhat",
              "version" : "7 (Core)",
              "platform" : "centos"
            },
            "containerized" : true,
            "ip" : [
              "172.27.0.7"
            ],
            "name" : "docker-fleet-agent",
            "id" : "6505f7ca36739e7eb909bdb52bf3ec18",
            "mac" : [
              "02:42:ac:1b:00:07"
            ],
            "architecture" : "x86_64"
          },
          "event" : {
            "agent_id_status" : "verified",
            "ingested" : "2021-10-18T12:22:00Z",
            "kind" : "event",
            "dataset" : "azure.eventhub"
          },
          "azure" : {
            "subscription_id" : "7657426D-C4C3-44AC-88A2-3B2CD59E6DBA",
            "resource" : {
              "provider" : "MICROSOFT.APPPLATFORM/SPRING",
              "name" : "RK-SPRINGCLOUD-TEST",
              "id" : "/SUBSCRIPTIONS/7657426D-C4C3-44AC-88A2-3B2CD59E6DBA/RESOURCEGROUPS/RK-TEST-RESOURCES/PROVIDERS/MICROSOFT.APPPLATFORM/SPRING/RK-SPRINGCLOUD-TEST",
              "group" : "RK-TEST-RESOURCES"
            },
            "eventhub" : {
              "level" : "Information",
              "correlationId" : "e0521f42-19ff-40e3-bc18-b3ec1f91cbf0",
              "operationName" : "Microsoft.Resourcehealth/healthevent/Resolved/action",
              "category" : "ResourceHealth",
              "resultType" : "Resolved",
              "properties" : {
                "eventCategory" : "ResourceHealth",
                "eventProperties" : {
                  "cause" : "Unknown",
                  "currentHealthStatus" : "Available",
                  "details" : "Unknown",
                  "previousHealthStatus" : "Degraded",
                  "title" : "Unknown",
                  "type" : "Unknown"
                }
              }
            }
          }
        },
        "sort" : [
          1634526881682
        ]
      }

@P1llus , @ruflin , @ravikesarwani , @andrewkroh, @jsoriano would love to hear your thoughts here

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-11-11T13:30:31.078+0000

• Duration: 13 min 8 sec

• Commit: 06ca2f1

Test stats 🧪

Test	Results
Failed	0
Passed	79
Skipped	0
Total	79

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[ruflin]

For the type, lets stick to logs for the moment. But as you pointed out, ideally we would route it to the correct data stream. For me the end goal is to have document based event routing in place (elastic/elasticsearch#63798) and then the package would only need to configure this. But it is not something we have yet. This leaves us with 2 places to route the data: On the input side or in the ingest pipeline. What values do we have available inside the event to make a decision on where it should end up?

[narph]

For the type, lets stick to logs for the moment. But as you pointed out, ideally we would route it to the correct data stream. For me the end goal is to have document based event routing in place (elastic/elasticsearch#63798) and then the package would only need to configure this. But it is not something we have yet. This leaves us with 2 places to route the data: On the input side or in the ingest pipeline. What values do we have available inside the event to make a decision on where it should end up?

the routing would be great, unfortunately, there isn't much that Azure gives away on detecting which is a log event, trace, metric, or even key value format (for key vault resources for example).

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[jsoriano]

Looks good.

[jsoriano]

There is a rename to move message to event.original, is this outdated?

[jsoriano]

Clarified offline, this pipeline is conditionally executed:

  - pipeline:
      if: "ctx?.tags != null && ctx.tags.contains('parse_message')"
      name: '{{ IngestPipeline "parsed-message" }}'