[Azure Logs] Set enabled: false as default for all data streams #4373

zmoog · 2022-10-04T10:21:36Z

What does this PR do?

Set enabled: false as the default status for all Azure Logs data streams.

Why?

When users install the collective Azure Logs integration, all data streams start as enabled. With this default setting, we end up in a situation where Filebeat spawns a lot of "azureeventhub" behind the scenes: one for each existing data stream, and all receiving data from the same event hub:

                                   ┌────────────────┐                      
                                   │   azurelogs    │                      
                                   │ <<event hub>>  │                      
                                   └────────────────┘                      
                                            │                              
                                            │                              
                                            │                              
              ┌──────────────┬──────────────┼──────────────────────┐       
              │              │              │                      │       
          $Default       $Default       $Default               $Default    
     ┌────────┼──────────────┼──────────────┼──────────────────────┼──────┐
     │        ▼              ▼              ▼                      ▼      │
     │ ┌────────────┐ ┌────────────┐ ┌────────────┐           ┌─────────┐ │
     │ │   signin   │ │   audit    │ │  activity  │           │   ...   │ │
     │ │ <<input>>  │ │ <<input>>  │ │ <<input>>  │  .....    │<<input>>│ │
     │ └────────────┘ └────────────┘ └────────────┘           └─────────┘ │
     │                                                                    │
     └─Filebeat───────────────────────────────────────────────────────────┘

Many (data streams) to one (event hub) is not a good default.

In this case, each input receives a copy of each message published on the event hub. Some data streams have pipeline that can filter incoming logs by category (sign-in, audit) but other will try to ingest any log category (generic event hub and platform logs).

A better default is to have all input disabled.

In this case, Filebeat spawns a new input only if the user explicitly enables the data stream, with no unintended input running with no reason.

In addition, since the Azure Logs version 1.4.1 update, we started to strongly recommend using one event hub per log group (a set of related log categories).

Here's the recommended setup using multiple event hubs, where each input receives only the intended log categories:

            ┌────────────────┐             ┌───────────────┐       ┌────────────────┐
            │     adlogs     │             │ activitylogs  │       │  firewalllogs  │
            │ <<event hub>>  │             │ <<event hub>> │       │ <<event hub>>  │
            └────────────────┘             └───────────────┘       └────────────────┘
                     │                             │                        │        
                     │                             │                        │        
                     │                             │                        │        
              ┌──────┴───────┐                     │                        │        
              │              │                     │                        │        
          $Default       $Default              $Default                 $Default     
     ┌────────┼──────────────┼─────────────────────┼────────────────────────┼───────┐
     │        ▼              ▼                     ▼                        ▼       │
     │ ┌────────────┐ ┌────────────┐        ┌────────────┐           ┌────────────┐ │
     │ │   signin   │ │   audit    │        │  activity  │           │  firewall  │ │
     │ │ <<input>>  │ │ <<input>>  │        │ <<input>>  │           │ <<input>>  │ │
     │ └────────────┘ └────────────┘        └────────────┘           └────────────┘ │
     │                                                                              │
     └─Filebeat─────────────────────────────────────────────────────────────────────┘

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

elasticmachine · 2022-10-04T10:36:37Z

🚀 Benchmarks report

Package `azure` 👍(0) 💚(1) 💔(9)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`springcloudlogs`	3816.79	2049.18	-1767.61 (-46.31%)	💔
`activitylogs`	1280.41	623.05	-657.36 (-51.34%)	💔
`application_gateway`	2074.69	1278.77	-795.92 (-38.36%)	💔
`auditlogs`	2207.51	1180.64	-1026.87 (-46.52%)	💔
`firewall_logs`	2004.01	1459.85	-544.16 (-27.15%)	💔
`identity_protection`	3267.97	1700.68	-1567.29 (-47.96%)	💔
`platformlogs`	4545.45	2132.2	-2413.25 (-53.09%)	💔
`provisioning`	2777.78	2008.03	-769.75 (-27.71%)	💔
`signinlogs`	1776.2	957.85	-818.35 (-46.07%)	💔

To see the full report comment with /test benchmark fullreport

elasticmachine · 2022-10-04T10:36:49Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-10-12T08:43:51.964+0000
Duration: 15 min 17 sec

Test stats 🧪

Test	Results
Failed	0
Passed	123
Skipped	0
Total	123

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-10-04T10:37:00Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`10/10`)	💚
Files	86.364% (`19/22`)	👎 -10.924
Classes	86.364% (`19/22`)	👎 -10.924
Methods	83.422% (`156/187`)	👎 -6.901
Lines	85.011% (`2779/3269`)	👎 -6.628
Conditionals	100.0% (`0/0`)	💚

endorama · 2022-10-04T13:11:22Z

Have we evaluated what will happen to users upgrading to this version? Will this alter significantly the behaviour of the current package?

kaiyan-sheng · 2022-10-04T14:15:43Z

+1 on @endorama's comment. This is a great change and let's make sure it's not breaking existing users.

zmoog · 2022-10-04T14:48:07Z

IIRC I tested this scenario, and the enabled setting is evaluated at install time only.

I will rerun the upgrade scenario and document it in the description.

zmoog · 2022-10-04T15:26:29Z

@endorama @kaiyan-sheng, I tested upgrading Azure Logs integration from version 1.3.0 to version 1.5.1.

In the following video, I:

Visit an existing integration version 1.3.0 to see that half of the data streams are enabled.
Run the upgrade
Check if the existing data streams are still enabled or disabled, as in version 1.3.0.
Check if the new Application Gateway data stream added in 1.5.0 is disabled (default setting).

CleanShot.2022-10-04.at.17.17.04.mp4

kaiyan-sheng

Looks good to me! Thank you for testing and the video! Could you add a changelog to this please?

zmoog · 2022-10-04T15:40:19Z

I have found only this downside.

When I try to install an "individual" integration (in the example below, Azure Active Directory) instead of the "collective" one (Azure Logs) the data stream is disabled.

This is both not great and okay.

Not great: you need to enable something, or the integration will non collect data.
Okay: Azure Active Directory has multiple data streams from different sub-services (Identity Protection and Provisioning service), so picking the used one can benefit efficiency.

Video:

CleanShot.2022-10-04.at.17.31.43.mp4

kaiyan-sheng · 2022-10-05T16:15:33Z

@zmoog I think it's Okay in this case because there are different data streams(audit, provisioning, sign-in...) under Azure Active Directory for users to choose from. If I understand correctly, we recommend users to use different event hubs for these log types under Azure Active Directory right?

zmoog · 2022-10-11T12:12:44Z

@zmoog I think it's Okay in this case because there are different data streams(audit, provisioning, sign-in...) under Azure Active Directory for users to choose from.

Yep, the idea is to invite users to enable only the data streams they use.

We also have other integrations with just one data stream. It would be disabled also for one data stream integration. Do you see this as a problem?

If I understand correctly, we recommend users to use different event hubs for these log types under Azure Active Directory right?

Each azureeventhub input connected to an event hub receives a copy of every message published.

Here's an example from Azure AD where:

Azure AD produces a new sign-in log with ID 42.
A diagnostic setting exports it to an event hub.
The two azureeventhub inputs connected to the event hub receive a copy of the sign-in log with ID 42 each.
The signinlogs data stream ingests it.
The auditlogs data stream drops it.

                                                                                             
                                                                         ┌──────────────────┐
                                                              signin:42  │    signinlogs    │
               signin:42                 signin:42                  ┌───▶│ <<data stream>>  │
  ┌─────────────┐        ┌──────────────┐        ┌──────────────┐   │    └──────────────────┘
  │  Azure AD   │        │  Diagnostic  │        │ logeventhub  │   │                        
  │ <<service>> │───────▶│   settings   │───────▶│<<event hub>> │───┤                        
  └─────────────┘        └──────────────┘        └──────────────┘   │                        
                                                                    │    ┌──────────────────┐
                                                                    │    │    auditlogs     │
                                                                    └───▶│ <<data stream>>  │
                                                             signin:42   └──────────────────┘

The end goal is to avoid the scenario where users install the Azure Logs integration, they export log categories for only 2-3 of them. But, since the default setting is "enabled", they end up having one azureeventhub receiving data for each data stream.

Today we already have 10 data streams in the Azure Logs integration. We already have new ones planned, so this inefficiency and potential problems will probably worsen over time.

In the latest Azure Logs doc update, we expanded the event hub section advising about properly setting up the integration. High-volume deployment would probably benefit from a dedicated event hub. I plan to further expand this content.

Let me know what you think! 🙇

kaiyan-sheng · 2022-10-11T19:36:22Z

We also have other integrations with just one data stream. It would be disabled also for one data stream integration. Do you see this as a problem?

My opinion is having false as the default to prevent users from using one azureeventhub is worth the extra step of enabling individual data streams.

zmoog · 2022-10-12T07:09:28Z

My opinion is having false as the default to prevent users from using one azureeventhub is worth the extra step of enabling individual data streams.

Yeah, I think the trade-off is worth too.

Thank you, @kaiyan-sheng, for taking some time to think about this! 🙇

The Azure Logs package has a growing number of data streams. When users install the collective Azure Logs integration, we end up in a situation where Filebeat spawns a lot of "azureeventhub" behind the scenes, all receiving data from the same event hub. Many (data streams) to one (event hub) are not a good default. Each input received a copy of each message published on the event hub. A better default is to have all input disabled. In this case, Filebeat spawns a new input only if the user explicitly enables the data stream.

zmoog added the enhancement New feature or request label Oct 4, 2022

zmoog self-assigned this Oct 4, 2022

zmoog marked this pull request as ready for review October 4, 2022 11:11

zmoog requested a review from a team as a code owner October 4, 2022 11:11

kaiyan-sheng approved these changes Oct 4, 2022

View reviewed changes

zmoog force-pushed the zmoog/azure-logs-disabled-by-default branch from 4c234a7 to a666258 Compare October 4, 2022 15:39

zmoog added 3 commits October 12, 2022 10:40

Disable the new Application Gateway data stream

17e5f8e

Add changelog

d065c0d

zmoog force-pushed the zmoog/azure-logs-disabled-by-default branch from a666258 to d065c0d Compare October 12, 2022 08:43

zmoog merged commit 6a1e520 into elastic:main Oct 13, 2022

zmoog deleted the zmoog/azure-logs-disabled-by-default branch October 13, 2022 10:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Azure Logs] Set enabled: false as default for all data streams #4373

[Azure Logs] Set enabled: false as default for all data streams #4373

zmoog commented Oct 4, 2022 •

edited

Loading

elasticmachine commented Oct 4, 2022 •

edited

Loading

elasticmachine commented Oct 4, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Oct 4, 2022 •

edited

Loading

endorama commented Oct 4, 2022

kaiyan-sheng commented Oct 4, 2022

zmoog commented Oct 4, 2022

zmoog commented Oct 4, 2022

kaiyan-sheng left a comment

zmoog commented Oct 4, 2022 •

edited

Loading

kaiyan-sheng commented Oct 5, 2022

zmoog commented Oct 11, 2022

kaiyan-sheng commented Oct 11, 2022

zmoog commented Oct 12, 2022

[Azure Logs] Set enabled: false as default for all data streams #4373

[Azure Logs] Set enabled: false as default for all data streams #4373

Conversation

zmoog commented Oct 4, 2022 • edited Loading

What does this PR do?

Why?

Checklist

elasticmachine commented Oct 4, 2022 • edited Loading

🚀 Benchmarks report

Package azure 👍(0) 💚(1) 💔(9)

elasticmachine commented Oct 4, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Oct 4, 2022 • edited Loading

🌐 Coverage report

endorama commented Oct 4, 2022

kaiyan-sheng commented Oct 4, 2022

zmoog commented Oct 4, 2022

zmoog commented Oct 4, 2022

kaiyan-sheng left a comment

Choose a reason for hiding this comment

zmoog commented Oct 4, 2022 • edited Loading

kaiyan-sheng commented Oct 5, 2022

zmoog commented Oct 11, 2022

kaiyan-sheng commented Oct 11, 2022

zmoog commented Oct 12, 2022

zmoog commented Oct 4, 2022 •

edited

Loading

elasticmachine commented Oct 4, 2022 •

edited

Loading

Package `azure` 👍(0) 💚(1) 💔(9)

elasticmachine commented Oct 4, 2022 •

edited

Loading

elasticmachine commented Oct 4, 2022 •

edited

Loading

zmoog commented Oct 4, 2022 •

edited

Loading