[Azure] Add Azure Application Gateway datastream #3892

legoguy1000 · 2022-07-28T17:12:16Z

What does this PR do?

Add Azure Application Gateway datastream

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes Azure WAF #1306

Screenshots

elasticmachine · 2022-07-28T17:15:24Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-10-04T10:23:58.856+0000
Duration: 15 min 51 sec

Test stats 🧪

Test	Results
Failed	0
Passed	123
Skipped	0
Total	123

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

zmoog · 2022-08-01T10:41:06Z

Hey @legoguy1000, thank you for your contribution! \o/

I am looking for the most appropriate team/person to review your PR.

zmoog · 2022-08-02T10:49:41Z

We may need to add instructions about which Azure resources users must set up to use the integration.

@legoguy1000, which diagnostic setting did you set up to send logs to the event hub?

Looking at the sample_event.json file content, my educated guess is that it is the diagnostic settings from the Application Gateway:

Am I right?

legoguy1000 · 2022-08-02T11:44:28Z

@zmoog I don't currently have an azure WAF to test this on, I just went off the documentation here, https://docs.microsoft.com/en-us/azure/web-application-firewall/ag/web-application-firewall-logs.

elasticmachine · 2022-08-03T13:26:30Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

zmoog · 2022-08-03T16:27:12Z

I don't currently have an azure WAF to test this on; I just went off the documentation here https://docs.microsoft.com/en-us/azure/web-application-firewall/ag/web-application-firewall-logs.

Got it, thanks! Interesting reading.

I see the log categories used as data sources are the following:

Application Gateway Access Log
Application Gateway Performance Log
Application Gateway Firewall Log

@legoguy1000 I noticed pipeline test documents for "Application Gateway Access Log" and "Application Gateway Firewall Log": are we supporting both, right?

zmoog · 2022-08-03T16:27:36Z

Questions for the security-external-integrations team:

Is there interest in the "Application Gateway Performance Log"?
I see this integration focuses on WAF, but should we also consider an integration about Application Gateway?

legoguy1000 · 2022-08-08T16:31:08Z

I don't currently have an azure WAF to test this on; I just went off the documentation here https://docs.microsoft.com/en-us/azure/web-application-firewall/ag/web-application-firewall-logs.

Got it, thanks! Interesting reading.

I see the log categories used as data sources are the following:

Application Gateway Access Log

Application Gateway Performance Log

Application Gateway Firewall Log

@legoguy1000 I noticed pipeline test documents for "Application Gateway Access Log" and "Application Gateway Firewall Log": are we supporting both, right?

Sorry I didn't see this earlier. I don't see any reason you wouldn't want both. One is just access logs, the other is the actual WAF security rules alert logs.

packages/azure/data_stream/waf_logs/elasticsearch/ingest_pipeline/default.yml

packages/azure/data_stream/waf_logs/fields/ecs.yml

efd6 · 2022-08-17T21:52:28Z

/test

elasticmachine · 2022-08-17T22:08:26Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`10/10`)	💚
Files	86.364% (`19/22`)	👎 -11.193
Classes	86.364% (`19/22`)	👎 -11.193
Methods	83.422% (`156/187`)	👎 -6.334
Lines	85.011% (`2779/3269`)	👎 -6.743
Conditionals	100.0% (`0/0`)	💚

legoguy1000 · 2022-08-23T03:54:29Z

Questions for the security-external-integrations team:

Is there interest in the "Application Gateway Performance Log"?

I see this integration focuses on WAF, but should we also consider an integration about Application Gateway?

The performance log is more of a metric but I guess we could add it. Also unless I'm reading something wrong, the waf is the application gateway or at least that's what it's log names are.

legoguy1000 · 2022-08-25T02:33:55Z

I think this is good for now.

zmoog · 2022-08-25T08:17:07Z

I see this integration is using the Application Gateway resource logs to collect WAF data; in particular, we are leveraging the following log categories:

ApplicationGatewayAccessLog
ApplicationGatewayFirewallLog

I just noticed that Front Door also produces WAF data using its resource logs named FrontdoorAccessLog and FrontdoorWebApplicationFirewallLog.

@efd6, do you think we should also support WAF data from Front Door's logs in this integration?

@efd6, another more general question: what if someone asks us to support Application Gateway logs or Front Door logs without focusing on WAF? For example, they want the access logs with no particular focus on WAF.

zmoog · 2022-08-25T08:49:19Z

Hey @legoguy1000, in a previous comment, you mentioned that you don't azure WAF to test this on. @efd6, did you get a chance to test this integration?

Otherwise, I can give it a try.

zmoog · 2022-08-25T08:53:42Z

@jamiehynds, if I am not wrong, this integration does not have a dashboard. Can we go with it and add it later, or should we add it in this PR? I believe the questions I asked Dan in a previous comment were also for you 😇

zmoog · 2022-08-25T08:57:14Z

One last thing.

@legoguy1000, thank you for your contribution! I'll keep an eye on this PR, and I won't let you wait again for more than necessary.

legoguy1000 · 2022-08-25T12:14:03Z

@zmoog you're all good, we're all busy with a million things. I could always rename the data stream to application gateway and we could create a separate one for front door?

efd6 · 2022-08-25T22:08:47Z

@zmoog No I have not tested this. Please take it for a spin.

jamiehynds · 2022-08-26T08:26:42Z

@jamiehynds, if I am not wrong, this integration does not have a dashboard. Can we go with it and add it later, or should we add it in this PR? I believe the questions I asked Dan in a previous comment were also for you 😇

Hey @zmoog - with regards to Frontdoor logs, we seem to have a community contribution for Frontdoor support (including a dashboard) - #2497. I'm not overly familiar on the differences between WAF and Frontdoor. Should these be separate integrations or should Frontdoor be a datastream under Azure WAF?

With regards to a dashboard, it's always our preference to include dashboards with new integrations as when we don't ship dashboards, it's easy to put it on the long finger and we end up with dashboards without integrations. Do you think we could add a dashboad as part of this PR?

elasticmachine · 2022-09-28T14:18:11Z

🚀 Benchmarks report

Package `azure` 👍(6) 💚(3) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`auditlogs`	1515.15	1088.14	-427.01 (-28.18%)	💔

To see the full report comment with /test benchmark fullreport

legoguy1000 · 2022-09-28T14:46:58Z

@P1llus updated.

P1llus

Added a few small nitpicks, could you also remove any changes done to the other datastreams? As they should be updated in another PR if necessary.

Could you also rename it to application_gateway rather than application_gateway_logs ?

After the changes and a review from obs-cloud-monitoring we can merge.

packages/azure/data_stream/application_gateway_logs/elasticsearch/ingest_pipeline/default.yml

...re/data_stream/activitylogs/_dev/test/pipeline/test-activitylogs-edgecases.log-expected.json

packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml

packages/azure/data_stream/application_gateway_logs/elasticsearch/ingest_pipeline/default.yml

packages/azure/_dev/build/docs/README.md

packages/azure/_dev/build/docs/application_gateway.md

packages/azure/data_stream/application_gateway_logs/agent/stream/azure-eventhub.yml.hbs

packages/azure/_dev/build/docs/README.md

packages/azure/data_stream/application_gateway/manifest.yml

This completes the transition from Azure WAF to Azure Application Gateway.

Like the AWS integration, we keep fields and event reference material in the individual integration docs. The README.md file was getting hard to read due to the extra long reference section.

We are working to define a more defined process to graduate from beta to a GA for each integration or data stream that we release.

zmoog · 2022-10-04T09:19:01Z

Hey @legoguy1000, thank you again for the initial work and the will to update it!

I made all the pending changes from my last comments and LGTM. I'll take care of the CI.

When all is green, we can merge.

Renamed field with the expected name.

legoguy1000 marked this pull request as ready for review July 28, 2022 22:18

legoguy1000 requested a review from a team as a code owner July 28, 2022 22:18

ebeahan requested a review from a team August 1, 2022 21:29

jamiehynds added the Team:Security-External Integrations label Aug 3, 2022

jamiehynds added New Integration Integration:azure Azure Logs labels Aug 3, 2022

efd6 reviewed Aug 17, 2022

View reviewed changes

legoguy1000 force-pushed the 1306-azure-waf branch from c789f17 to 3a96c01 Compare August 17, 2022 21:35

efd6 approved these changes Aug 17, 2022

View reviewed changes

P1llus self-requested a review September 28, 2022 07:52

P1llus self-assigned this Sep 28, 2022

legoguy1000 added 3 commits September 28, 2022 14:03

Add remaining fields

7705c05

add action

90b9985

update per comments

a9b37d3

legoguy1000 force-pushed the 1306-azure-waf branch from 3a96c01 to a9b37d3 Compare September 28, 2022 14:03

Rename WAF to application gateway & upgrade to ECS 8.4.0

9605efe

legoguy1000 changed the title ~~[Azure] Add Azure WAF datastream~~ [Azure] Add Azure Application Gateway datastream Sep 28, 2022

P1llus reviewed Sep 29, 2022

View reviewed changes

zmoog self-requested a review September 29, 2022 09:30

zmoog reviewed Sep 29, 2022

View reviewed changes

packages/azure/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/azure/_dev/build/docs/application_gateway.md Show resolved Hide resolved

packages/azure/data_stream/application_gateway_logs/agent/stream/azure-eventhub.yml.hbs Outdated Show resolved Hide resolved

update per comments

a5b1eb0

legoguy1000 force-pushed the 1306-azure-waf branch from 37149d7 to a5b1eb0 Compare September 29, 2022 13:57

legoguy1000 added 2 commits September 29, 2022 14:07

update docs

b7b8323

remove changes

8ed7b6f

legoguy1000 force-pushed the 1306-azure-waf branch from 9d9081a to 8ed7b6f Compare September 29, 2022 14:09

P1llus and others added 2 commits October 3, 2022 11:31

Merge branch 'main' into 1306-azure-waf

5319276

update readme

705967a

P1llus approved these changes Oct 4, 2022

View reviewed changes

zmoog reviewed Oct 4, 2022

View reviewed changes

packages/azure/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/azure/data_stream/application_gateway/manifest.yml Outdated Show resolved Hide resolved

zmoog added 3 commits October 4, 2022 11:06

Update title and description

17afcda

This completes the transition from Azure WAF to Azure Application Gateway.

Move reference to individual integrations

bf60152

Like the AWS integration, we keep fields and event reference material in the individual integration docs. The README.md file was getting hard to read due to the extra long reference section.

Set Application Gateway datastream as beta release

b107b24

We are working to define a more defined process to graduate from beta to a GA for each integration or data stream that we release.

zmoog added 2 commits October 4, 2022 11:25

Regenerate main README.md file

d24ca24

Fix sample event

ff5870a

Renamed field with the expected name.

zmoog force-pushed the 1306-azure-waf branch from 56984fc to ff5870a Compare October 4, 2022 10:23

zmoog approved these changes Oct 4, 2022

View reviewed changes

P1llus merged commit 6adb710 into elastic:main Oct 4, 2022

legoguy1000 deleted the 1306-azure-waf branch October 4, 2022 12:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Azure] Add Azure Application Gateway datastream #3892

[Azure] Add Azure Application Gateway datastream #3892

legoguy1000 commented Jul 28, 2022 •

edited

Loading

elasticmachine commented Jul 28, 2022 •

edited

Loading

Build stats

Test stats 🧪

zmoog commented Aug 1, 2022

zmoog commented Aug 2, 2022

legoguy1000 commented Aug 2, 2022

elasticmachine commented Aug 3, 2022

zmoog commented Aug 3, 2022

zmoog commented Aug 3, 2022

legoguy1000 commented Aug 8, 2022

efd6 commented Aug 17, 2022

elasticmachine commented Aug 17, 2022 •

edited

Loading

legoguy1000 commented Aug 23, 2022

legoguy1000 commented Aug 25, 2022

zmoog commented Aug 25, 2022 •

edited

Loading

zmoog commented Aug 25, 2022

zmoog commented Aug 25, 2022

zmoog commented Aug 25, 2022

legoguy1000 commented Aug 25, 2022

efd6 commented Aug 25, 2022

jamiehynds commented Aug 26, 2022

elasticmachine commented Sep 28, 2022 •

edited

Loading

legoguy1000 commented Sep 28, 2022

P1llus left a comment

zmoog commented Oct 4, 2022

[Azure] Add Azure Application Gateway datastream #3892

[Azure] Add Azure Application Gateway datastream #3892

Conversation

legoguy1000 commented Jul 28, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jul 28, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

zmoog commented Aug 1, 2022

zmoog commented Aug 2, 2022

legoguy1000 commented Aug 2, 2022

elasticmachine commented Aug 3, 2022

zmoog commented Aug 3, 2022

zmoog commented Aug 3, 2022

legoguy1000 commented Aug 8, 2022

efd6 commented Aug 17, 2022

elasticmachine commented Aug 17, 2022 • edited Loading

🌐 Coverage report

legoguy1000 commented Aug 23, 2022

legoguy1000 commented Aug 25, 2022

zmoog commented Aug 25, 2022 • edited Loading

zmoog commented Aug 25, 2022

zmoog commented Aug 25, 2022

zmoog commented Aug 25, 2022

legoguy1000 commented Aug 25, 2022

efd6 commented Aug 25, 2022

jamiehynds commented Aug 26, 2022

elasticmachine commented Sep 28, 2022 • edited Loading

🚀 Benchmarks report

Package azure 👍(6) 💚(3) 💔(1)

legoguy1000 commented Sep 28, 2022

P1llus left a comment

Choose a reason for hiding this comment

zmoog commented Oct 4, 2022

legoguy1000 commented Jul 28, 2022 •

edited

Loading

elasticmachine commented Jul 28, 2022 •

edited

Loading

elasticmachine commented Aug 17, 2022 •

edited

Loading

zmoog commented Aug 25, 2022 •

edited

Loading

elasticmachine commented Sep 28, 2022 •

edited

Loading

Package `azure` 👍(6) 💚(3) 💔(1)