[Azure] Add Monitor App Service Logs data stream #4818

lucian-ioan · 2022-12-13T13:44:20Z

What does this PR do?

Adds App Service data stream with support for the following log categories:

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Document providing instruction on how to collect each log category:
https://docs.google.com/document/d/1AJREQvAE2mc1QU19YKgZ4QJB6OHsyw8fy_WlLRYJV74

Research how to collect HTTP logs
Research how to collect App Service Console Logs
Research how to collect App Service Application Logs
Research how to collect Access Audit Logs
Research how to collect IPSecurity Audit logs
Research how to collect App Service Platform logs

How to test this PR locally

Related issues

-Relates elastic/obs-infraobs-team#826

Screenshots

elasticmachine · 2022-12-13T14:12:17Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-20T21:27:37.747+0000
Duration: 14 min 29 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8
Skipped	0
Total	8

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-12-21T17:19:58Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	87.5% (`7/8`)	👎 -12.5
Classes	87.5% (`7/8`)	👎 -12.5
Methods	75.0% (`18/24`)	👎 -25.0
Lines	77.808% (`284/365`)	👎 -9.023
Conditionals	100.0% (`0/0`)	💚

zmoog

This first iteration on AppServiceAuditLogs and AppServiceHTTPLogs looks promising.

I added a few comments I'd love to hear you about.

zmoog · 2022-12-29T19:36:35Z

packages/azure/data_stream/appservices/agent/stream/azure-eventhub.yml.hbs

+storage_account_container: {{storage_account_container}}
+{{else}}
+{{#if eventhub}}
+storage_account_container: filebeat-firewalllogs-{{eventhub}}


I believe this firewalllogs is a leftover, isn't it?

zmoog · 2022-12-29T19:43:26Z

...ata_stream/appservices/elasticsearch/ingest_pipeline/appservice-auditlogs-inner-pipeline.yml

+processors:
+  - rename:
+      field: azure.appservices.ResourceId
+      target_field: azure.resource.id
+      ignore_missing: true
+  - rename:
+      field: azure.appservices.Category
+      target_field: azure.appservices.category
+  - rename:
+      field: azure.appservices.OperationName
+      target_field: azure.appservices.operation_name
+  - rename:
+      field: azure.appservices.Properties.Protocol
+      target_field: azure.appservices.properties.protocol
+  - rename:
+      field: azure.appservices.Properties.User
+      target_field: azure.appservices.properties.user
+  - rename:
+      field: azure.appservices.Properties.UserAddress
+      target_field: azure.appservices.properties.user_address
+  - rename:
+      field: azure.appservices.Properties.UserDisplayName
+      target_field: azure.appservices.properties.user_display_name
+  - convert:
+      field: azure.appservices.properties.user_address
+      type: ip
+      ignore_missing: true
+  - remove:
+      field:
+        - azure.appservices.Properties
+      ignore_missing: true
+on_failure:


By removing the azure.appservices.Properties field, we risk dropping sub-fields that we are unaware of or Azure may add in the future.

What if we rename Properties into properties first and then rename the other fields? We won't lose unexpected field this way.

zmoog · 2022-12-29T19:54:58Z

packages/azure/data_stream/appservices/fields/fields.yml

+- name: azure.appservices
+  type: group
+  fields:
+    - name: category
+      type: keyword
+      description: The category of the operation.
+    - name: operation_name
+      type: keyword
+      description: The operation name.
+    - name: properties
+      type: group
+      fields:
+        - name: protocol
+          type: keyword
+          description: Authentication protocol.
+        - name: user
+          type: keyword
+          description: Username used for publishing access.
+        - name: user_address
+          type: ip
+          description: Client IP address of the publishing user.
+        - name: user_display_name
+          type: keyword
+          description: Email address of a user in case publishing was authorized via AAD authentication.
+        - name: client_ip
+          type: ip
+          description: IP address of the client.
+        - name: computer_name
+          type: keyword
+          description: The name of the server on which the log file entry was generated.
+        - name: cookie
+          type: keyword
+          description: Cookie on HTTP request.
+        - name: cs_bytes
+          type: long
+          description: Number of bytes received by server.
+        - name: cs_host
+          type: keyword
+          description: Host name header on HTTP request.
+        - name: cs_method
+          type: keyword
+          description: The request HTTP verb.
+        - name: cs_uri_query
+          type: keyword
+          description: URI query on HTTP request.
+        - name: cs_uri_stem
+          type: keyword
+          description: The target of the request.
+        - name: cs_username
+          type: keyword
+          description: The name of the authenticated user on HTTP request.
+        - name: referer
+          type: keyword
+          description: The site that the user last visited. This site provided a link to the current site.
+        - name: result
+          type: keyword
+          description: Success / Failure of HTTP request.
+        - name: s_port
+          type: keyword
+          description: Server port number.
+        - name: sc_bytes
+          type: long
+          description: Number of bytes sent by server.
+        - name: sc_status
+          type: long
+          description: HTTP status code.
+        - name: sc_substatus
+          type: keyword
+          description: Substatus error code on HTTP request.
+        - name: sc_win32status
+          type: keyword
+          description: Windows status code on HTTP request.
+        - name: time_taken
+          type: long
+          description: Time taken by HTTP request in milliseconds.
+        - name: user_agent
+          type: keyword
+          description: User agent on HTTP request.


What about grouping the fields in this list by log categories?

Adding comments to visually separate the shared ones from those belonging to a specific log category can be helpful.

zmoog · 2022-12-29T20:00:10Z

What log categories are you working on right now?

zmoog · 2022-12-29T20:07:10Z

Let me know which log categories you are working on, and I'll try to pick at least one or two of the other ones to find the sample logs.

lucian-ioan · 2023-01-19T12:28:30Z

Currently, I have samples for the 4 log categories marked and will update this PR to contain the yaml for the other 2.

zmoog · 2023-01-24T12:20:21Z

Hey @lucianpy, how is the log hunting going? Please let me know if you need help with the log collection, testing, or field/pipeline building!

lucian-ioan · 2023-01-26T10:05:25Z

Hey @zmoog, no new progress as of yet. Worst case I will just push the 4 categories that I do have. Still want to try to get them this week. If not succesful, I will ask for your help on Monday.

zmoog · 2023-02-01T08:39:23Z

Hey @zmoog, no new progress as of yet. Worst case I will just push the 4 categories that I do have. Still want to try to get them this week. If not succesful, I will ask for your help on Monday.

Hey @lucianpy, can I help with something?

lucian-ioan · 2023-02-02T09:38:15Z

Hey @zmoog, could you take a look at IPSecurity Audit Logs ?

It probably needs some IP rules set up and then just an access to the application by a restricted IP for example might trigger it.
You can also review the HTTP logs.

lucian-ioan · 2023-02-16T10:50:44Z

As an update here, I now have all of the 6 log categories that we need.

Unfortunately, App Service Platform Logs and App Service Console Logs can't be processed directly due to double newlines apearing in some fields similar to: elastic/beats#34092

Currently looking at a fix for this issue.

lucian-ioan · 2023-02-23T12:19:33Z

One thing that needs to be decided is what to do in case of same name fields for different log categories.
For example, the CIp field for HTTPLogs is just a usual IP address whereas the CIp for IPSecAuditLogs is a socket (ip:port).

A solution that I found is to modify the fields for each log category, such as azure.appservices.client_id becomes azure.appservices.httplogs.client_id. The disadvantage to this is that the json that ends up in Kibana will be more verbose. What do you think @zmoog ?

Also, updated the PR with the document providing instructions on how to collect all the log categories!

zmoog · 2023-03-02T10:50:18Z

Quick update after the sync over zoom in the last few days to discuss the following two topics.

One thing that needs to be decided is what to do in case of same name fields for different log categories.

We decided to try to map the source document into a destination field as an object.

For example, if the source has an IP address only, we can use a destination field like this:

{
  "ip": {
    "address": "10.1.0.1"
  }
}

Instead, if it has an IP address and a port, we can use:

{
  "ip": {
    "address": "10.1.0.1",
    "port": 8080
  }
}

This is only an example, not the recommended names 😄 , I suggest exploring what other integration are doing and the reference names and structure in the Elastic Common Schema (ECS).

Unfortunately, App Service Platform Logs and App Service Console Logs can't be processed directly due to double newlines appearing in some fields similar to: elastic/beats#34092

We need to reproduce this issue to share our findings with the Azure folks to understand this issue better.

Suppose this turns out to be something we can't avoid. In that case, one possible approach is the update the azure-eventhub input at https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/azureeventhub/input.go#L167-L203, and the sanitize the []byte we receive from the Event Hub before trying to unmarshal it to unpack the elements in the records list.

zmoog · 2023-03-02T10:52:40Z

@lucianpy how is it going with the newlines issue? Are you and Gabriel able to reproduce the issue again?

lucian-ioan · 2023-03-02T12:04:56Z

@zmoog we were not able to reproduce the issue so I will add the sanitization just in case it ever happens again.

lucian-ioan · 2023-03-14T09:18:41Z

To be noted here that there's a pretty big discrepancy between the fields I was able to fetch for Console Logs, Platform Logs and App Logs and the azure official documentation.

Initially I thought there might be some optional fields that only come up in specific instances but some fields are not in the documentation at all, like EventStampType, EventStampName etc.

Wasn't able to fetch any different variation of logs than those I have here. What do you think about this issue @zmoog?

lucian-ioan · 2023-03-14T14:34:16Z

Suppose this turns out to be something we can't avoid. In that case, one possible approach is the update the azure-eventhub input at https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/azureeventhub/input.go#L167-L203, and the sanitize the []byte we receive from the Event Hub before trying to unmarshal it to unpack the elements in the records list.

I found I can only sanitize the event for appservices in integrations, since this is where the issue was found. The advantage of this is not modifying the azure-eventhub code directly in beats. What do you think @zmoog?

zmoog · 2023-03-14T16:29:35Z

To be noted here that there's a pretty big discrepancy between the fields I was able to fetch for Console Logs, Platform Logs and App Logs and the azure official documentation.

Initially I thought there might be some optional fields that only come up in specific instances but some fields are not in the documentation at all, like EventStampType, EventStampName etc.

Wasn't able to fetch any different variation of logs than those I have here. What do you think about this issue @zmoog?

I had similar experiences in other log categories. The docs do not seem 100% accurate, and on average there are ~5% of the fields that are different (present, not present, or with a slightly different name).

I usually link to the docs, but take the log samples as a de-factor spec.

zmoog · 2023-03-14T19:28:06Z

I found I can only sanitize the event for appservices, since this is where the issue was found. The advantage of this is not modifying the azure-eventhub code directly in beats. What do you think @zmoog?

There is one main issue in trying to "patch" the invalid JSON string in the pipeline vs. the input.

The diagnostic setting sends data to the event hub in batches, so the input receives something like this:

{
  "records": [
    {
      "time": "2023-03-08T10:33:13.8810000Z",
      "resourceId": "/SUBSCRIPTIONS/6b173463-3d87-43c5-98bb-90af439f1be0/RESOURCEGROUPS/WHATEVER/PROVIDERS/MICROSOFT.INSIGHTS/COMPONENTS/WHATEVER-2",
      "ResourceGUID": "0e1fdb93-b4b6-468b-9408-70ee34e56b22",
      "Type": "AppRequests",
      "AppRoleInstance": "Maurizios-MacBook-Pro.local",
      "AppRoleName": "Web",
      "ClientIP": "0.0.0.0",
      "ClientOS": "Darwin 22.3.0",
      "ClientType": "PC",
      "IKey": "89c2be53-f9c7-4ac0-8191-fb31898b774c",
      "_BilledSize": 976,
      "OperationName": "GET /",
      "OperationId": "ba20ab37cafe455c96db689d52c6b4a9",
      "ParentId": "ba20ab37cafe455c96db689d52c6b4a9",
      "SDKVersion": "node:2.3.5",
      "Properties": {
        "_MS.ProcessedByMetricExtractors": "(Name:'Requests', Ver:'1.1')"
      },
      "Id": "|ba20ab37cafe455c96db689d52c6b4a9.",
      "Name": "GET /",
      "Url": "http://localhost:3000/",
      "Success": true,
      "ResultCode": "200",
      "DurationMs": 1,
      "PerformanceBucket": "<250ms",
      "ItemCount": 1
    },
    {
      "time": "2023-03-08T10:33:13.8810000Z",
      "resourceId": "/SUBSCRIPTIONS/6b173463-3d87-43c5-98bb-90af439f1be0/RESOURCEGROUPS/WHATEVER/PROVIDERS/MICROSOFT.INSIGHTS/COMPONENTS/WHATEVER-2",
      "ResourceGUID": "0e1fdb93-b4b6-468b-9408-70ee34e56b22",
      "Type": "AppRequests",
      "AppRoleInstance": "Maurizios-MacBook-Pro.local",
      "AppRoleName": "Web",
      "ClientIP": "0.0.0.0",
      "ClientOS": "Darwin 22.3.0",
      "ClientType": "PC",
      "IKey": "89c2be53-f9c7-4ac0-8191-fb31898b774c",
      "_BilledSize": 976,
      "OperationName": "GET /",
      "OperationId": "ba20ab37cafe455c96db689d52c6b4a9",
      "ParentId": "ba20ab37cafe455c96db689d52c6b4a9",
      "SDKVersion": "node:2.3.5",
      "Properties": {
        "_MS.ProcessedByMetricExtractors": "(Name:'Requests', Ver:'1.1')"
      },
      "Id": "|ba20ab37cafe455c96db689d52c6b4a9.",
      "Name": "GET /",
      "Url": "http://localhost:3000/",
      "Success": true,
      "ResultCode": "200",
      "DurationMs": 1,
      "PerformanceBucket": "<250ms",
      "ItemCount": 1
    }
  ]
}

In this case, we have two records.

When the input fails to parse the JSON, it packs the whole { "records": [...] } batch of two records into the message field and sends it to the data stream as one document, instead of creating one two documents (one for each record).

In a case like this, the pipeline receives one document with two records, and I am not sure if it is possible to create two documents out of it. My understaning is that the pipeline in:out ratio is 1:1.

In this example, we have two records, but it is common to have ten or more records in a batch.

lucian-ioan · 2023-03-15T12:16:54Z

@zmoog, took a second look at your example and you are right, there are issues with doing it directly from integrations.

Should I open a different PR in beats for this?

zmoog · 2023-03-15T16:18:14Z

@zmoog, took a second look at your example and you are right, there are issues with doing it directly from integrations.

Should I open a different PR in beats for this?

I believe we need to tackle this at the input level with a PR on Beats, but it’s better to double-check; let's ask other team members and hear their advice.

(We're asking offline, we'll update this conversation afterwards)

lucian-ioan · 2023-03-28T09:55:34Z

As an update here, the consensus of the team is to takle this directly in Beats.

Currently waiting on the sanitization PR there to merge before moving forward.

botelastic · 2023-04-27T09:58:35Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2023-06-01T10:27:45Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2023-07-01T10:53:28Z

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

zmoog

🚢

zmoog · 2023-01-16T07:54:34Z

packages/azure/data_stream/appservices/agent/stream/azure-eventhub.yml.hbs

-{{#if eventhub}}
-storage_account_container: filebeat-firewalllogs-{{eventhub}}
-{{/if}}


Why did you remove it? We need a default value, don't you think?

zmoog

During a chat with @lucian-ioan, @gpop63, and @devamanv, we agreed that the best way forward for this App Service integration is the following:

Move the App Services integration from Azure Logs into an independent integration package.
Release App Services as:
- maturity level: experimental
- version: v0.0.1
- no dashboards

This approach allow us to:

Add a metrics data stream in the App Services package so that we can have everything about App Services in a single package.
Have the option to add a dedicated data stream for each log category instead of using a single data stream as of today.
Collect user feedback to adjust field mapping, ingest pipelines, and dashboards.

@lalit-satapathy @rameshelastic @SubhrataK let us know if you think we should take a different route. We plan to move forward with this in the current sprint.

elasticmachine · 2023-07-24T07:24:39Z

Package azure_app_service - 0.0.1 containing this change is available at https://epr.elastic.co/search?package=azure_app_service

* standalone app service integration * default value for storage_account_container * add code owner

hooooh2

lucian-ioan force-pushed the add_azure_http-logs branch from 6e193f5 to cb8230e Compare December 13, 2022 14:01

lucian-ioan force-pushed the add_azure_http-logs branch from cb8230e to 66b0442 Compare December 21, 2022 17:05

lucian-ioan changed the title ~~[Azure] Add Monitor App Service HTTP Logs data stream~~ [Azure] Add Monitor App Service Logs data stream Dec 21, 2022

lucian-ioan requested a review from zmoog December 21, 2022 17:19

zmoog reviewed Dec 29, 2022

View reviewed changes

zmoog self-assigned this Dec 29, 2022

zmoog added enhancement New feature or request Team:Cloud-Monitoring Label for the Cloud Monitoring team labels Dec 29, 2022

zmoog assigned lucian-ioan and unassigned zmoog Dec 29, 2022

botelastic bot added the Stalled label Apr 27, 2023

lucian-ioan assigned zmoog May 2, 2023

botelastic bot removed the Stalled label May 2, 2023

botelastic bot added the Stalled label Jun 1, 2023

botelastic bot closed this Jul 1, 2023

lucian-ioan reopened this Jul 4, 2023

botelastic bot removed the Stalled label Jul 4, 2023

lucian-ioan marked this pull request as ready for review July 11, 2023 05:16

lucian-ioan requested a review from a team as a code owner July 11, 2023 05:16

zmoog approved these changes Jul 19, 2023

View reviewed changes

zmoog reviewed Jul 19, 2023

View reviewed changes

lucian-ioan added 2 commits July 20, 2023 23:17

standalone app service integration

b3ed938

default value for storage_account_container

df9ea0c

lucian-ioan force-pushed the add_azure_http-logs branch from 8a82037 to df9ea0c Compare July 20, 2023 20:27

lucian-ioan added 2 commits July 21, 2023 00:24

Merge remote-tracking branch 'upstream/main' into add_azure_http-logs

3209ad8

add code owner

b0c3324

lucian-ioan force-pushed the add_azure_http-logs branch from 267a8ee to b0c3324 Compare July 20, 2023 21:27

lucian-ioan merged commit 32b759e into elastic:main Jul 21, 2023
1 check passed

gizas pushed a commit that referenced this pull request Sep 5, 2023

[Azure] Add Monitor App Service Logs data stream (#4818)

a486c6f

* standalone app service integration * default value for storage_account_container * add code owner

lucian-ioan deleted the add_azure_http-logs branch September 6, 2023 11:36

hooooh2 reviewed Dec 24, 2023

View reviewed changes

andrewkroh added Integration:azure_app_service Azure App Service New Integration labels Aug 13, 2024

[Azure] Add Monitor App Service Logs data stream #4818

[Azure] Add Monitor App Service Logs data stream #4818

Conversation

lucian-ioan commented Dec 13, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Dec 13, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Dec 21, 2022 • edited Loading

🌐 Coverage report

zmoog left a comment

Choose a reason for hiding this comment

zmoog Dec 29, 2022

Choose a reason for hiding this comment

zmoog Dec 29, 2022

Choose a reason for hiding this comment

zmoog Dec 29, 2022

Choose a reason for hiding this comment

zmoog commented Dec 29, 2022

zmoog commented Dec 29, 2022

lucian-ioan commented Jan 19, 2023

zmoog commented Jan 24, 2023

lucian-ioan commented Jan 26, 2023

zmoog commented Feb 1, 2023

lucian-ioan commented Feb 2, 2023 • edited Loading

lucian-ioan commented Feb 16, 2023 • edited Loading

lucian-ioan commented Feb 23, 2023 • edited Loading

zmoog commented Mar 2, 2023

zmoog commented Mar 2, 2023

lucian-ioan commented Mar 2, 2023

lucian-ioan commented Mar 14, 2023 • edited Loading

lucian-ioan commented Mar 14, 2023 • edited Loading

zmoog commented Mar 14, 2023

zmoog commented Mar 14, 2023

lucian-ioan commented Mar 15, 2023 • edited Loading

zmoog commented Mar 15, 2023 • edited Loading

lucian-ioan commented Mar 28, 2023

botelastic bot commented Apr 27, 2023

botelastic bot commented Jun 1, 2023

botelastic bot commented Jul 1, 2023

zmoog left a comment

Choose a reason for hiding this comment

zmoog Jan 16, 2023

Choose a reason for hiding this comment

zmoog left a comment • edited by devamanv Loading

Choose a reason for hiding this comment

elasticmachine commented Jul 24, 2023

hooooh2 left a comment

Choose a reason for hiding this comment

lucian-ioan commented Dec 13, 2022 •

edited

Loading

elasticmachine commented Dec 13, 2022 •

edited

Loading

elasticmachine commented Dec 21, 2022 •

edited

Loading

lucian-ioan commented Feb 2, 2023 •

edited

Loading

lucian-ioan commented Feb 16, 2023 •

edited

Loading

lucian-ioan commented Feb 23, 2023 •

edited

Loading

lucian-ioan commented Mar 14, 2023 •

edited

Loading

lucian-ioan commented Mar 14, 2023 •

edited

Loading

lucian-ioan commented Mar 15, 2023 •

edited

Loading

zmoog commented Mar 15, 2023 •

edited

Loading

zmoog left a comment •

edited by devamanv

Loading