Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for sprincloud logs inside the platformcloudlogs pipeline #1462

Merged
merged 9 commits into from
Aug 11, 2021

Conversation

narph
Copy link
Contributor

@narph narph commented Aug 9, 2021

What does this PR do?

Add support for sprincloud logs inside the platformcloudlogs pipeline if the azure_log_forwarder.resource_type value is Microsoft.AppPlatform/Spring then logs should be transferred to springcloudlogs

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

Ex:


POST _bulk 
{"create":{"_index":"logs-azure.platformlogs-default"}}
{"@timestamp":"2020-10-28T11:59:26.373Z","agent":{"name":"MicrosoftAgent","version":"1.0.0"},"data_stream":{"namespace":"default","type":"logs","dataset":"azure.platformlogs"},"input":{"type":"azure-log-forwarder"}, "azure_log_forwarder" : {"service_provider" : "Microsoft.ContainerService","category" : "kube-apiserver","resource_type": "Microsoft.AppPlatform/Spring"},"message":"""{ "time": "2021-07-01T19:30:30.535404056Z", "LogFormat": "RAW", "resourceId": "/SUBSCRIPTIONS/EDD63B67-0BA2-4837-A4EB-CD484E9FF623/RESOURCEGROUPS/SA-HEMANT/PROVIDERS/MICROSOFT.APPPLATFORM/SPRING/HM-SC-PETCLINIC", "operationName": "Microsoft.AppPlatform/Spring/logs", "category": "ApplicationConsole", "level": "Informational", "location": "westus2", "properties": {"Log":"2021-07-01 19:30:30.535 INFO 1 --- [oundedElastic-9] c.c.c.ConfigServicePropertySourceLocator : Located environment: name=admin-server, profiles=[mysql], label=null, version=638a1af7fc8d331d7eb26a571275e954632717e8, state=null\n","Stream":"stdout","AppName":"admin-server","InstanceName":"admin-server-default-12-8459d44f68-g4b5f","ServiceId":"c41fd000b1a5450eb234039376da26de","ServiceName":"hm-sc-petclinic"}}""","tags":["forwarded"],"service":{"type":"azure"},"event":{"module":"azure","dataset":"azure.platformlogs"}}

should return:

 {
        "_index" : ".ds-logs-azure.springcloudlogs-default-2021.08.10-000001",
        "_type" : "_doc",
        "_id" : "VQ3zL3sBVkyHZY6EtYa6",
        "_score" : 1.0,
        "_source" : {
          "azure_log_forwarder" : {
            "resource_type" : "Microsoft.AppPlatform/Spring",
            "service_provider" : "Microsoft.ContainerService",
            "category" : "kube-apiserver"
          },
          "agent" : {
            "name" : "MicrosoftAgent",
            "version" : "1.0.0"
          },
          "log" : {
            "level" : "Informational"
          },
          "tags" : [
            "forwarded"
          ],
          "geo" : {
            "name" : "westus2"
          },
          "cloud" : {
            "provider" : "azure"
          },
          "input" : {
            "type" : "azure-log-forwarder"
          },
          "@timestamp" : "2021-07-01T19:30:30.535Z",
          "ecs" : {
            "version" : "1.11.0"
          },
          "data_stream" : {
            "namespace" : "default",
            "type" : "logs",
            "dataset" : "azure.springcloudlogs"
          },
          "service" : {
            "type" : "azure"
          },
          "event" : {
            "agent_id_status" : "missing",
            "ingested" : "2021-08-10T12:05:05.886250800Z",
            "kind" : "event",
            "module" : "azure",
            "action" : "Microsoft.AppPlatform/Spring/logs",
            "dataset" : "azure.springcloudlogs"
          },
          "azure" : {
            "subscription_id" : "EDD63B67-0BA2-4837-A4EB-CD484E9FF623",
            "springcloudlogs" : {
              "operation_name" : "Microsoft.AppPlatform/Spring/logs",
              "log_format" : "RAW",
              "category" : "ApplicationConsole",
              "event_category" : "Administrative",
              "properties" : {
                "app_name" : "admin-server",
                "instance_name" : "admin-server-default-12-8459d44f68-g4b5f",
                "log" : """2021-07-01 19:30:30.535 INFO 1 --- [oundedElastic-9] c.c.c.ConfigServicePropertySourceLocator : Located environment: name=admin-server, profiles=[mysql], label=null, version=638a1af7fc8d331d7eb26a571275e954632717e8, state=null
""",
                "stream" : "stdout",
                "service_name" : "hm-sc-petclinic",
                "service_id" : "c41fd000b1a5450eb234039376da26de"
              }
            },
            "resource" : {
              "provider" : "MICROSOFT.APPPLATFORM/SPRING",
              "name" : "HM-SC-PETCLINIC",
              "id" : "/SUBSCRIPTIONS/EDD63B67-0BA2-4837-A4EB-CD484E9FF623/RESOURCEGROUPS/SA-HEMANT/PROVIDERS/MICROSOFT.APPPLATFORM/SPRING/HM-SC-PETCLINIC",
              "group" : "SA-HEMANT"
            }
          }
        }
      }

changing the value for azure_log_forwarder.resource_type to something else or removing it, will create the usual platformlog

@elasticmachine
Copy link

elasticmachine commented Aug 9, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-08-10T14:50:21.524+0000

  • Duration: 13 min 58 sec

  • Commit: 0bf6337

Test stats 🧪

Test Results
Failed 0
Passed 72
Skipped 0
Total 72

Trends 🧪

Image of Build Times

Image of Tests

@narph narph self-assigned this Aug 10, 2021
@narph narph added Integration:azure Azure Logs Team:Platforms Label for the Integrations - Platforms team labels Aug 10, 2021
@elasticmachine
Copy link

Pinging @elastic/integrations (Team:Platforms)

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added pipeline LGTM. Could you please open an issue in elastic-package describing what would be needed to add pipeline tests for this? Thanks!

@narph narph merged commit d66dc88 into elastic:master Aug 11, 2021
@narph narph deleted the adapt-springlogs branch August 11, 2021 07:45
eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022
…elastic#1462)

* add support for sprincloud logs

* pipeline

* formatting

* fix

* update pipeline

* add logs

* test

* test

* remove tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Integration:azure Azure Logs Team:Platforms Label for the Integrations - Platforms team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants