Skip to content

[gcp] Fix ingest pipeline to parse cluster notifications in audit data stream#17830

Open
brijesh-elastic wants to merge 2 commits intoelastic:mainfrom
brijesh-elastic:gcp-2.47.2
Open

[gcp] Fix ingest pipeline to parse cluster notifications in audit data stream#17830
brijesh-elastic wants to merge 2 commits intoelastic:mainfrom
brijesh-elastic:gcp-2.47.2

Conversation

@brijesh-elastic
Copy link
Collaborator

Proposed commit message

gcp: fix ingest pipeline to parse cluster notifications in audit data stream

This PR introduces support for parsing (GKE cluster notifications)[1]. Additionally, it expands
the `labels.payload` JSON string to ensure it is indexable for improved searchability.

This example is taken from the documentation and has been mocked for testing purposes.

[1] https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-notifications 

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/gcp directory.
  • Run the following command to run tests.

elastic-package test -v

Related issues

@brijesh-elastic brijesh-elastic self-assigned this Mar 16, 2026
@brijesh-elastic brijesh-elastic requested review from a team as code owners March 16, 2026 07:01
@brijesh-elastic brijesh-elastic added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:gcp Google Cloud Platform bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Mar 16, 2026
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@github-actions
Copy link
Contributor

✅ Vale Linting Results

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

@elastic-vault-github-plugin-prod

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link

💚 Build Succeeded

cc @brijesh-elastic

@brijesh-elastic brijesh-elastic requested a review from kcreddy March 16, 2026 07:58
@andrewkroh andrewkroh added Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Mar 16, 2026
Comment on lines +5 to +7
"input": {
"type": "gcp-pubsub"
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

input.type comes from agent. we can omit this inside pipeline tests raw event.

| gcp.audit.logentry_operation.producer | Optional. An arbitrary producer identifier. The combination of id and producer must be globally unique. | keyword |
| gcp.audit.metadata | Service-specific data about the request, response, and other information associated with the current audited event. | flattened |
| gcp.audit.method_name | The name of the service method or operation. For API calls, this should be the name of the API method. For example, 'google.datastore.v1.Datastore.RunQuery'. | keyword |
| gcp.audit.notification | | keyword |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add field description?

Comment on lines +4 to +5
- description: Fix ingest pipeline to parse cluster notifications in audit data stream.
type: bugfix
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be an enhancement. Let me know what you think.

Comment on lines +24 to +31
"cluster_location": "us-central1-a",
"cluster_name": "test-qa",
"payload": {
"currentVersion": "1.33.5-gke.2228001",
"operation": "operation-1234567890",
"resource": "projects/elastic-test/locations/us-central1-a/clusters/test-qa/nodePools/test-custom",
"resourceType": "NODE_POOL",
"targetVersion": "1.33.5-gke.2326000"
Copy link
Contributor

@kcreddy kcreddy Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add orchestrator fields?

"resourceType": "NODE_POOL",
"targetVersion": "1.33.5-gke.2326000"
},
"project_id": "elastic-test",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:gcp Google Cloud Platform Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[gcp.audit] Ingest pipeline fails to parse plain-text upgrade events

4 participants