Add null checks and ignore_missing checks to the rename processor #7845

ishleenk17 · 2023-09-18T05:50:21Z

Bug

What does this PR do?

This PR adds the if check for event.original and the ignore_missing check for the renaming of message to event.original

This PR does changes for below Packages:

apache
apache_tomcat
cassandra
ceph
citrix_adc
golang
hadoop
haproxy
iis
kafka
microsoft_sqlserver
nagios_xi
nginx
oracle
oracle_weblogic
php_fpm
postgresql
rabbitmq
redis
traefik

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Relates Add checks to rename processor for the message field for all Observability Integrations #7822

elasticmachine · 2023-09-18T06:07:20Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-09-21T12:01:57.740+0000
Duration: 39 min 57 sec

Test stats 🧪

Test	Results
Failed	0
Passed	547
Skipped	0
Total	547

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-09-18T06:07:33Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`53/53`)	💚
Files	93.939% (`62/66`)	👎 -6.061
Classes	93.939% (`62/66`)	👎 -6.061
Methods	91.304% (`609/667`)	👎 -3.757
Lines	92.538% (`7974/8617`)	👍 12.692
Conditionals	100.0% (`0/0`)	💚

muthu-mps · 2023-09-21T04:01:16Z

@ishleenk17,
As this PR closes the #7822 , Are we consider addressing all the cloud native and cloud monitoring integration changes as well with this PR ?

Observations

As the event.original is renamed and the grok processor is applied without validating the preserve_original_event flag. Will this be an extra overhead when we remove the event.original field at the end after performing rename and applying grok?

Remove processor Sample

- remove:
      field: event.original
      if: "ctx.tags == null || !(ctx.tags.contains('preserve_original_event'))"
      ignore_failure: true
      ignore_missing: true

shmsr · 2023-09-21T06:00:58Z

Can we also note down the names of the packages touched in this PR (in the PR description itself) so that it is trackable?

Also, in the PR description:

Closes #7822

^ This issue will be closed once this PR gets merged and not all packages are taken care of, right? So can we do s/Closes/Relates

shmsr · 2023-09-21T07:23:13Z

Looks good to me.

shmsr

Left a comment in PR comments.

ishleenk17 · 2023-09-21T07:56:40Z

Can we also note down the names of the packages touched in this PR (in the PR description itself) so that it is trackable?

Also, in the PR description:

Closes #7822

^ This issue will be closed once this PR gets merged and not all packages are taken care of, right? So can we do s/Closes/Relates

Makes sense. Updated!

ishleenk17 · 2023-09-21T08:24:20Z

As the event.original is renamed and the grok processor is applied without validating the preserve_original_event flag. Will this be an extra overhead when we remove the event.original field at the end after performing rename and applying grok?

@muthu-mps , the renaming of message to event.original was already happening. We are mainly adding if checks.
Also since we are having check for preserve original in the end after all the groking has been done, so it should be ok.
For tracking preserve original issue I have raised this github ticket

muthu-mps · 2023-09-21T08:46:35Z

@ishleenk17 - As part of the new validation rules here we should also remove the message field. Can we consider removing message field as a part of this change ?

ishleenk17 · 2023-09-21T09:09:08Z

@ishleenk17 - As part of the new validation rules here we should also remove the message field. Can we consider removing message field as a part of this change ?

Muthu, removing of fields that have been renamed should be tackled at a generic level, not specific to event.original. Some examples, in haproxy, iis for eg. there exists many renamed fields which are not removed. So, I don't think we should keep this removal as a validation check in. I will comment same in this ticket as well.

muthu-mps

LGTM!

elasticmachine · 2023-09-21T13:05:33Z

Package apache - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=apache

elasticmachine · 2023-09-21T13:10:25Z

Package apache_tomcat - 0.12.3 containing this change is available at https://epr.elastic.co/search?package=apache_tomcat

elasticmachine · 2023-09-21T13:15:23Z

Package cassandra - 1.9.1 containing this change is available at https://epr.elastic.co/search?package=cassandra

elasticmachine · 2023-09-21T13:20:57Z

Package ceph - 1.0.1 containing this change is available at https://epr.elastic.co/search?package=ceph

elasticmachine · 2023-09-21T13:26:16Z

Package citrix_adc - 1.0.1 containing this change is available at https://epr.elastic.co/search?package=citrix_adc

elasticmachine · 2023-09-21T13:31:03Z

Package golang - 0.4.3 containing this change is available at https://epr.elastic.co/search?package=golang

elasticmachine · 2023-09-21T13:36:10Z

Package hadoop - 1.1.8 containing this change is available at https://epr.elastic.co/search?package=hadoop

elasticmachine · 2023-09-21T13:41:21Z

Package haproxy - 1.8.3 containing this change is available at https://epr.elastic.co/search?package=haproxy

elasticmachine · 2023-09-21T13:46:18Z

Package iis - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=iis

elasticmachine · 2023-09-21T13:55:51Z

Package kafka - 1.9.2 containing this change is available at https://epr.elastic.co/search?package=kafka

elasticmachine · 2023-09-21T14:00:37Z

Package microsoft_sqlserver - 2.2.1 containing this change is available at https://epr.elastic.co/search?package=microsoft_sqlserver

elasticmachine · 2023-09-21T14:05:44Z

Package nagios_xi - 1.1.1 containing this change is available at https://epr.elastic.co/search?package=nagios_xi

elasticmachine · 2023-09-21T14:10:55Z

Package nginx - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=nginx

elasticmachine · 2023-09-21T14:15:51Z

Package oracle - 1.20.2 containing this change is available at https://epr.elastic.co/search?package=oracle

elasticmachine · 2023-09-21T14:20:37Z

Package oracle_weblogic - 1.0.1 containing this change is available at https://epr.elastic.co/search?package=oracle_weblogic

elasticmachine · 2023-09-21T14:25:49Z

Package php_fpm - 0.6.2 containing this change is available at https://epr.elastic.co/search?package=php_fpm

elasticmachine · 2023-09-21T14:30:38Z

Package postgresql - 1.16.1 containing this change is available at https://epr.elastic.co/search?package=postgresql

elasticmachine · 2023-09-21T14:35:26Z

Package rabbitmq - 1.10.1 containing this change is available at https://epr.elastic.co/search?package=rabbitmq

elasticmachine · 2023-09-21T14:50:31Z

Package redis - 1.11.1 containing this change is available at https://epr.elastic.co/search?package=redis

elasticmachine · 2023-09-21T15:27:28Z

Package traefik - 1.8.1 containing this change is available at https://epr.elastic.co/search?package=traefik

Initial commit for event.original if check

bfef982

Bulk checkin for if and ignore_missing checks

fba1cf6

ishleenk17 self-assigned this Sep 18, 2023

ishleenk17 marked this pull request as ready for review September 18, 2023 09:37

ishleenk17 requested a review from a team as a code owner September 18, 2023 09:37

Merge branch 'main' into event_original_ifcheck

3852ef6

ishleenk17 changed the title ~~Initial commit for event.original if check~~ Add null checks and ignore_missing checks to the rename processor Sep 18, 2023

shmsr reviewed Sep 21, 2023

View reviewed changes

shmsr approved these changes Sep 21, 2023

View reviewed changes

Merge branch 'main' into event_original_ifcheck

af15be1

update manifest file for apache

9e3e714

muthu-mps self-requested a review September 21, 2023 09:27

muthu-mps approved these changes Sep 21, 2023

View reviewed changes

ishleenk17 and others added 4 commits September 21, 2023 15:49

Merge branch 'main' into event_original_ifcheck

4727139

Update manifest for mssql

89f4293

Merge branch 'main' into event_original_ifcheck

8022d57

update php_fpm manifest

a5a0979

ishleenk17 merged commit 9d22b59 into elastic:main Sep 21, 2023
4 checks passed

ishleenk17 mentioned this pull request Sep 22, 2023

Add checks to rename processor for the message field for all Observability Integrations #7822

Closed

35 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add null checks and ignore_missing checks to the rename processor #7845

Add null checks and ignore_missing checks to the rename processor #7845

ishleenk17 commented Sep 18, 2023 •

edited

elasticmachine commented Sep 18, 2023 •

edited

Build stats

Test stats 🧪

elasticmachine commented Sep 18, 2023 •

edited

muthu-mps commented Sep 21, 2023

shmsr commented Sep 21, 2023 •

edited

shmsr commented Sep 21, 2023

shmsr left a comment

ishleenk17 commented Sep 21, 2023 •

edited

ishleenk17 commented Sep 21, 2023

muthu-mps commented Sep 21, 2023 •

edited

ishleenk17 commented Sep 21, 2023

muthu-mps left a comment

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

Add null checks and ignore_missing checks to the rename processor #7845

Add null checks and ignore_missing checks to the rename processor #7845

Conversation

ishleenk17 commented Sep 18, 2023 • edited

What does this PR do?

Checklist

Related issues

elasticmachine commented Sep 18, 2023 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Sep 18, 2023 • edited

🌐 Coverage report

muthu-mps commented Sep 21, 2023

Observations

Remove processor Sample

shmsr commented Sep 21, 2023 • edited

shmsr commented Sep 21, 2023

shmsr left a comment

Choose a reason for hiding this comment

ishleenk17 commented Sep 21, 2023 • edited

ishleenk17 commented Sep 21, 2023

muthu-mps commented Sep 21, 2023 • edited

ishleenk17 commented Sep 21, 2023

muthu-mps left a comment

Choose a reason for hiding this comment

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

elasticmachine commented Sep 21, 2023

ishleenk17 commented Sep 18, 2023 •

edited

elasticmachine commented Sep 18, 2023 •

edited

elasticmachine commented Sep 18, 2023 •

edited

shmsr commented Sep 21, 2023 •

edited

ishleenk17 commented Sep 21, 2023 •

edited

muthu-mps commented Sep 21, 2023 •

edited