[Infoblox NIOS] Update timestamp parsing logic

[kcreddy]

Proposed commit message

There are 2 style of log formats (regards to how timestamps are handled).

1. <29>Mar 21 09:53:51 infoblox.localdomain httpd[]: 2022-03-18 13:24:41.705Z [admin]: ........
2. <30>Mar 27 08:32:59 infoblox.localdomain dhcpd[2567]: ............

Event 1: Contains 2 timestamps inside the log. In current default pipeline, the first timestamp Mar 21 09:53:51 is set as event.created and then individual pipelines sets second timestamp as @timestamp if present.
Event 2: Contains 1 timestamp inside the log. In current default pipeline, the only timestamp Mar 27 08:32:59 is set as event.created and a set processor copies that into @timestamp if @timestamp is not already present from individual pipelines.

But, when Logstash is involved in between Agent and Elasticsearch, it adds @timestamp to the event before ingest pipeline. Hence, the logs format 2, i.e., containing single timestamp, skips the set processor leading to incorrect @timestamp values and also not conforming to ECS norm @timestamp < event.created < event.ingested.

This PR fixes the problem by:

1. Modifying individual pipelines to not set @timestamp inside them.
2. Set override: true in the default pipeline, thus overriding the value of @timestamp with event's timestamp.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

elastic-package stack down && elastic-package build && elastic-package stack up --version=8.11.0 -d -v --services=elasticsearch && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v

Run pipeline tests for the package
--- Test results for package: infoblox_nios - START ---
╭───────────────┬─────────────┬───────────┬────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME      │ RESULT │ TIME ELAPSED │
├───────────────┼─────────────┼───────────┼────────────────┼────────┼──────────────┤
│ infoblox_nios │ log         │ pipeline  │ test-audit.log │ PASS   │  12.847542ms │
│ infoblox_nios │ log         │ pipeline  │ test-dhcp.log  │ PASS   │  16.714875ms │
│ infoblox_nios │ log         │ pipeline  │ test-dns.log   │ PASS   │  10.268166ms │
╰───────────────┴─────────────┴───────────┴────────────────┴────────┴──────────────╯
--- Test results for package: infoblox_nios - END   ---
Done

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

AFAICS the ordering of the timestamps is coming from the structure of the event. Is this the case? If so, I think it would be helpful to make this clear in the commit message. If not, can you explain the logic more clearly?

[efd6]

Should this deletion also happen in the next date processor?

[kcreddy]

I've made logic bit clearer and this is irrelevant now.

[kcreddy]

AFAICS the ordering of the timestamps is coming from the structure of the event. Is this the case? If so, I think it would be helpful to make this clear in the commit message. If not, can you explain the logic more clearly?

I've made commit message more clearer. Let me know if that explains the issue. Thanks!

[elasticmachine]

Package infoblox_nios - 1.19.3 containing this change is available at https://epr.elastic.co/search?package=infoblox_nios