Cisco FTD | Create security_event fields from flattened security field

[kcreddy]

Proposed commit message

• WHAT: Parses out most fields from flattened cisco.ftd.security into new field cisco.ftd.security_event.
• WHY: Allows aggregation of fields under cisco.ftd.security_event which was not possible due to flattened cisco.ftd.security field.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Run this command for pipeline tests:
$ elastic-package stack down && elastic-package build && elastic-package stack up --version=8.10.3 -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v

Expected files are now updated with new field cisco.ftd.security_event containing the fields listed from the linked issue.

Run pipeline tests for the package
2023/11/14 11:36:16 DEBUG GET https://127.0.0.1:5601/api/status
--- Test results for package: cisco_ftd - START ---
╭───────────┬─────────────┬───────────┬────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                      │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼────────────────────────────────┼────────┼──────────────┤
│ cisco_ftd │ log         │ pipeline  │ test-asa-fix.log               │ PASS   │       5.95ms │
│ cisco_ftd │ log         │ pipeline  │ test-asa.log                   │ PASS   │ 141.360791ms │
│ cisco_ftd │ log         │ pipeline  │ test-dns.log                   │ PASS   │  28.012791ms │
│ cisco_ftd │ log         │ pipeline  │ test-filtered.log              │ PASS   │   3.302541ms │
│ cisco_ftd │ log         │ pipeline  │ test-firepower-management.log  │ PASS   │  31.007625ms │
│ cisco_ftd │ log         │ pipeline  │ test-ftd-endpoint-profile.log  │ PASS   │  49.369333ms │
│ cisco_ftd │ log         │ pipeline  │ test-ftd-fix.log               │ PASS   │      7.219ms │
│ cisco_ftd │ log         │ pipeline  │ test-ftd-inbound-outbound.log  │ PASS   │  20.108375ms │
│ cisco_ftd │ log         │ pipeline  │ test-intrusion.log             │ PASS   │  18.601208ms │
│ cisco_ftd │ log         │ pipeline  │ test-no-type-id.log            │ PASS   │  14.540875ms │
│ cisco_ftd │ log         │ pipeline  │ test-not-ip.log                │ PASS   │    4.35875ms │
│ cisco_ftd │ log         │ pipeline  │ test-sample.log                │ PASS   │  30.343333ms │
│ cisco_ftd │ log         │ pipeline  │ test-security-connection.log   │ PASS   │    10.5255ms │
│ cisco_ftd │ log         │ pipeline  │ test-security-file-malware.log │ PASS   │  32.704292ms │
│ cisco_ftd │ log         │ pipeline  │ test-security-malware-site.log │ PASS   │   7.476917ms │
╰───────────┴─────────────┴───────────┴────────────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_ftd - END   ---
Done

Related issues

• Closes Cisco FTD Component Template Maps cisco.ftd.security as flattened #5769

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-11-20T07:24:54.727+0000

• Duration: 19 min 38 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22
Skipped	0
Total	22

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (1/1)	💚
Files	100.0% (1/1)	💚 3.851
Classes	100.0% (1/1)	💚 3.851
Methods	100.0% (22/22)	💚 7.805
Lines	71.773% (1490/2076)	👎 -16.713
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

ISTM that this could be claimed to be a breaking change; if a user has an @custom pipeline that copies fields out of cisco.ftd.security.* and they are in the set of fields that are now moved to cisco.ftd.security_event.* then their custom pipeline will fail to result in documents whose structure downstream analysis may depend on.

[kcreddy]

if a user has an @custom pipeline that copies fields out of cisco.ftd.security.* and they are in the set of fields that are now moved to cisco.ftd.security_event.* then their custom pipeline will fail.

I am not sure how we can handle this problem efficiently because @custom pipeline is always applied at the very end of the pipeline. Instead of moving, we can just copy the fields into cisco.ftd.security_event.*, thus keeping existing fields intact, but that creates a lot of duplicates. We could possibly inform the users that this upgrade will be a breaking change from the Fleet UI if thats how breaking changes are handled.

[P1llus]

if a user has an @custom pipeline that copies fields out of cisco.ftd.security.* and they are in the set of fields that are now moved to cisco.ftd.security_event.* then their custom pipeline will fail.

I am not sure how we can handle this problem efficiently because @custom pipeline is always applied at the very end of the pipeline. Instead of moving, we can just copy the fields into cisco.ftd.security_event.*, thus keeping existing fields intact, but that creates a lot of duplicates. We could possibly inform the users that this upgrade will be a breaking change from the Fleet UI if thats how breaking changes are handled.

I think we should ensure it is in the changelog title at least, as that is more prominent in the UI now compared to before, so that people know exactly what is going on.
We could keep the fields as well, but with that amount of fields it is a bit waste. I think we should just let @jamiehynds conclude on which approach we should take.

From my side its all good as long as its shown in the changelog and readme 👍

[kcreddy]

Updated the changelog title with NOTE - This is a breaking change.

[narph]

cc @jamiehynds

[jamiehynds]

@kcreddy sorry I missed the original ping from Marius. As long as we're reflecting the breaking change in the changelog (and thus appearing in the UI when a user upgrades the integration), all good with me.

[kcreddy]

As long as we're reflecting the breaking change in the changelog (and thus appearing in the UI when a user upgrades the integration),

Hey @jamie @P1llus , I have added following to the changelog description: NOTE - This is a breaking change. Is this enough for it to appear in the UI? Is there any documentation to follow for breaking changes?
If the note alone is enough, I can go ahead and merge the PR.

[elasticmachine]

Package cisco_ftd - 3.0.0 containing this change is available at https://epr.elastic.co/search?package=cisco_ftd