ping_federate: strip brackets from IPv6 in audit CEF before decode_cef

[navnit-elastic]

Proposed commit message

ping_federate: strip brackets from IPv6 in audit CEF before decode_cef

PingFederate can emit CEF audit logs with IPv6 in brackets
(e.g. src=[2a01:599:110:e652:...]), which violates CEF spec and causes
decode_cef to fail with "value is not a valid IP address".

Add a script processor before decode_cef in TCP, UDP, and filestream
streams to remove brackets around IPv6 addresses so the decoder
receives a valid IP.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

System Tests:

--- Test results for package: ping_federate - START ---
╭───────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├───────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ ping_federate │ admin       │ system    │ logfile   │ PASS   │    41.068725s │
│ ping_federate │ audit       │ system    │ logfile   │ PASS   │ 36.846062666s │
│ ping_federate │ audit       │ system    │ tcp       │ PASS   │ 40.067340083s │
│ ping_federate │ audit       │ system    │ tls       │ PASS   │ 40.995306209s │
│ ping_federate │ audit       │ system    │ udp       │ PASS   │ 41.058248209s │
╰───────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: ping_federate - END   ---
Done

Related issues

Screenshots

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

Can you add a link to documentation that supports the rationale for this change? The CEF parser that we are using does not support IPv6 (the src field described in the v25 spec says that it is an IPv4, "Identifies the source that an event refers to in an IP network. The format is an IPv4 address. Example: “192.168.10.1”." ISTM that this is a workaround for a format that's not documented to support IPv6 (so it's not a bugfix, at least not in the processor; it is likely a bug in the data source if we are seeing IPv6 data coming through, though note paragraph below regarding spec versions).

If there is a later version of the spec that does support IPv6, we should probably make a change to the processor in beats. We can apply this workaround, but it should be noted as such.

[andrewkroh]

Relates: elastic/beats#40269 (for CEF spec updates, mentions IPv6)

[navnit-elastic]

The CEF parser that we are using does not support IPv6 (the src field described in the v25 spec says that it is an IPv4, "Identifies the source that an event refers to in an IP network. The format is an IPv4 address. Example: “192.168.10.1”." ISTM that this is a workaround for a format that's not documented to support IPv6 (so it's not a bugfix, at least not in the processor; it is likely a bug in the data source if we are seeing IPv6 data coming through

The failure is due to brackets around the IP, which are not part of the CEF src format in the spec.

The change strips those brackets so the parser receives a valid IP string; it’s a format normalization, not an IPv6‑vs‑IPv4 feature.

The CEF v25 “src is IPv4” point is a separate spec detail; if the processor later supports IPv6 per a newer spec, the same “no brackets” rule would still apply.

[efd6]

Yes, it looks like Ping Federate had the same misconception as I did; they have misunderstood IPv6 syntax in a URI as being the general syntax. So what I said about IPv6 updates fixing it was wrong, but the comment about the nature of the change is now more correct; this is not a bugfix. The bug is in Ping Federate.

We should probably relax the CEF processor, I'm sure others will have made the same mistake.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: a3b5451

History

• 💚 Build #38998 succeeded b163a1b

cc @navnit-elastic

[efd6]

Just a note that the IPv6 pattern is more complex than this. An indication of this is from the grok patterns here

[navnit-elastic]

Thanks for the note.

This step only strips the brackets; it doesn’t validate the contents. Real validation happens in decode_cef, which will still reject invalid IPs.

[efd6]

Yeah, my concern (minor) was that if there was anything else in the message that is [hexdigitsandcolons] then it would be mutated as well even if it were not an IPv6. I think this is reasonably unlikely, but it is less unlikely than something that specifically looks like an IPv6 and isn't an IPv6.

[navnit-elastic]

Thanks for clarifying.

Agreed - there's a small risk. We will keep this in mind for the future.

[elastic-vault-github-plugin-prod]

Package ping_federate - 1.2.0 containing this change is available at https://epr.elastic.co/package/ping_federate/1.2.0/