[panw] Fix ignore_missing on threat XFF convert processor.

[ie-ops]

Executive summary

Added ignore_missing: true to the convert processor for _temp_.forwarded_ip in the THREAT pipeline. Without this guard, any THREAT log lacking an X-Forwarded-For header would fail the processor with a 'field not present' error, causing the entire document to error rather than process successfully. A new test fixture (URL subtype with no XFF) was added to cover this previously untested path.

Proposed commit message

[panw] Fix ignore_missing on threat XFF convert processor.

Root cause

The convert processor in threat.yml reads _temp_.forwarded_ip without ignore_missing: true; when a THREAT URL log arrives without the optional X-Forwarded-For HTTP header, the field is absent and Elasticsearch throws an error before the on_failure handler can fire.

Approach

Add ignore_missing: true to the convert processor tagged convert__temp__forwarded_ip_to_network_forwarded_ip_e6722dcc in threat.yml (line 195). This mirrors the pattern already used in default.yml for the same network.forwarded_ip field (tag convert_network_forwarded_ip_8bdee41f), where ignore_missing: true is present. The on_failure rename block below already has ignore_missing: true so no change is needed there. Add a test fixture for a URL-subtype THREAT event that has no XFF value, bump the package version to 5.5.1, and add a bugfix changelog entry.

Implementation

1. Step 1: In packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/threat.yml at line 195, add ignore_missing: true to the convert processor tagged convert__temp__forwarded_ip_to_network_forwarded_ip_e6722dcc (after the type: ip line).
2. Step 2: Create a new test fixture log file packages/panw/data_stream/panos/_dev/test/pipeline/test-panw-panos-threat-url-no-xff-sample.log containing the sanitized event (URL-subtype THREAT log with no XFF value).
3. Step 3: Create the corresponding config file test-panw-panos-threat-url-no-xff-sample.log-config.yml matching the pattern of the existing threat test config.
4. Step 4: Generate the expected output file test-panw-panos-threat-url-no-xff-sample.log-expected.json by running elastic-package test pipeline and capturing the result; verify no pipeline_error tag is present in the output.
5. Step 5: In packages/panw/manifest.yml, bump the version from 5.5.0 to 5.5.1.
6. Step 6: In packages/panw/changelog.yml, prepend a new entry for version 5.5.1 with a bugfix entry: 'Fix missing ignore_missing guard on XFF convert processor in threat pipeline, preventing errors for THREAT logs without an X-Forwarded-For header.'.

Pipeline changes

• In threat.yml, add ignore_missing: true to the convert processor (tag: convert__temp__forwarded_ip_to_network_forwarded_ip_e6722dcc) that reads _temp_.forwarded_ip and writes to network.forwarded_ip. No other processor changes required.

Field / mapping changes

—

Sanitized error message

field [forwarded_ip] not present as part of path [_temp_.forwarded_ip]

Sanitized log (event_sanitized excerpt)

2026-06-27 14:25:58-04:00 host-vpn-1.example.local INF 1,2026/06/27 14:25:58,4A9B2C5D7F8E1A3B,THREAT,url,2562,2026/06/27 14:25:58,198.51.100.50,3.234.152.120,192.0.2.75,3.234.152.120,AllowWebAllowlist,,,ssl,vsys1,LOB,Untrusted,tunnel.1,ethernet1/1,ForwardAllLogsViaAWS,2026/06/27 14:25:58,1121335,1,11699,443,10829,443,0x40b400,tcp,alert,"60f119f1e98643dfb9d91a7c75b505a1.us-east-1.elasticsearch.example.com/",(9999),Web-URL-Allowlist,informational,client-to-server,7603943486192104764,0x0,10.0.0.0-10.255.255.255,United States,,,0,,,0,,,,,,,,0,0,0,0,0,,host-vpn-1.example.local,,,,,0,,0,,N/A,N/A,AppThreat-0-0,0x0,0,4294967295,," RIACoderAllowedURLs,business-and-economy,low-risk",89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,2026-06-27T14:25:58.415-04:00,,,,encrypted-tunnel,networking,browser-based,4,"used-by-malware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,pervasive-use",,ssl,no,no,

Reviewer concerns

• The new test log entry in test-panw-panos-threat-sample.log uses a short/truncated record (fewer trailing commas) — reviewers should confirm the parser tolerates variable-length THREAT records for this PAN-OS version (2049 log format).
• The expected output timezone +09:30 (Adelaide) is unusual for a PA-220 device; this is driven by the syslog receiver offset and is correct if intentional, but worth a second look.
• The changelog PR link placeholder (pull/1) should be updated to the real PR number before merge.

Self-review findings

—

Risk and classification

• Plan risk level: low
• Tags: pipeline, processors, test-fixture
• Impact: medium

Links

• Issue: (no issue number)
• Issue title: panw.panos [PIPELINE_FIX]: field [forwarded_ip] not present as part of path [temp.forwarded_ip]
• Pipeline case: 09b1d6e47e099cdc

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

Pinging @elastic/integration-experience (Team:Integration-Experience)

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[vera-review-bot]

Severity: 🟡 Medium confidence: high path: packages/panw/data_stream/panos/_dev/test/pipeline/test-panw-panos-threat-sample.log:220

The new sample log leaves the X-Forwarded-For column empty, so its expected output is identical with and without the ignore_missing fix and it guards nothing. Add a sample with the XFF column populated so the convert branch is actually exercised.

Details

In threat.yml the csv processor maps CSV column 42 to _temp_.forwarded_ip. In this new line that column is empty (the run ...,0,,,,,,,,0,0,0,0,0,,PA-220,... after url_idx), and the Elasticsearch csv processor skips empty values when empty_value is not configured, so _temp_.forwarded_ip is never set.

Before this PR that missing field made the convert throw, but the processor's on_failure rename already carried ignore_missing: true (threat.yml:206), so the failure was swallowed with no error.message, no event.kind: pipeline_error, and no field written. The rendered document is therefore byte-identical before and after adding ignore_missing: true — this test case would pass unchanged against main.

The missing-XFF path is also already covered: the pre-existing samples on lines 218 and 219 (pa555 and FW-DNS-11) both have an empty column 42 and pass today.

What is genuinely untested is the populated-XFF behaviour. network.forwarded_ip and panw.panos.forwarded_ip appear nowhere in test-panw-panos-threat-sample.log-expected.json, and the four x_forwarded_for entries that do appear come from default.yml:1372 (_temp_.srcuser renamed when label bit 0x00080000 is set), not from this convert processor's on_failure.

Recommendation:

Populate CSV column 42 so the convert actually runs and writes network.forwarded_ip/panw.panos.forwarded_ip. Using the same line with the XFF slot filled (3rd empty field after url_idx):

Nov 30 16:44:37 PA-220 1,2018/11/30 16:44:37,012801096514,THREAT,url,2049,2018/11/30 16:44:37,192.168.15.224,175.16.199.1,192.168.1.63,175.16.199.1,new_outbound_from_trust,,,ssl,vsys1,trust,untrust,ethernet1/2,ethernet1/1,send_to_mac,2018/11/30 16:44:37,28192,1,52985,443,37680,443,0x403000,tcp,alert,"consent.cmp.oath.com/",(9999),Web-URL-Allowlist,informational,client-to-server,7727,0x0,192.168.0.0-192.168.255.255,United States,0,,0,,,0,,,192.0.2.55,,,,,0,0,0,0,0,,PA-220,,,,,0,,0,,N/A,N/A,AppThreat-0-0,0x0,0,4294967295,,

A second variant with a non-IP value in that slot (for example unknown-proxy) would cover the on_failure fallback into panw.panos.x_forwarded_for. Regenerate expectations with elastic-package test pipeline -g.

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Severity: 🔵 Low confidence: high path: packages/panw/changelog.yml:4

The changelog says the fix prevents "errors" for THREAT logs without an XFF header, but no error was ever surfaced — reword it to describe the redundant processor failure this actually removes.

Details

The convert processor's on_failure rename already sets ignore_missing: true (threat.yml:206), so a missing _temp_.forwarded_ip was caught and discarded silently: no error.message was appended, event.kind was never set to pipeline_error, and no tag was added. Users saw no errors on THREAT logs without an X-Forwarded-For header.

As written, this entry tells operators that documents were failing to ingest, which will send them looking for data loss that did not occur. What the change actually removes is a thrown-and-caught processor exception on every THREAT log lacking an XFF value.

Recommendation:

Describe the real effect:

- version: "5.5.1"
  changes:
    - description: Add `ignore_missing` to the XFF convert processor in the threat pipeline to avoid a redundant processor failure on THREAT logs without an X-Forwarded-For value.
      type: bugfix
      link: https://github.com/elastic/integrations/pull/20402

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Review summary

Issues found across the latest commits 4241d9a — 1 medium, 1 low

• 🟡 The new sample log leaves the X-Forwarded-For column empty, so its expected output is identical with and without the ignore_missing fix and it guards nothing. Add a sample with the XFF column populated so the convert branch is actually exercised. (link) (Unresolved)
• 🔵 The changelog says the fix prevents "errors" for THREAT logs without an XFF header, but no error was ever surfaced — reword it to describe the redundant processor failure this actually removes. (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 4241d9a

History

• 💔 Build #46857 failed 907ed57