[akamai] Fix Pipeline for Populating Http Message Headers

[mohitjha-elastic]

Proposed commit message

akamai: fix pipeline for populating http message headers.

The Ingest pipeline failed when responseHeaders or requestHeaders could not be parsed 
as Name: value lines, including Akamai’s {p} placeholder and RFC 7230 obs-fold continuations
(e.g. folded Content-Security-Policy). Non-parseable string values are dropped with a remove 
processor when the field contains no colon, so the KV step is not run on placeholders.
Obs-fold sequences (\r\n followed by space or tab) are normalized to a single space with 
gsub before KV, which matches the spec’s treatment of folded linear whitespace.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install the elastic package locally.
• Start the elastic stack using the elastic package.
• Move to integrations/packages/akamai directory.
• Run the following command to run tests.

elastic-package test -v

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[chrisberkhout]

I put some comments about issues in the current implementation, but I think it would be better to switch to a different implementation that keeps the header values the same by placing some markers to support key-value splitting, rather than removing tricky content from the actual values. There's an implementation of that here:

integrations/packages/beelzebub/data_stream/logs/elasticsearch/ingest_pipeline/default.yml

Lines 208 to 227 in 40db1ba

	- gsub:
	if: ctx.json?.event?.Headers == null
	description: "Fix field separator to support KVP extraction"
	field: _tmp.HeadersText
	ignore_missing: true
	pattern: '],\[Key: '
	replacement: ',##'
	- kv:
	if: ctx.json?.event?.Headers == null
	description: "Extract headers as key value pairs."
	field: _tmp.HeadersText
	ignore_missing: true
	ignore_failure: true
	field_split: ',##'
	value_split: ', values: '
	target_field: json.event.Headers
	trim_key: ' '
	trim_value: ' '
	strip_brackets: true

[mohitjha-elastic]

Thanks @chrisberkhout
I’ve switched to the gsub + kv–only style for obs-fold handling, without a follow-up script.

Non-parseable placeholders (e.g. {p}) are dropped with a remove processor when the string contains no :``, replacing the earlier painless blocks.
This keeps the pipeline to ingest processors only for the folding path (no marker round-trip script) while still avoiding KV failures on folded CSP-style headers.

Please review the updated pipeline logic and let me know what you think.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f6e2ba6

History

• 💚 Build #40796 succeeded a31119e

cc @mohitjha-elastic

[chrisberkhout]

Thanks for making the suggested improvements!

I originally misunderstood the gsub. Replacing \r\n[\t ]+ with a single space gives us the original value by removing the obsolete line folding ( "obs-fold") defined in RFC 7230. That normalization doesn't change the semantic value, it just converts from the folded presentation into the canonical single-line form. So that's what we want.

[elastic-vault-github-plugin-prod]

Package akamai - 3.1.1 containing this change is available at https://epr.elastic.co/package/akamai/3.1.1/