Cloudflare Logpush integration does not support the correct available timestamp formats to match the output format options from the source

[travisestill]

Related GitHub issues and PR:

• Cloudflare Logpush - http_request ingest pipeline does not set @Timestamp #5571
• cloudflare_logpush: use EdgeStartTimestamp event timestamp #5599

Issue: The @timestamp field for docs ingested from Logpush to Elasticsearch is pulled from EdgeStartTimestamp and the pipeline processor is defined to accept any of the following formats:

integrations/packages/cloudflare_logpush/data_stream/http_request/elasticsearch/ingest_pipeline/default.yml

Lines 40 to 59 in 35f9966

	- set:
	if: ctx.json?.EdgeStartTimestamp != null
	field: '@timestamp'
	copy_from: json.EdgeStartTimestamp
	- date:
	field: json.EdgeEndTimestamp
	if: ctx.json?.EdgeEndTimestamp != null && ctx.json.EdgeEndTimestamp != ''
	formats:
	- ISO8601
	- uuuu-MM-dd'T'HH:mm:ssX
	- uuuu-MM-dd'T'HH:mm:ss.SSSX
	- yyyy-MM-dd'T'HH:mm:ssZ
	- yyyy-MM-dd'T'HH:mm:ss.SSSZ
	- UNIX_MS
	timezone: UTC
	target_field: cloudflare_logpush.http_request.edge.end_time
	on_failure:
	- append:
	field: error.message
	value: '{{{_ingest.on_failure_message}}}'

However, per the Logpull documentation, the available formats that can be configured are as follows:

timestamp_format: string to specify format for timestamps, such as unixnano, unix, or rfc3339. Default unixnano.

As a result, if the data comes through as UNIX for example (seconds from 1970) it would be interpreted as a UNIX_MS timestamp (milliseconds from 1970) resulting in a completely inaccurate timestamp.

Possible workarounds may include:

• Refactoring the Elasticsearch pipeline processor to use the given format.
• Reconfiguring the log output options in Cloudflare to accommodate the values are accepted by the pipeline (not recommended).

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

@travisestill Can you clarify please. The issue refers to logpush, but the linked code is in logpull.

[travisestill]

My apologies. I hadn't noticed that the link here (#5571 (comment)) was redirecting to the Logpull code.

Here's the correct code:

integrations/packages/cloudflare_logpush/data_stream/http_request/elasticsearch/ingest_pipeline/default.yml

Lines 40 to 59 in 35f9966

	- set:
	if: ctx.json?.EdgeStartTimestamp != null
	field: '@timestamp'
	copy_from: json.EdgeStartTimestamp
	- date:
	field: json.EdgeEndTimestamp
	if: ctx.json?.EdgeEndTimestamp != null && ctx.json.EdgeEndTimestamp != ''
	formats:
	- ISO8601
	- uuuu-MM-dd'T'HH:mm:ssX
	- uuuu-MM-dd'T'HH:mm:ss.SSSX
	- yyyy-MM-dd'T'HH:mm:ssZ
	- yyyy-MM-dd'T'HH:mm:ss.SSSZ
	- UNIX_MS
	timezone: UTC
	target_field: cloudflare_logpush.http_request.edge.end_time
	on_failure:
	- append:
	field: error.message
	value: '{{{_ingest.on_failure_message}}}'

And the Logpush docs suggest the same formats as the preferred: https://developers.cloudflare.com/logs/reference/log-output-options/

timestamp_format: string to specify format for timestamps, such as unixnano, unix, or rfc3339. Default unixnano.

[travisestill]

I've updated the original details to fix the previous references to logpull. The issue was reported against the Logpush integration, however, I think this may actually pertain to both Logpush and Logpull, because of the previously mentioned code compared to the options available. Both want to use unixnano (default), unix, or rfc3339, but neither pipeline code accommodates that.

[efd6]

Thanks. I have a fix for the logpull case. It can be applied to the logpush data streams too.

[travisestill]

Thanks @efd6

Just wanted to add that since unixnano is the default, we should probably handle that too.

[efd6]

The change handles that. We can't handle it natively since the date processor does not have a UNIX_NS option, so what I'm doing is converting everything to millis and then using UNIX_MS.

[travisestill]

Thanks, @efd6, for the quick turn around!

I noticed the fixed version is 1.8.1, but the version in the EPR and corresponding pipelines is 1.4 (Elasticsearch version 8.6.2).

Is the availability of the fix a way out from being available, or do the versions in the UI not reflect the release versioning?

[andrewkroh]

The latest version request 8.7.1.

curl "https://epr.elastic.co/search?package=cloudflare_logpush&all=true" | jq -c '.[] | {"version": .version, "stack": .conditions.kibana.version}'
{"version":"1.0.0","stack":"^8.0.0"}
{"version":"1.1.0","stack":"^8.0.0"}
{"version":"1.1.1","stack":"^8.0.0"}
{"version":"1.2.0","stack":"^8.0.0"}
{"version":"1.2.1","stack":"^8.0.0"}
{"version":"1.3.0","stack":"^8.0.0"}
{"version":"1.3.1","stack":"^8.0.0"}
{"version":"1.4.0","stack":"^8.0.0"}
{"version":"1.5.0","stack":"^8.7.0"}
{"version":"1.6.0","stack":"^8.7.1"}
{"version":"1.7.0","stack":"^8.7.1"}
{"version":"1.8.0","stack":"^8.7.1"}
{"version":"1.8.1","stack":"^8.7.1"}