Skip to content

Log data format change between 2.1.0 and 2.2.0 #1310

Closed
@thenodon

Description

@thenodon
Contributor

Description:
I currently doing a evaluation of parseable and just updated from 2.1.0 to 2.2.0. But when upgrading to 2.2.0 the log format is different where all my otel attributes gets located into a key called other_attributes. This change looks like it was created in commit #1298

In 2.1.0 the log data looks like this in the parseable UI:

application: "com.docker.backend"
availability_zone: "zone-1"
body: "May 10 13:31:50 homebase com.docker.backend[192162]: [11:31:50.442155604Z][vm.lifecycle-server.apiproxy] << GET /v1.46/containers/43cc5c7628cca59ccdbd36f07deb237f2bad085cf2837947a8ca749cd87f080d/json (1.495144ms)"
flags: 0
host: "homebase"
log.file.name: "syslog"
log_record_dropped_attributes_count: 0
message: "[11:31:50.442155604Z][vm.lifecycle-server.apiproxy] << GET /v1.46/containers/43cc5c7628cca59ccdbd36f07deb237f2bad085cf2837947a8ca749cd87f080d/json (1.495144ms)"
observed_time_unix_nano: "2025-05-10T11:31:50.584"
p_format: "otel-logs"
p_src_ip: "127.0.0.1"
p_user_agent: "OpenTelemetry Collector Contrib/0.125.0 (linux/amd64)"
pid: "192162"
program: "com.docker.backend"
resource_dropped_attributes_count: 0
schema_url: ""
scope_dropped_attributes_count: 0
scope_log_schema_url: ""
scope_name: ""
scope_version: ""
severity_number: 0
severity_text: "SEVERITY_NUMBER_UNSPECIFIED"
span_id: ""
time_unix_nano: "2025-05-10T11:31:50"
timestamp: "May 10 13:31:50"
timestamp_my_nano: "1746876710000000000"
trace_id: ""

but in 2.2.0 it looks like this

body: May 10 13:33:34 homebase com.docker.backend[192162]: [11:33:34.340657800Z][main.apiproxy         ] << GET /v1.46/containers/43cc5c7628cca59ccdbd36f07deb237f2bad085cf2837947a8ca749cd87f080d/json (4.17477ms)
flags: 0
host: homebase
log_record_dropped_attributes_count: 0
observed_time_unix_nano: 2025-05-10T11:33:34.384
other_attributes: {"timestamp":"May 10 13:33:34","program":"com.docker.backend","pid":"192162","message":"[11:33:34.340657800Z][main.apiproxy         ] << GET /v1.46/containers/43cc5c7628cca59ccdbd36f07deb237f2bad085cf2837947a8ca749cd87f080d/json (4.17477ms)","log.file.name":"syslog","timestamp_my_nano":"1746876814000000000","application":"com.docker.backend","availability_zone":"zone-1"}
p_format: otel-logs
p_src_ip: 127.0.0.1
p_user_agent: OpenTelemetry Collector Contrib/0.125.0 (linux/amd64)
resource_dropped_attributes_count: 0
schema_url: 
scope_dropped_attributes_count: 0
scope_log_schema_url: 
scope_name: 
scope_version: 
severity_number: 0
severity_text: SEVERITY_NUMBER_UNSPECIFIED
span_id: 
time_unix_nano: 2025-05-10T11:33:34
trace_id: 

So all the attributes I created in the otel collector like program, is now just part of the other_attributes structure.
Im not sure I understand the logic behind this change or how the query with SQL is possible in a simple way on the attributes.
This change broke all the dashboards, alerts etc that was based on doing SQL against parseable. With SQL as the query language all attributes should be possible to query based on the otel attributes, like select on, aggregate, group by etc. I think this is key to be a otel native tool.

Activity

nitisht

nitisht commented on May 11, 2025

@nitisht
Member

Thanks for reporting @thenodon , we'll get back on this soon

added a commit that references this issue on May 12, 2025
0eefe77
added a commit that references this issue on May 14, 2025
f240dde
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @thenodon@nitisht@nikhilsinhaparseable

    Issue actions

      Log data format change between 2.1.0 and 2.2.0 · Issue #1310 · parseablehq/parseable