trace_id field extraction from logs

Dear Team,
We have installed Signoz using helm charts by supplying override-values.yaml in our EKS cluster. After installation -we configured OTEL URL in our apps.

In the Frontend dashboard, we were able to see logs coming in but we were not getting value for trace_id field. I have attached the screenshot where traced_id is showing body{} but not coming in the interested fields.

Can someone please help here with what could be the issue? I have attached my override-values.yaml and screenshot missing trace_id from logs

override-values.yaml :

global:
storageClass: gp2
cloud: gcp

clickhouse:
installCustomStorageClass: true
otel:
receiver:
otlp:
protocols:
grpc:
endpoint: otel-collector.default.svc.cluster.local:4317
exporter:
jaeger:
endpoint: http://jaeger-query:16686
logging:
loglevel: debug
logformat: json
processor:
batch:
timeout: 5s
attribute:
actions:
- key: trace_id
action: upsert
from_attribute: log
value_type: string
otlp:
protocols:
grpc:
parser:
json:
timestamp:
field: "time"
format: "RFC3339Nano"
attributes:
- key: log
field: "message"
- key: trace_id
field: "trace_id"

[nityanandagohain]

You will have to use the trace parser for this https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/trace_parser.md

Here is an example https://github.com/SigNoz/logs-benchmark/blob/0b2451e6108d8fa5fdd5808c4e174bd52b9d55d3/signoz/signoz-client/otel-collector-config.yaml#L22

[charanparvathaneni]

@nityanandagohain
We are using below override-values.yaml but still not seeing the trace_id value in the trace_id field. Please help

global:
storageClass: gp2
cloud: gcp

clickhouse:
installCustomStorageClass: true

receivers:
tcplog:
listen_address: "0.0.0.0:2255"
attributes: {}
resource:
host: "mys"
add_attributes: false
operators:
- type: regex_parser
regex: '^.(?P{.})'
output: parsedata
- type: json_parser
id: parsedata
parse_from: attributes.body
- type: move
from: attributes["dummyBytes"]
to: body
- type: time_parser
parse_from: attributes.datetime
layout_type: strptime
layout: '%d/%b/%Y:%H:%M:%S %z'
- type: trace_parser
trace_id:
parse_from: attributes.traceId
span_id:
parse_from: attributes.spanId
- type: remove
field: attributes.body
- type: remove
field: attributes.datetime
- type: remove
field: attributes.traceId
- type: remove
field: attributes.spanId
- type: move
from: attributes["user-identifier"]
to: attributes.user_identifier

processors:
batch:
send_batch_size: 30000
send_batch_max_size: 31000
timeout: 10s

extensions:
health_check:
endpoint: 0.0.0.0:13133
zpages:
endpoint: 0.0.0.0:55679
pprof:
endpoint: 0.0.0.0:1777

exporters:
logging:
loglevel: debug
log_attributes:
- trace_id
- span_id
- body
otlp/signoz:
endpoint: 172.31.1.23:4317
tls:
insecure: true

service:
telemetry:
metrics:
address: 0.0.0.0:8888
logs:
level: "debug"
extensions:
- health_check
- zpages
- pprof
pipelines:
logs:
receivers: [tcplog]
processors: [batch]
exporters: [otlp/signoz]

[nityanandagohain]

@charanparvathaneni what is the format of logs you are sending, the above configuration is for parsing logs of this format.

{
  "host": "6.127.39.239",
  "user-identifier": "wisoky5015",
  "datetime": "09/Nov/2022:04:48:14 +0000",
  "method": "HEAD",
  "request": "/revolutionary/roi",
  "protocol": "HTTP/2.0",
  "status": 501,
  "bytes": 11466,
  "referer": "http://www.seniorrevolutionize.name/intuitive/matrix/24/7",
  "traceId": "3aee4c94836ffe064b37f054dc0e1c50",
  "spanId": "4b37f054dc0e1c50",
  "dummyBytes": "kingship grumble swearer hording wiggling dipper execution's crock's clobbers Spaniard's priestess's organises Harrods's Josef's Wilma meditating movable's photographers polestars pectorals Coventries rehearsal Arkhangelsk's Kasai barometer junkier Osgood's impassively illogical cardsharp's underbrush shorter patronymics plagiarises gargle's Chandra intransigent deathtrap extraneously hairless inferred glads frail polka jeez ohm bigoted sari's cannonades vestibule's pursuer vanity ceremony Kodaly's swaths disturbance's belt Samoan's invulnerability dumping Alfreda padded corrosive universal dishwasher's multiplier decisive selloff eastbound moods Konrad's appositive's reset's ingenuously checkup's overselling evens Darrin's supernumerary liberalism productivity's legrooms Lorenzo including Robbin savourier foxglove's buckshot businesswomen abalones tare Chaitin Stephenson's inpatients distinction cryings aspic empire's healed perspiring"
}

[charanparvathaneni]

@nityanandagohain
our application-sample log looks like below.Please help .

########################
{
"hostname": "account-api-c695c7cd7-pvk9n",
"level": "info",
"message": "Response /api/account/users/me",
"method": "GET",
"responseBody": {
"_id": "64636f00ab5974d2945c7c4b",
"authUserId": "56e989ef-96e0-409f-a167-5a3fb1ac1362",
"createdAt": "Tue May 16 2023 11:54:40 GMT+0000 (Coordinated Universal Time)",
"createdBy": "9570e51d-720b-4195-af89-be611aca2c9d",
"email": "doctor104@mailinator.com",
"firstName": "Sudheer",
"lastName": "Sudheer",
"phoneNumber": {
"dialCode": "+91",
"value": 999999999
},
"status": "Invited",
"tenant": [
{
"status": "InActive",
"tenantId": "6463609cf72b9f2b0b54cabc"
}
],
"updatedAt": "Tue May 16 2023 11:54:40 GMT+0000 (Coordinated Universal Time)"
},
"span_id": "46a992e6f848706f",
"timestamp": "2023-05-18T05:57:20.791Z",
"trace_flags": "01",
"trace_id": "8c67114211b24caf694625e933175897",
"type": "RESPONSE",
"url": "/api/account/users/me"
}