rsyslog omotel module for native OTLP logging

[flantel]

Hi,

In the latest versions of rsyslog (not yet released in any of the packages) there is a omotel module which provides native OTLP exports from rsyslog. https://docs.rsyslog.com/doc//configuration/modules/omotel.html

I can confirm the logs get put into Homer11, however, I still have to work out how to get the format correct so they are searchable by call-id.

Any one else looked at this? How do you currently get your kamailio logs to OTLP?

Great work on homer11!

-Barry

[adubovikov]

Hi @flantel

good timing! This works out of the box with Homer 11, you just need the Call-ID to be present in the log line itself.

Homer's OTLP ingest stores log records as-is into the otlp_logs table (body + attributes + trace/span ids) - it doesn't try to parse SIP Call-IDs out of them. The correlation happens at query time: when you open a SIP transaction in Homer, the OTLP Logs tab searches for the call's Call-ID as a substring across the log body and attributes. So the only requirement is that Kamailio prints the Call-ID in every log line.

1. Kamailio side - add a log prefix with $ci in kamailio.cfg (see core cookbook: log_prefix):

log_prefix="{$mt $hdr(CSeq) $ci} "

This prefixes every log message with the message type, CSeq and Call-ID. One caveat: the prefix only applies to logs printed while Kamailio is processing a SIP message (route blocks). Logs coming from timers, evapi workers etc. have no SIP context, so they won't carry a Call-ID and won't be found by call-id search.

2. rsyslog side - a plain omotel action pointed at Homer's OTLP/HTTP port is enough, no special template needed:

module(load="omotel")
action(
  type="omotel"
  endpoint="http://homer-host:4318"
  path="/v1/logs"
  protocol="http/json"
  resource='{ "service.name": "kamailio" }'
)

protocol="http/protobuf" works too. Setting service.name is optional but recommended - it lands in the service_name column (otherwise omotel defaults it to rsyslog), and it's what the user_agent search filter matches on.

A note on the top-level search: the call_id filter on the otlp_logs proto in the search UI maps to an exact trace_id match - OTLP trace ids must be 32 hex chars, so a raw SIP Call-ID can't go there. Call-ID correlation for logs is done from the SIP transaction view (the OTLP Logs tab), which is the intended flow: search the call by Call-ID, open the transaction, and the matching log lines show up next to the SIP ladder.

Regards,
Alexandr

[adubovikov]

@flantel can we close the discussion ? Did you figure it out ?

[flantel]

Hi Alexandr. I did indeed. Many thanks.

[adubovikov]

Super :-) Enjoy!