loki.source.syslog - no internal labels like __syslog(.+)available.

[Nachtfalkeaw]

What's wrong?

loki.source.syslog should provide _syslog_message* internal labels which if needed can be relabeled into static labels.
However with grafana alloy 1.5.1 this is not working and you have to use a workaround.

Steps to reproduce

The important part in this config is this: You ne a loki.relabel with action = labelmap.

loki.relabel "syslog_01" {
    rule {
      action = "labelmap"
      regex = "__syslog_(.+)"
    }

Second and most important ist you need to address the component as:
relabel_rules = loki.relabel.syslog_01.rules
and NOT as
forward_to = [loki.relabel.syslog_01.receiver]

loki.source.syslog "syslog_receiver_udp_514" {
  // RFC5424 listeners on port 514
  listener {
    address = "0.0.0.0:514"
    protocol = "udp"
    use_incoming_timestamp = false
    label_structured_data = true
    use_rfc5424_message = true
    max_message_length = 0
    syslog_format = "rfc5424"
    labels = {
      protocol = "udp",
      format = "rfc5424",
      port = "514",
      service_name = "syslog_rfc5424_514",
    }
  }
  relabel_rules = loki.relabel.syslog_01.rules
//  forward_to = [loki.relabel.syslog_01.receiver]
  forward_to = [loki.process.syslog_processor.receiver]


}


loki.source.syslog "syslog_receiver_tcp_514" {
  listener {
    address = "0.0.0.0:514"
    protocol = "tcp"
    use_incoming_timestamp = false
    idle_timeout = "120s"
    label_structured_data = true
    use_rfc5424_message = true
    max_message_length = 0
    syslog_format = "rfc5424"
    labels = {
      protocol = "tcp",
      format = "rfc5424",
      port = "514",
      service_name = "syslog_rfc5424_514",
    }
  }
  relabel_rules = loki.relabel.syslog_01.rules
//  forward_to = [loki.relabel.syslog_01.receiver]
  forward_to = [loki.process.syslog_processor.receiver]


}

loki.source.syslog "syslog_receiver_udp_5514" {
  // RFC3164 listeners on port 5514
  listener {
    address = "0.0.0.0:5514"
    protocol = "udp"
    use_incoming_timestamp = false
    max_message_length = 0
    syslog_format = "rfc3164"
    labels = {
      protocol = "udp",
      format = "rfc3164",
      port = "5514",
      service_name = "syslog_rfc3164_5514",
    }
  }
  relabel_rules = loki.relabel.syslog_01.rules
//  forward_to = [loki.relabel.syslog_01.receiver]
  forward_to = [loki.process.syslog_processor.receiver]


}

loki.source.syslog "syslog_receiver_tcp_5514" {
  listener {
    address = "0.0.0.0:5514"
    protocol = "tcp"
    use_incoming_timestamp = false
    idle_timeout = "120s"
    max_message_length = 0
    syslog_format = "rfc3164"
    labels = {
      protocol = "tcp",
      format = "rfc3164",
      port = "5514",
      service_name = "syslog_rfc3164_5514",
    }
  }
  relabel_rules = loki.relabel.syslog_01.rules
//  forward_to = [loki.relabel.syslog_01.receiver]
  forward_to = [loki.process.syslog_processor.receiver]

}

loki.relabel "syslog_01" {
    rule {
      action = "labelmap"
      regex = "__syslog_(.+)"
    }

    rule {
        source_labels = ["__syslog_message_hostname"]
        target_label  = "instance"
    }

    rule {
        source_labels = ["__syslog_message_severity"]
        target_label  = "level"
    }

    rule {
        source_labels = ["__syslog_message_app_name"]
        target_label  = "application"
    }

    rule {
        source_labels = ["__syslog_message_facility"]
        target_label  = "facility"
    }

    rule {
        source_labels = ["__syslog_connection_hostname"]
        target_label  = "connection_hostname"
    }

    rule {
        source_labels = ["__syslog_message_proc_id"]
        target_label  = "proc_id"
    }

    rule {
        source_labels = ["__syslog_message_msg_id"]
        target_label  = "msg_id"
    }

    rule {
        source_labels = ["__syslog_connection_ip_address"]
        target_label  = "connection_ip_address"
    }


  forward_to = [loki.process.syslog_processor.receiver]
}



loki.process "syslog_processor" {
  forward_to = [loki.write.loki.receiver]


   // Add specific syslog fields to structured metadata
   stage.structured_metadata {
     values = {
       level                 = "__syslog_message_severity",
       severity              = "__syslog_message_severity",
       facility              = "__syslog_message_facility",
       hostname              = "__syslog_message_hostname",
       instance              = "__syslog_message_hostname",
       app_name              = "__syslog_message_app_name",
       proc_id               = "__syslog_message_proc_id",
       msg_id                = "__syslog_message_msg_id",
       connection_ip_address = "__syslog_connection_ip_address",
       connection_hostname   = "__syslog_connection_hostname",
     }
   }

   // Keep only the message content
   stage.output {
     source = "__syslog_message"
   }
}

This issue was shortly discussed in slack:
https://grafana.slack.com/archives/C01050C3D8F/p1733952739924029?thread_ts=1733928788.934089&cid=C01050C3D8F

PS:
You should update the documentation of loki.source.syslog and list all the available internal labels which can be used.
Add an example how to remove the labels and add it as structured_metadata to avoid high cardinality.

System information

Linux ubiquiti 5.10.0-33-amd64 #1 SMP Debian 5.10.226-1 (2024-10-03) x86_64 GNU/Linux

Software version

Grafana Alloy 1.5.1

Configuration

Logs

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[VorpalBlade]

It appears from https://grafana.com/docs/alloy/latest/reference/components/loki/loki.source.syslog/ that the documentation states:

All header fields from the parsed RFC5424 messages are brought in as internal labels, prefixed with _syslog.

However, the actual prefix is __syslog_message_. I don't need the labelmap thing suggested by the author. Simply doing something like this works (to map the fields I'm interested in):

loki.relabel "remote" {
  forward_to = []

  rule {
    source_labels = ["__syslog_message_severity"]
    target_label  = "severity"
  }

  rule { 
    source_labels = ["__syslog_message_hostname"]
    target_label  = "hostname"
  }
  rule {
    source_labels = ["__syslog_message_app_name"]
    target_label  = "appname"
  }
  rule {
    source_labels = ["__syslog_message_proc_id"]
    target_label  = "procid"
  }
  rule {
    source_labels = ["__syslog_message_msg_id"]
    target_label  = "msgid"
  }
}

loki.source.syslog "remote_logs" {
  listener {
    address = "0.0.0.0:514"
  }
  forward_to = [...]
  relabel_rules = loki.relabel.remote.rules
}

That said, the documentation is very confusing, especially to me as a new user for alloy and the whole grafana/loki ecosystem.

[Nachtfalkeaw]

@VorpalBlade the Bug I describe seems Not to happen all the time. The Bug/behaviour was confirmed by @dehaansa in slack.