Need help with expressions.

[benisai]

Hi, I would love to use this, but I dont know how to setup the expression via json.

Here is an example of my json output (json is written to file per line)
{"ct_id":0,"ct_mark":0,"detected_application":303,"detected_application_name":"126.netify.google","detected_protocol":188,"detected_protocol_name":"QUIC","detection_guessed":1,"dhc_hit":true,"digest":"99b10c8d5fa0dd3fdd17abe1bfab5b82bad037f2","first_seen_at":1674703379228,"first_update_at":1674703379228,"host_server_name":"clients2.google.com","ip_nat":false,"ip_protocol":17,"ip_version":4,"last_seen_at":1674703379269,"local_ip":"10.0.5.143","local_mac":"c8:2a:dd:88:de:e3","local_origin":true,"local_port":40180,"other_ip":"142.250.176.14","other_mac":"20:6d:31:11:10:d2","other_port":443,"other_type":"remote","vlan_id":0,"interface":"br-lan","internal":true,"type":"flow"}

Here is my promtail.yml config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://lokimiddleware:3100/loki/api/v1/push

- job_name: netify-logs
  static_configs:
  - targets:
      - localhost
    labels:
      job: netifylogs
      __path__: /var/log/netify/*log
    pipeline_stages:
      - json:
          expressions:
            detected_protocol_name: detected_protocol_name
            host_server_name: host_server_name
            local_ip: local_ip
            local_mac: local_mac
            local_port: local_port
            other_ip: local_ip
            other_ip: other_ip
            other_mac: other_mac
            other_port: other_port
      - labels:
          host_server_name:
          local_ip:
          local_mac:
          local_port:
          other_ip:
          other_mac:
          other_port:
      - template:
          source: output_msg
          template: 'detected_protocol_name="{{ .detected_protocol_name }}" local_ip="{{ .local_ip }}" local_mac="{{ .local_mac }}"  local_port="{{ .local_port }}"  GeoIP_Source="{{ .other_ip }}" other_mac="{{ .other_mac }}" other_port="{{ .other_port }}"'
      - output:
          source: output_msg

[benisai]

The container logs for promtail show this error:

Unable to parse config: /etc/promtail/promtail-config.yaml: yaml: unmarshal errors:
field pipeline_stages not found in type struct { Targets []string "yaml:"targets""; Labels model.LabelSet "yaml:"labels"" }

[Nihiue]

To use this middleware, just include GeoIP_Source=XX in output template, and your template is fine.

The real issue is that the indent level of pipeline_stages field is wrong, so promtail could not parse it.

Please look into
https://grafana.com/docs/loki/latest/clients/promtail/pipelines/
https://spacelift.io/blog/yaml

[benisai]

Thanks for the info. I think I've formatted my yaml correct so I can see stuff importing into loki. I see GeoIP_source, but It doesnt seem to be working.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://lokimiddleware:3100/loki/api/v1/push


scrape_configs:
  - job_name: netify
    static_configs:
    - targets:
        - localhost
      labels:
        job: netifylogs
        __path__: /var/log/netify/*log
    pipeline_stages:
      - json:
          expressions:
            detected_protocol_name: detected_protocol_name
            host_server_name: host_server_name
            local_ip: local_ip
            local_mac: local_mac
            local_port: local_port
            other_ip: other_ip
            other_mac: other_mac
            other_port: other_port
      - labels:
          host_server_name:
          local_ip:
          local_mac:
          local_port:
          other_ip:
          other_mac:
          other_port:
      - template:
          source: output_msg
          template: 'detected_protocol_name:"{{ .detected_protocol_name }}" local_ip:"{{ .local_ip }}" local_mac:"{{ .local_mac }}" local_port:"{{ .local_port }}"  GeoIP_Source:"{{.other_ip }}" other_mac:"{{ .other_mac }}" other_port:"{{ .other_port }}"' 

      - output:
          source: output_msg

Middleware is giving this error, which I guess makes sense as it's not a public ip

error: target reject {"name":"server"}

Here is what gets imported into loki:

2023-02-02 21:31:00 | detected_protocol_name:"MDNS" local_ip:"10.0.5.170" local_mac:"78:64:c0:34:7a:ee" local_port:"5353" GeoIP_Source:"224.0.0.251" other_mac:"01:00:5e:00:00:fb" other_port:"5353"

[benisai]

ok, so I can see the public IP in loki and I see the GeoIP, but its not replacing with the location/city. Can you send me a snippet of your log file?

[Nihiue]

Try to use = instead of : in the template

[benisai]

it works!!!! I needed the = instead of : in my template.

[benisai]

Do you have a grafana dashboard that I can use as an example?

[benisai]

Does the data fields need to match naming conventions that geomap is looking for? I can't make worldmap or geomap work :)

From Grafana:
The Geomap panel needs a source of geographical data. This data comes from a database query, and there are four mapping options for your data.

Auto automatically searches for location data. Use this option when your query is based on one of the following names for data fields.
geohash: “geohash”
latitude: “latitude”, “lat”
longitude: “longitude”, “lng”, “lon”
lookup: “lookup”