Cannot query logs from eventrouter with {name="eventrouter"}

[minhdanh]

I followed this blog post: https://grafana.com/blog/2019/08/21/how-grafana-labs-effectively-pairs-loki-and-kubernetes-events/
and installed eventrouter to the cluster (using the official helm chart) (to namespace loki).

But now when I query loki using {name="eventrouter"}, loki is returning empty results. Although I can see the logs from the eventrouter pods:

I0826 10:58:53.717599       6 glogsink.go:42] {"verb":"ADDED","event":{"metadata":{"name":"sentry-cron-6d7699fccd.15be73535642adba","namespace":"sentry","selfLink":"/api/v1/namespaces/sentry/events/sentry-cron-6d7699fccd.15be73535642adba","uid":"11cf530c-c7ef-11e9-aaf1-42010aa40066","resourceVersion":"758942","creationTimestamp":"2019-08-26T10:48:42Z"},"involvedObject":{"kind":"ReplicaSet","namespace":"sentry","name":"sentry-cron-6d7699fccd","uid":"008cf918-c4df-11e9-aaf1-42010aa40066","apiVersion":"apps/v1","resourceVersion":"29283987"},"reason":"SuccessfulCreate","message":"Created pod: sentry-cron-6d7699fccd-56zvr","source":{"component":"replicaset-controller"},"firstTimestamp":"2019-08-26T10:48:42Z","lastTimestamp":"2019-08-26T10:48:42Z","count":1,"type":"Normal"}}

I also tried to deploy eventrouter to kube-system namespace but that didn't change anything.
Not sure what I'm doing wrong here.

[minhdanh]

@gouthamve Do you have any ideas?

[minhdanh]

I changed the sink type to stdout, and could query the events using {app="eventrouter"}.
But the json logs looks not parsed correctly by Loki. The whole json line generated by eventrouter is not parsed and considered as a long string:

[nv30]

I changed the sink type to stdout, and could query the events using {app="eventrouter"}.
But the json logs looks not parsed correctly by Loki. The whole json line generated by eventrouter is not parsed and considered as a long string:

Hi! Did you somehow manage to fix it?

[yfried]

Hi @minhdanh, did you manage to fix this? I'm having the same problem

[unguiculus]

It's normal that you get the JSON displayed like this. You can use a parser expression to extract labels.

See https://grafana.com/docs/loki/latest/logql/#parser-expression

Check out my demo which includes the eventrouter and also a dashboard for it.

https://github.com/unguiculus/loki-minio-demo

[UrosCvijan]

Hi,
Did you do anytihng special so you get only json in the loki log? As for some reason I get some timestamp in front of json and cant to parsing.. How to remove this first part from the log:

I just followed the sample code, used stdout as well, so nothing special. And if I do kubectl get logs form the eventrouter I do get nice log, starting with {"verb": etc...

Is it something that I can change in loki config maybe? Thanks in advance

[hangrum]

I hope this helpful.

{app="eventrouter"} | regexp "(?P<json>{.*)" | line_format "{{.json}}"|json |line_format "NAMESPACE: {{ .event_involvedObject_namespace }} KIND: {{ .event_involvedObject_kind }} MSG: {{ .event_message }}"

[UrosCvijan]

Thanks. I finally used promtail pipeline stage to do replace, to remove the first part of the log:

    pipelineStages:
      - cri: {}
      - replace:
          expression: '^\d{1,2}:\d{1,2}:\d{1,2} \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{9}Z (stdout|stderr) F'
          replace: ""

This left only the good log, which can later I passed through json parser (i think, as what they asked me to do later they said ,we don't need it), so with that i could easily search and filter per field

[Jyoti492]

@minhdanh I am facing the issue for scraping the kubernetes events logs to Grafana loki. I have eventrouter installed along with its logs shows all kubernetes events but apprantely nothing shown when I query grafana loki. Can you please help me out?