Listening for custom resource events

[toddkazakov]

It's not obvious from the code why the watcher isn't getting events about custom resources, but it doesn't look like it's a configuration issue. Could you please clarify if that's the expected behaviour? Here's an example event:

Name:             thanos.16118765932ddc8c                                                                                                                                                                                                                                             
Namespace:        monitoring                                                                                                                                                                                                                                                          
Labels:           <none>                                                                                                                                                                                                                                                              
Annotations:      <none>                                                                                                                                                                                                                                                              
API Version:      v1                                                                                                                                                                                                                                                                  
Count:            177                                                                                                                                                                                                                                                                 
Event Time:       <nil>                                                                                                                                                                                                                                                               
First Timestamp:  2020-05-23T02:30:13Z                                                                                                                                                                                                                                                
Involved Object:                                                                                                                                                                                                                                                                      
  API Version:       helm.fluxcd.io/v1                                                                                                                                                                                                                                                
  Kind:              HelmRelease                                                                                                                                                                                                                                                      
  Name:              thanos                                                                                                                                                                                                                                                           
  Namespace:         monitoring                                                                                                                                                                                                                                                       
  Resource Version:  111701013                                                                                                                                                                                                                                                        
  UID:               75ed7978-fd31-4601-810a-2ef0b3ea3e39                                                                                                                                                                                                                             
Kind:                Event                                                                                                                                                                                                                                                            
Last Timestamp:      2020-05-23T11:18:10Z                                                                                                                                                                                                                                             
Message:             managed release 'thanos' in namespace 'monitoring' sychronized                                                                                                                                                                                                   
Metadata:                                                                                                                                                                                                                                                                             
  Creation Timestamp:  2020-05-23T02:30:13Z                                                                                                                                                                                                                                           
  Resource Version:    111953392                                                                                                                                                                                                                                                      
  Self Link:           /api/v1/namespaces/monitoring/events/thanos.16118765932ddc8c                                                                                                                                                                                                   
  UID:                 9af8aa5f-f678-48cb-995d-81ad3334bacd                                                                                                                                                                                                                           
Reason:                ReleaseSynced                                                                                                                                                                                                                                                  
Reporting Component:                                                                                                                                                                                                                                                                  
Reporting Instance:                                                                                                                                                                                                                                                                   
Source:                                                                                                                                                                                                                                                                               
  Component:  helm-operator                                                                                                                                                                                                                                                           
Type:         Normal                                                                                                                                                                                                                                                                  
Events:       <none>

Other standard events are successfully captured.

[mustafaakin]

It should work, it should just not be able to read the additional labels ore metadata of them since they are not available in default view role. I need to reproduce it, I might have overlooked it before because when I run it from my IDE, it always runs with the administrator role.

[toddkazakov]

That was my understanding as well. Alas, it doesn't work, at least not with the roles in the deploy directory.

[mustafaakin]

I tested it with installing Tekton to my Minikube. It gets dumped, I'm not really sure what is going on.

Here are the test logs:

2020-06-22T15:02:05Z DBG app/pkg/exporter/channel_registry.go:56 > sending event to sink event="pod status \"Ready\":\"False\"; message: \"containers with unready status: [step-test-me]\"" sink=dump
2020-06-22T15:02:08Z DBG app/pkg/kube/watcher.go:63 > Received event msg="Successfully pulled image \"alpine\"" namespace=default object=&ObjectReference{Kind:Pod,Namespace:default,Name:test-plz-pod-7xzr2,UID:9c585164-c06e-4490-b3e6-46b46b474d85,APIVersion:v1,ResourceVersion:10467,FieldPath:spec.containers{step-test-me},} reason=Pulled
2020-06-22T15:02:08Z DBG app/pkg/exporter/channel_registry.go:56 > sending event to sink event="Successfully pulled image \"alpine\"" sink=dump
2020-06-22T15:02:08Z DBG app/pkg/kube/watcher.go:63 > Received event msg="Created container step-test-me" namespace=default object=&ObjectReference{Kind:Pod,Namespace:default,Name:test-plz-pod-7xzr2,UID:9c585164-c06e-4490-b3e6-46b46b474d85,APIVersion:v1,ResourceVersion:10467,FieldPath:spec.containers{step-test-me},} reason=Created
2020-06-22T15:02:08Z DBG app/pkg/exporter/channel_registry.go:56 > sending event to sink event="Created container step-test-me" sink=dump
2020-06-22T15:02:08Z DBG app/pkg/kube/watcher.go:63 > Received event msg="Started container step-test-me" namespace=default object=&ObjectReference{Kind:Pod,Namespace:default,Name:test-plz-pod-7xzr2,UID:9c585164-c06e-4490-b3e6-46b46b474d85,APIVersion:v1,ResourceVersion:10467,FieldPath:spec.containers{step-test-me},} reason=Started
2020-06-22T15:02:08Z DBG app/pkg/exporter/channel_registry.go:56 > sending event to sink event="Started container step-test-me" sink=dump
2020-06-22T15:02:08Z DBG app/pkg/kube/watcher.go:63 > Received event msg="Not all Steps in the Task have finished executing" namespace=default object=&ObjectReference{Kind:TaskRun,Namespace:default,Name:test-plz,UID:99df443b-dc85-415f-83a5-fa5ab45a23b0,APIVersion:tekton.dev/v1beta1,ResourceVersion:10485,FieldPath:,} reason=Running
2020-06-22T15:02:08Z DBG app/pkg/exporter/channel_registry.go:56 > sending event to sink event="Not all Steps in the Task have finished executing" sink=dump
2020-06-22T15:02:08Z DBG app/pkg/kube/watcher.go:63 > Received event msg="Not all Steps in the Task have finished executing" namespace=default object=&ObjectReference{Kind:TaskRun,Namespace:default,Name:test-plz,UID:99df443b-dc85-415f-83a5-fa5ab45a23b0,APIVersion:tekton.dev/v1beta1,ResourceVersion:10485,FieldPath:,} reason=Running
2020-06-22T15:02:08Z DBG app/pkg/exporter/channel_registry.go:56 > sending event to sink event="Not all Steps in the Task have finished executing" sink=dump
2020-06-22T15:02:09Z DBG app/pkg/kube/watcher.go:63 > Received event msg="All Steps have completed executing" namespace=default object=&ObjectReference{Kind:TaskRun,Namespace:default,Name:test-plz,UID:99df443b-dc85-415f-83a5-fa5ab45a23b0,APIVersion:tekton.dev/v1beta1,ResourceVersion:10497,FieldPath:,} reason=Succeeded
2020-06-22T15:02:09Z DBG app/pkg/exporter/channel_registry.go:56 > sending event to sink event="All Steps have completed executing" sink=dump

I also logged the involved object's name, I don't think it's a rights issue. Which type of Kubernetes cluster you run?

[mustafaakin]

I can also confirm logging to stdout also worked in Minikube with that role:

{
  "metadata": {
    "name": "test-plz.161ae67430de99d3",
    "namespace": "default",
    "selfLink": "/api/v1/namespaces/default/events/test-plz.161ae67430de99d3",
    "uid": "227a8fdc-2966-4173-a796-30ca967fd864",
    "resourceVersion": "12242",
    "creationTimestamp": "2020-06-22T15:13:24Z",
    "managedFields": [
      {
        "manager": "controller",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2020-06-22T15:13:24Z"
      }
    ]
  },
  "reason": "Succeeded",
  "message": "All Steps have completed executing",
  "source": {
    "component": "taskrun-controller"
  },
  "firstTimestamp": "2020-06-22T15:13:24Z",
  "lastTimestamp": "2020-06-22T15:13:24Z",
  "count": 1,
  "type": "Normal",
  "eventTime": null,
  "reportingComponent": "",
  "reportingInstance": "",
  "involvedObject": {
    "kind": "TaskRun",
    "namespace": "default",
    "name": "test-plz",
    "uid": "e27d9b52-b1f5-4e43-b91c-51fedf1c9544",
    "apiVersion": "tekton.dev/v1beta1",
    "resourceVersion": "12239",
    "labels": {
      "app.kubernetes.io/managed-by": "tekton-pipelines"
    }
  }
}

With this config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: monitoring
data:
  config.yaml: |
    logLevel: error
    logFormat: json
    route:
      routes:
        - match:
            - receiver: "dump"
    receivers:
      - name: "dump"
        file:
          path: "/dev/stdout"

[Nanduyana]

i have a similar issue with the default config from deploy directory

except the change in the elastic search url

apiVersion: v1
kind: ConfigMap
metadata:
name: event-exporter-cfg
namespace: {{ .Release.Namespace }}
data:
config.yaml: |
logLevel: debug
route:
match:
- receiver: dump
receivers:
- name: "dump"
elasticsearch:
hosts:
- http://elasticsearch-master.elasticsearch.svc:9200
index: kube-events-ns

with a custom pod running in another namespace , the corresponding events for those pods are not pushed to ES

below is the repo i used without any changes (except for the elasticsearch URL)

https://github.com/opsgenie/kubernetes-event-exporter/tree/deployment-and-readme-fix/deploy