Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addon: Stream all Kubernetes events to a topic #60

Closed
wants to merge 17 commits into from

Conversation

solsson
Copy link
Contributor

@solsson solsson commented Aug 5, 2017

A quite interesting log with messages like (kubectl -n test-kafka logs -l test-target=events-topic):

{
  "type": "MODIFIED",
  "object": {
    "kind": "Event",
    "apiVersion": "v1",
    "metadata": {
      "name": "pzoo-0.14d7585c3d972d2d",
      "namespace": "kafka",
      "selfLink": "/api/v1/namespaces/kafka/events/pzoo-0.14d7585c3d972d2d",
      "uid": "e0ed084c-784d-11e7-a1b0-42010a840078",
      "resourceVersion": "108989",
      "creationTimestamp": "2017-08-03T13:15:53Z"
    },
    "involvedObject": {
      "kind": "Pod",
      "namespace": "kafka",
      "name": "pzoo-0",
      "uid": "c4d1c2e6-784d-11e7-a1b0-42010a840078",
      "apiVersion": "v1",
      "resourceVersion": "469107",
      "fieldPath": "spec.containers{zookeeper}"
    },
    "reason": "Unhealthy",
    "message": "Readiness probe failed: ",
    "source": {
      "component": "kubelet",
      "host": "gke-eu-west-3-default-pool-f275612a-f7m1"
    },
    "firstTimestamp": "2017-08-03T13:15:53Z",
    "lastTimestamp": "2017-08-05T04:54:23Z",
    "count": 592,
    "type": "Warning"
  }
}

(#31 should take note about this particular event)

Reopens #39.

Includes RBAC policy from #59

@solsson solsson added the addon label Aug 5, 2017
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
	at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:145)
	at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:99)
	at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:59)
	at org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:82)
	... 40 more
solsson added a commit that referenced this pull request Aug 5, 2017
at least for now, as it allows exec into the pods to investigate.
We've been having frequent restarts that are not due to OOMKilled (i.e. not #49).
Now failed probes will lead to unready pods, which we can monitor for using #60.
and kafkacat will produce each line separately
we don't send error messages to the topic
(even if it's json -- format is different and it's pretty printed)
to see the error message (like missing RBAC) you basically have to run the curl command.

I'd like startup to be as fast as possible, to not miss events, so I hesitate to add checks.
@solsson
Copy link
Contributor Author

solsson commented Oct 26, 2017

Tried to improve error handling, for example when RBAC is missing, but with no real success. You basically need to exec to the pod (which is most likely crashlooping fast) to try the curl manually. Status noted in fc5a836.

TODO test if the curl error (with -f) gets into the topic, thus breaking consumers that assume they will get JSON. Status on that is in d628275.

Silly change, but 0.8 days is very short.

Google helped me verify the math.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants