Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Addon: Stream all Kubernetes events to a topic #60

Closed
wants to merge 17 commits into from
Closed

Addon: Stream all Kubernetes events to a topic #60

wants to merge 17 commits into from

Commits on Aug 5, 2017

  1. Creates a pod for expermentation with kubectl and kafkacat

    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    30b888a View commit details
    Browse the repository at this point in the history

  2. Produces and reads events

    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    4bdfb2b View commit details
    Browse the repository at this point in the history

  3. wip

    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    ccdd8be View commit details
    Browse the repository at this point in the history

  4. Quite interesting way to test a command from yaml - see it in logs

    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    5ee97a2 View commit details
    Browse the repository at this point in the history

  5. Produces proper json. Let's see when curl gives up and the container … 

    …restarts.
    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    4da60f8 View commit details
    Browse the repository at this point in the history

  6. Testing compression but solsson/kafka:0.11.0.0 throws ... 

    Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
	at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:145)
	at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:99)
	at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:59)
	at org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:82)
	... 40 more
    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    efacc18 View commit details
    Browse the repository at this point in the history

  7. Works with my current kafka image, and the test consumer still sees JSON

    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    b6575a5 View commit details
    Browse the repository at this point in the history

  8. Latest build suppors snappy compression

    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    a30f22e View commit details
    Browse the repository at this point in the history

  9. Adds RBAC policy for curl, default service account

    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    f626a77 View commit details
    Browse the repository at this point in the history

  10. Moves the example comsumer to the test namespace

    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    4f13cf5 View commit details
    Browse the repository at this point in the history

  11. Not a proper test yet, but the PR documents how to use it to see events

    @solsson
    solsson committed Aug 5, 2017
    Configuration menu
    Copy the full SHA
    8935cf5 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2017

  1. Use an image with new kafkacat and curl versions

    @solsson
    solsson committed Oct 26, 2017
    Configuration menu
    Copy the full SHA
    8d0d993 View commit details
    Browse the repository at this point in the history

  2. Makes things a bit less obscure if for example RBAC fails

    @solsson
    solsson committed Oct 26, 2017
    Configuration menu
    Copy the full SHA
    b9e7b8a View commit details
    Browse the repository at this point in the history

  3. Argh, curl will pretty print the json ... 

    and kafkacat will produce each line separately
    @solsson
    solsson committed Oct 26, 2017
    Configuration menu
    Copy the full SHA
    e4c5179 View commit details
    Browse the repository at this point in the history

  4. bash -e is important, in combination with curl -f, so ... 

    we don't send error messages to the topic
(even if it's json -- format is different and it's pretty printed)
    @solsson
    solsson committed Oct 26, 2017
    Configuration menu
    Copy the full SHA
    d628275 View commit details
    Browse the repository at this point in the history

  5. kafkacat's debug output solves the silence, but ... 

    to see the error message (like missing RBAC) you basically have to run the curl command.

I'd like startup to be as fast as possible, to not miss events, so I hesitate to add checks.
    @solsson
    solsson committed Oct 26, 2017
    Configuration menu
    Copy the full SHA
    fc5a836 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2017

  1. Fix retention from 0.8 days to 8 days 

    Silly change, but 0.8 days is very short.

Google helped me verify the math.
    @atamon @solsson
    atamon authored and solsson committed Oct 30, 2017
    Configuration menu
    Copy the full SHA
    7d4d30d View commit details
    Browse the repository at this point in the history