Improve Kubernetes events to be CloudEvents 1.0 compatible

[tomkerkhove]

As of today, Kubernetes already provides extension points by emitting events by different SIGs - This is great!

However, there are a few hurdles:

• The event contract is specific to Kubernetes which makes it harder to integrate with
• Events do not have a unique subjects which consumers can use to subscribe to and have to figure it out themselves
• There is poor documentation on what event is being emitted when

What would you like to be added: Now that CloudEvents has reached the 1.0 milestone it would be good if Kubernetes could start emitting events in that format.

As part of that transition, I would love to use this change to build a centralized list with all the events and their subjects to make it easier to consume htem.

Why is this needed: Emitting events according to the CloudEvent specification would make it easier for external tools and vendors to plug in and extend Kubernetes where they have to.

/cc @clemensv

[tomkerkhove]

This one would be for @kubernetes/sig-instrumentation-feature-requests I think

[k8s-ci-robot]

@tomkerkhove: Reiterating the mentions to trigger a notification:
@kubernetes/sig-instrumentation-feature-requests

In response to this:

This one would be for @kubernetes/sig-instrumentation-feature-requests I think

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[brancz]

This is fairly vague what this would mean in terms of changes. I would suggest that you join the sig instrumentation meetings and we start discussing this there. If there is some amount of interest, then I'd say we flesh out a KEP and understand what this would really mean for the project. The existing events API is replied upon by a lot of people so we can't just go and break it, it's literally an API with stability guarantees.

[tomkerkhove]

I'm happy to join the bi-weekly call but the https://github.com/kubernetes/community/tree/master/sig-instrumentation points me to a temporary page due to a move. When is the next one?

The current approach is a good start but there are a few things that can be improved:

• Custom format that we have to parse which makes it harder to integrate with
• No type field which uniquely identify the nature of the event, think DeploymentCreated, DeploymentScaled, etc. These allow us to subscribe to only one event type rather than having to subscribe to them all and see if it's interesting for us. (filtering would be nice to have in Kube, but doesn't have to be the case as other systems like Azure Event Grid can do that based on CloudEvents)

Kubernetes Event Sample

{
    "action": null,
    "apiVersion": "v1",
    "count": 1,
    "eventTime": null,
    "firstTimestamp": "2019-11-25T09:34:16Z",
    "involvedObject": {
        "apiVersion": "v1",
        "fieldPath": null,
        "kind": "Pod",
        "name": "order-processor-85ccb86f6-qfccj",
        "namespace": "k8s-issue",
        "resourceVersion": "165777",
        "uid": "bf93d0c6-0f66-11ea-b446-00155d83711f"
    },
    "kind": "Event",
    "lastTimestamp": "2019-11-25T09:34:16Z",
    "message": "Successfully assigned k8s-issue/order-processor-85ccb86f6-qfccj to docker-desktop",
    "metadata": {
        "annotations": null,
        "clusterName": null,
        "creationTimestamp": "2019-11-25T09:34:16Z",
        "deletionGracePeriodSeconds": null,
        "deletionTimestamp": null,
        "finalizers": null,
        "generateName": null,
        "generation": null,
        "initializers": null,
        "labels": null,
        "managedFields": null,
        "name": "order-processor-85ccb86f6-qfccj.15da5e1339c58574",
        "namespace": "k8s-issue",
        "ownerReferences": null,
        "resourceVersion": "165783",
        "selfLink": "/api/v1/namespaces/k8s-issue/events/order-processor-85ccb86f6-qfccj.15da5e1339c58574",
        "uid": "bf9aaf5c-0f66-11ea-b446-00155d83711f"
    },
    "reason": "Scheduled",
    "related": null,
    "reportingComponent": "",
    "reportingInstance": "",
    "series": null,
    "source": {
        "component": "default-scheduler",
        "host": null
    },
    "type": "Normal"
}

Just to be clear - I am not saying the current approach sucks, I'm just suggesting to use a CNCF-supported standard instead, please don't get me wrong!

I fully understand that there are a lot of customers on the current eventing API and I'm not suggesting to just kill this but merely have them side-by-side or provide a migration to CloudEvents.

[duglin]

Please don't forget about "binary" vs "structured" CloudEvents. CloudEvents supports the idea of not changing an existing event/message to become a CloudEvent. You just add a few extra bits of metadata (e.g. http headers). You could also use the CloudEvents defined structure too if you want, but it's not required.

[tomkerkhove]

Oh that's right - So if we add the additional (HTTP) headers we can get the metadata without changing the effective event payload we have today - Is that correct?

[duglin]

correct - as long as the receiver can handle "binary" CloudEvents

[tomkerkhove]

Here's the spec on that - HTTP Protocol Binding for CloudEvents - Version 1.0

That would be the perfect middle ground I think!

[RainbowMango]

I'm happy to join the bi-weekly call but the https://github.com/kubernetes/community/tree/master/sig-instrumentation points me to a temporary page due to a move. When is the next one?

Next meeting on Nov 28th. Welcome.
Join the google group to get auto-invited to meetings and google docs: https://groups.google.com/forum/#!forum/kubernetes-sig-instrumentation

Meeting notes: https://docs.google.com/document/d/17emKiwJeqfrCsv0NZ2FtyDbenXGtTNCsDEiLbPa7x7Y

[tomkerkhove]

I'll try to join day after tomorrow, or is it cancelled due to Thanksgiving?

[ehashman]

I pinged @piosz to confirm; both SIG chairs aren't based in the US but much of the SIG is and @brancz is currently out on vacation. I don't see any agenda items so I suspect it will be cancelled.

[tomkerkhove]

Thanks for checking - I'll join but if nobody shows up join later on instead.

[rektide]

some discussion in this thread about structured logging- just want to create a link here to kubernetes/enhancements#1367 , which is an enhancement for structured logging.

[brancz]

I think we're on the same page.

I fully understand that there are a lot of customers on the current eventing API and I'm not suggesting to just kill this but merely have them side-by-side or provide a migration to CloudEvents.

This is the possibility I see for something like this as well. It would need to be fleshed out in detail though.

[tomkerkhove]

Awesome, happy to help where I can.

[tomkerkhove]

I've updated the title to indicate that it's more about compatibility rather than change/impact current approach drastically.

[tomkerkhove]

Adding @kubernetes/sig-api-machinery-feature-requests as discussed in the SIG instrumentation standup.

I will provide a sample based on Kubernetes event in next couple of days.

[k8s-ci-robot]

@tomkerkhove: Reiterating the mentions to trigger a notification:
@kubernetes/sig-api-machinery-feature-requests

In response to this:

Adding @kubernetes/sig-api-machinery-feature-requests as discussed in the SIG instrumentation standup.

I will provide a sample based on Kubernetes event in next couple of days.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tomkerkhove]

I've had a quick sync with @duglin who would be able to give an introduction to CloudEvents (@clemensv might also join). Given the fact that the timing of the standup on 9th is not ideal and I don't want to hijack the whole meeting it might be good to do it in a separate call?

[brancz]

There are people involved in sig instrumentation spread all over the world, it's never ideal for everyone, I'd prefer not to deviate from normal meetings. As this isn't too pressing, what do you think about dedicating the meeting after that to this topic?

[tomkerkhove]

That would be 23rd of January then? Fine for me, does that work for you @duglin?

[brancz]

Yes exactly the 23rd of January. If that works for you feel free to put it into the meeting notes already.

[tomkerkhove]

Fine for me, will add if @duglin agrees as well.

[tomkerkhove]

This won't work I guess, but can only try:

/remove-lifecycle stale

Eventually we'll get there, just need to have time to work out a POC.

[duglin]

https://www.youtube.com/watch?v=9uKm5pD7GWM :-)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tomkerkhove]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ehashman]

@tomkerkhove this needs a KEP, it can't be implemented in the scope of an issue so we can ensure this gets the necessarily visibility.

[tomkerkhove]

Please ignore my ignorance but how does one write a KEP and do I need to understand the Kubernetes codebase?

[dims]

@tomkerkhove please see https://github.com/kubernetes/enhancements#table-of-contents

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[tomkerkhove]

/remove-lifecycle rotten

I'll write a KEP at some point

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[tomkerkhove]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[tomkerkhove]

/remove-lifecycle stale

[serathius]

/lifecycle frozen
That should help

[tomkerkhove]

Thank you!