How to properly stop and cleanup EventBroadcaster

[relyt0925]

Our service interacts with many different Kuberentes clusters and creates event broadcasters against each of the API Servers when it's performing actions and then once done expects to cleanup the broadcaster entirely to make sure there are no leaks.

import (
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/tools/record"

	"k8s.io/api/core/v1"
	"k8s.io/client-go/kubernetes/scheme"
	v1core "k8s.io/client-go/kubernetes/typed/core/v1"
)

// NewEventRecorder returns a new Event recorder
func NewEventRecorder(clientset *kubernetes.Clientset, namespace string, publisherID string) (record.EventRecorder, func(), error) {
	broadcaster := record.NewBroadcaster()
	shutdownInterface := broadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: v1core.New(clientset.CoreV1().RESTClient()).Events(namespace)})
	return broadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: publisherID}), func() { shutdownInterface.Stop() }, nil
}

What we are seeing that despite calling Stop() below the loop goroutine hangs around indefinitetly

goroutine 89892 [chan receive, 810 minutes]:
github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc015ad8440)
	/go/src/github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 +0x66
created by github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch.NewBroadcaster
	/go/src/github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xcc

goroutine 182245 [chan receive, 324 minutes]:
github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc07dbaa380)
	/go/src/github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 +0x66
created by github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch.NewBroadcaster
	/go/src/github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xcc

goroutine 88795 [chan receive, 816 minutes]:
github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc00d85f040)
	/go/src/github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 +0x66
created by github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch.NewBroadcaster
	/go/src/github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xcc

goroutine 96642 [chan receive, 774 minutes]:
github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc00e89bf00)
	/go/src/github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 +0x66
created by github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch.NewBroadcaster
	/go/src/github.ibm.com/alchemy-containers/armada-bootstrap/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xcc

It appears it might be waiting to force the events through but I believe there ultimately needs to be a way to do a complete shutdown and not wait for the events to go through or that it at least shuts down after some period of time so resource leaks don't occur

[relyt0925]

Almost effectively looking for a ForceShutdown in the case where events aren't critical to be delivered and instead are just best effort.

func (m *Broadcaster) Shutdown() {

func (m *Broadcaster) ForceShutdown() {

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mattmoor]

@relyt0925 I'm curious if you got an answer to this? I was just battling something along these lines myself.