Lock-free broadcaster #91602

sxllwx · 2020-05-30T10:49:48Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

lock-free broadcaster, use chan to ensure thread safety.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Hello the maintainer of kubernetes. When I read the implementation of the broadcaster, I found this comment // TODO: see if this lock is needed now that new watchers go through the incoming channel.. I think that the current implementation of the Broadcaster requires this mutex.
One of the reasons is that although the chan named "incoming" was passed when the watcher was added, the channel was not passed when the watcher was closed. This may cause a critical condition.
Pull Request implementation is to close the watcher and the broadcaster through the "incoming" chan, so the use of the mutex can be safely deleted. When I benchmarked, I found that the performance of sending messages to the broadcaster has improved to some extent.
The following are my test results:

current impl

$ go test -test.bench=".*" -count=16                                   
goos: linux
goarch: amd64
pkg: k8s.io/apimachinery/pkg/watch
BenchmarkBroadCaster-8   	13309177	        87.9 ns/op
BenchmarkBroadCaster-8   	13411250	        91.6 ns/op
BenchmarkBroadCaster-8   	13588131	        87.9 ns/op
BenchmarkBroadCaster-8   	13501332	        88.0 ns/op
BenchmarkBroadCaster-8   	13539446	        88.3 ns/op
BenchmarkBroadCaster-8   	13581561	        88.5 ns/op
BenchmarkBroadCaster-8   	13566960	        87.4 ns/op
BenchmarkBroadCaster-8   	13513597	        87.4 ns/op
BenchmarkBroadCaster-8   	13460937	        87.7 ns/op
BenchmarkBroadCaster-8   	13634308	        87.5 ns/op
BenchmarkBroadCaster-8   	13473927	        87.9 ns/op
BenchmarkBroadCaster-8   	13501562	        88.1 ns/op
BenchmarkBroadCaster-8   	13294520	        88.1 ns/op
BenchmarkBroadCaster-8   	13609996	        89.1 ns/op
BenchmarkBroadCaster-8   	13558424	        87.8 ns/op
BenchmarkBroadCaster-8   	13607326	        91.1 ns/op
PASS
ok  	k8s.io/apimachinery/pkg/watch	20.567s

pull request impl

$ go test -test.bench=".*" -count=16                                   
goos: linux
goarch: amd64
pkg: k8s.io/apimachinery/pkg/watch
BenchmarkBroadCaster-8   	15557988	        75.8 ns/op
BenchmarkBroadCaster-8   	15815998	        75.9 ns/op
BenchmarkBroadCaster-8   	15683714	        76.1 ns/op
BenchmarkBroadCaster-8   	15733852	        76.4 ns/op
BenchmarkBroadCaster-8   	15718236	        75.8 ns/op
BenchmarkBroadCaster-8   	15794850	        75.7 ns/op
BenchmarkBroadCaster-8   	15728948	        76.1 ns/op
BenchmarkBroadCaster-8   	15745780	        75.9 ns/op
BenchmarkBroadCaster-8   	15789714	        75.7 ns/op
BenchmarkBroadCaster-8   	15730546	        78.4 ns/op
BenchmarkBroadCaster-8   	15748945	        76.0 ns/op
BenchmarkBroadCaster-8   	15606589	        78.1 ns/op
BenchmarkBroadCaster-8   	15013153	        79.9 ns/op
BenchmarkBroadCaster-8   	14925436	        79.9 ns/op
BenchmarkBroadCaster-8   	14981156	        79.7 ns/op
BenchmarkBroadCaster-8   	15015866	        79.9 ns/op
PASS
ok  	k8s.io/apimachinery/pkg/watch	20.462s

Thank you for reading ~

Does this PR introduce a user-facing change?:

None

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2020-05-30T10:49:51Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2020-05-30T10:49:55Z

Welcome @sxllwx!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2020-05-30T10:49:56Z

Hi @sxllwx. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sxllwx · 2020-05-30T11:12:15Z

/check-cla

sxllwx · 2020-05-30T11:22:15Z

/assign @deads2k

sxllwx · 2020-06-02T09:03:08Z

/retest

sxllwx · 2020-06-02T09:48:02Z

/test pull-kubernetes-e2e-kind

deads2k · 2020-06-16T13:34:42Z

staging/src/k8s.io/apimachinery/pkg/watch/mux.go

@@ -196,7 +193,10 @@ func (m *Broadcaster) Action(action EventType, obj runtime.Object) {
 // have received the data yet as it can remain sitting in the buffered
 // channel.
 func (m *Broadcaster) Shutdown() {
-	close(m.incoming)
+	m.blockQueue(func() {


this change takes the serial close and then wait and makes the the close async, not guaranteed before the wait.

this change takes the serial close and then wait and makes the the close async, not guaranteed before the wait.

Reading through, I think the change still works.

deads2k · 2020-06-16T13:40:27Z

staging/src/k8s.io/apimachinery/pkg/watch/mux.go

-func (b *Broadcaster) blockQueue(f func()) {
+func (m *Broadcaster) blockQueue(f func()) {
+	select {
+	case <-m.stopped:


previously, sending after shutdown would result in a write to a closed channel, right? Can you make a unit test to prove that it used to panic?

blockQueue is called by Watch, WatchWithPrefix, stopWatching, Shutdown. The Action method sends data directly to incoming chan instead of calling blockQueue. There is a subtle bug here. If the Broadcaster has been closed, calling Watch will get a watch.Interface with a value of nil instead of Panic as before. If I call panic here, calling Watcher's Stop method after closing the Broadcaster will cause Panic. In order to maintain the consistency of the interface, in the Watch && WatchWithPrefix method, determine whether broadcasterWatcher is nil to decide whether to call panic.

deads2k · 2020-06-16T13:42:09Z

staging/src/k8s.io/apimachinery/pkg/watch/mux.go

@@ -96,10 +94,15 @@ func (obj functionFakeRuntimeObject) DeepCopyObject() runtime.Object {
 // The purpose of this terrible hack is so that watchers added after an event
 // won't ever see that event, and will always see any event after they are
 // added.


update the comment on the Watch and WatchforPrefix, to indicate that it blocks until the watch is accepted.

deads2k · 2020-06-16T13:42:45Z

minor comments, there is a clear benchmark benefit here.

deads2k · 2020-06-16T13:42:51Z

/approve

deads2k · 2020-06-22T13:25:43Z

staging/src/k8s.io/apimachinery/pkg/watch/mux.go

@@ -127,18 +129,20 @@ func (m *Broadcaster) Watch() Interface {
 		}
 		m.watchers[id] = w
 	})
+	if w == nil {


Add a comment saying this matches previous behavior, but that we're open to reassessing in the future.

deads2k · 2020-06-22T13:25:47Z

staging/src/k8s.io/apimachinery/pkg/watch/mux.go

@@ -156,26 +160,27 @@ func (m *Broadcaster) WatchWithPrefix(queuedEvents []Event) Interface {
 			w.result <- e
 		}
 	})
+	if w == nil {
+		panic("broadcaster already stopped")


Add a comment saying this matches previous behavior, but that we're open to reassessing in the future.

deads2k · 2020-06-22T13:26:45Z

/lgtm
needs squash

/hold

Holding for a squash and a comment. Ping me after and I'll retag.

sxllwx · 2020-07-13T05:25:45Z

/assign @deads2k

sxllwx · 2020-08-05T13:49:38Z

@deads2k PTAL

deads2k · 2020-08-05T14:28:53Z

/lgtm
/approve
/hold cancel

k8s-ci-robot · 2020-08-05T14:29:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, sxllwx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/apimachinery/pkg/OWNERS~~ [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fejta-bot · 2020-08-05T16:31:48Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

justaugustus · 2020-08-27T23:07:41Z

/test pull-kubernetes-e2e-kind-ipv6

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 30, 2020

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 30, 2020

k8s-ci-robot requested review from logicalhan and smarterclayton May 30, 2020 10:51

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 30, 2020

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 30, 2020

k8s-ci-robot assigned deads2k May 30, 2020

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 31, 2020

k8s-ci-robot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Jun 2, 2020

deads2k reviewed Jun 16, 2020

View reviewed changes

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 16, 2020

deads2k reviewed Jun 22, 2020

View reviewed changes

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Jun 22, 2020

lock-free broadcaster, use chan to ensure thread safety

e6e1769

sxllwx force-pushed the lock_free_broadcaster branch from 78990f0 to e6e1769 Compare July 13, 2020 03:08

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Aug 5, 2020

justaugustus added this to the v1.20 milestone Aug 27, 2020

k8s-ci-robot merged commit 16bd0bc into kubernetes:master Aug 28, 2020

github-actions bot mentioned this pull request Sep 3, 2020

Week Ending August 30, 2020 dev-obs/actus#219

Open

sxllwx mentioned this pull request Feb 9, 2021

REQUEST: New membership for sxllwx kubernetes/org#2486

Closed

6 tasks

aojea mentioned this pull request Mar 4, 2022

Fix Panic Condition #108080

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lock-free broadcaster #91602

Lock-free broadcaster #91602

sxllwx commented May 30, 2020 •

edited

Loading

k8s-ci-robot commented May 30, 2020

k8s-ci-robot commented May 30, 2020

k8s-ci-robot commented May 30, 2020

sxllwx commented May 30, 2020

sxllwx commented May 30, 2020

sxllwx commented Jun 2, 2020

sxllwx commented Jun 2, 2020

deads2k Jun 16, 2020

deads2k Jun 16, 2020

sxllwx Jun 22, 2020

deads2k Jun 16, 2020

sxllwx Jun 22, 2020

deads2k Jun 16, 2020

sxllwx Jun 22, 2020

deads2k commented Jun 16, 2020

deads2k commented Jun 16, 2020

deads2k Jun 22, 2020

sxllwx Jun 23, 2020

deads2k Jun 22, 2020

sxllwx Jun 23, 2020

deads2k commented Jun 22, 2020

sxllwx commented Jul 13, 2020

sxllwx commented Aug 5, 2020

deads2k commented Aug 5, 2020

k8s-ci-robot commented Aug 5, 2020

fejta-bot commented Aug 5, 2020

justaugustus commented Aug 27, 2020

Lock-free broadcaster #91602

Lock-free broadcaster #91602

Conversation

sxllwx commented May 30, 2020 • edited Loading

k8s-ci-robot commented May 30, 2020

k8s-ci-robot commented May 30, 2020

k8s-ci-robot commented May 30, 2020

sxllwx commented May 30, 2020

sxllwx commented May 30, 2020

sxllwx commented Jun 2, 2020

sxllwx commented Jun 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k commented Jun 16, 2020

deads2k commented Jun 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k commented Jun 22, 2020

sxllwx commented Jul 13, 2020

sxllwx commented Aug 5, 2020

deads2k commented Aug 5, 2020

k8s-ci-robot commented Aug 5, 2020

fejta-bot commented Aug 5, 2020

justaugustus commented Aug 27, 2020

sxllwx commented May 30, 2020 •

edited

Loading