Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changefeedccl: deflake TestChangefeedRetryableSinkError #30172

Merged
merged 2 commits into from Sep 13, 2018

Conversation

benesch
Copy link
Contributor

@benesch benesch commented Sep 13, 2018

TestChangefeedRetryableSinkError was performing many rounds of bcrypt,
which is excruciatingly slow when the race detector is on. Put its test
cluster into insecure mode to skip bcrypt hashing.

Fix #30156.
Fix #30161.

Release note: None

@benesch benesch requested review from danhhz, mrtracy and a team September 13, 2018 01:25
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@benesch benesch requested a review from a team September 13, 2018 01:34
Copy link
Contributor

@mrtracy mrtracy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent, thanks for tracking that down so quickly. LGTM.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained

Copy link
Member

@nvanbenschoten nvanbenschoten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

The channel race was surprising to me though. Do you have any references explaining why we would see that? Is len(ch) considered an unsynchronized read?

@benesch
Copy link
Contributor Author

benesch commented Sep 13, 2018

The channel race was surprising to me though. Do you have any references explaining why we would see that? Is len(ch) considered an unsynchronized read?

Me too. len(ch) is kind of a useless operation, I think. Here's two Google Groups threads that touch on the issue:

Neither says definitively that len(ch) is an unsynchronized read, but I gathered as much.

@benesch
Copy link
Contributor Author

benesch commented Sep 13, 2018

@benesch
Copy link
Contributor Author

benesch commented Sep 13, 2018

Er, I guess that's just the code used by reflect. The actual code is always inlined by the compiler, but it again looks pretty darn unsynchronized: https://github.com/golang/go/blob/de28555c0b33fcaa02779d55ea9289135280ae9f/src/cmd/compile/internal/gc/ssa.go#L4421

TestChangefeedRetryableSinkError was performing many rounds of bcrypt,
which is excruciatingly slow when the race detector is on. Put its test
cluster into insecure mode to skip bcrypt hashing.

Fix cockroachdb#30156.
Fix cockroachdb#30161.

Release note: None
The previous pattern used to drain a channel

    if len(ch) > 0 {
        <-ch
    }

is considered a race by the race detector if close(ch) occurs at the
same time. Adjust the channel-draining code to be race-safe by instead
perfomring repeated nonblocking reads.

Fix cockroachdb#30159.

Release note: None
Copy link
Contributor

@rjnn rjnn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch! LGTM.

Reviewed 2 of 2 files at r1, 1 of 1 files at r2.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained


pkg/storage/store.go, line 1751 at r2 (raw file):

			case <-ch:
				// Drain all notifications from the channel.
			loop:

TIL! Thanks.

@benesch
Copy link
Contributor Author

benesch commented Sep 13, 2018

bors r=nvanbenschoten,arjunravinarayan,mrtracy

craig bot pushed a commit that referenced this pull request Sep 13, 2018
30172: changefeedccl: deflake TestChangefeedRetryableSinkError r=nvanbenschoten,arjunravinarayan,mrtracy a=benesch

TestChangefeedRetryableSinkError was performing many rounds of bcrypt,
which is excruciatingly slow when the race detector is on. Put its test
cluster into insecure mode to skip bcrypt hashing.

Fix #30156.
Fix #30161.

Release note: None

Co-authored-by: Nikhil Benesch <nikhil.benesch@gmail.com>
@craig
Copy link
Contributor

craig bot commented Sep 13, 2018

Build succeeded

@benesch
Copy link
Contributor Author

benesch commented Sep 18, 2018

The len(ch) saga continues upstream: golang/go#27070.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants