Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix deadlock in disconnecting querier #5063

Merged
merged 5 commits into from
Jan 10, 2022

Conversation

cyriltovena
Copy link
Contributor

@cyriltovena cyriltovena commented Jan 6, 2022

There was a race condition between the broadcast triggered by the
canceled context and the condition waiting: if the broadcast happend
before the wait, wait never receives any broadcast.

I also ported back all tests missing from the code we forked.

Signed-off-by: Cyril Tovena cyril.tovena@gmail.com

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena cyriltovena requested a review from a team as a code owner January 6, 2022 13:53
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Copy link
Collaborator

@trevorwhitney trevorwhitney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good as far as I can tell, but this is a new area of the code for me.

It took me a while to understand how concurrency was reset, I wonder if some more verbose test names would help the next reader?

defer cancel()

// We use random port here, hopefully without any gRPC server.
cc, err := grpc.DialContext(ctx, "localhost:999", grpc.WithInsecure())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to not use the mechanism used in the frontend_test.go for getting a random port to guarantee no existing server?

	l, err := net.Listen("tcp", "")
	require.NoError(t, err)

	server := grpc.NewServer()

	h, p, err := net.SplitHostPort(l.Addr().String())
	require.NoError(t, err)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest the best is to use in-memory grpc server. I'll swap this for it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using google.golang.org/grpc/test/bufconn I'll send a commit to show you.

pkg/querier/worker/worker_test.go Outdated Show resolved Hide resolved
pkg/querier/worker/worker_test.go Outdated Show resolved Hide resolved
cyriltovena and others added 3 commits January 10, 2022 10:00
Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>
Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Copy link
Contributor

@dannykopping dannykopping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cyriltovena cyriltovena merged commit 6ef934b into grafana:main Jan 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants