Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pkg/queue] Add StartConsumersWithFactory function #2714

Merged
merged 5 commits into from
Jan 7, 2021

Conversation

mx-psi
Copy link
Contributor

@mx-psi mx-psi commented Jan 5, 2021

Which problem is this PR solving?

Short description of the changes

  • Add StartConsumersWithFactory function to allow for keeping state per-consumer.
  • Rewrite StartConsumers with a dummy factory function.
  • Add unit test and benchmark for the new function.

This provides a way to keep state for each consumer of a bounded queue,
which is useful in certain performance-critical setups. Fixes jaegertracing#2685.

Signed-off-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
@mergify mergify bot requested a review from jpkrohling January 5, 2021 11:29
@codecov
Copy link

codecov bot commented Jan 5, 2021

Codecov Report

Merging #2714 (ab42619) into master (a6e29ad) will increase coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2714      +/-   ##
==========================================
+ Coverage   95.73%   95.74%   +0.01%     
==========================================
  Files         216      216              
  Lines        9588     9593       +5     
==========================================
+ Hits         9179     9185       +6     
  Misses        336      336              
+ Partials       73       72       -1     
Impacted Files Coverage Δ
pkg/queue/bounded_queue.go 93.05% <100.00%> (+0.51%) ⬆️
plugin/storage/integration/integration.go 77.34% <0.00%> (-0.56%) ⬇️
cmd/query/app/static_handler.go 96.52% <0.00%> (ø)
cmd/query/app/server.go 90.16% <0.00%> (+1.63%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a6e29ad...ab42619. Read the comment docs.

@mx-psi mx-psi marked this pull request as ready for review January 5, 2021 11:58
@mx-psi mx-psi requested a review from a team as a code owner January 5, 2021 11:58
The common part with the assertions was moved to a new `checkQueue`
function that is used by `TestBoundedQueue` and `TestBoundedQueueWithFactory`.

Signed-off-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
mx-psi and others added 3 commits January 7, 2021 11:14
Signed-off-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Refactor unit tests using a `helper` function that takes a function
to start consumers.

Signed-off-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
@mx-psi
Copy link
Contributor Author

mx-psi commented Jan 7, 2021

@yurishkuro Thanks for the comments, I think I addressed everything, hopefully this looks better :)

Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@@ -33,7 +33,7 @@ import (
// In this test we run a queue with capacity 1 and a single consumer.
// We want to test the overflow behavior, so we block the consumer
// by holding a startLock before submitting items to the queue.
func TestBoundedQueue(t *testing.T) {
func helper(t *testing.T, startConsumers func(q *BoundedQueue, consumerFn func(item interface{}))) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hehe, I didn't mean the name helper literally, was just too lazy to type what you had, but that's fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hahaha, I couldn't come up with anything better so I left it like you said 😄

@@ -112,6 +112,18 @@ func TestBoundedQueue(t *testing.T) {
assert.False(t, q.Produce("x"), "cannot push to closed queue")
}

func TestBoundedQueue(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

much cleaner, minimal changes

@yurishkuro yurishkuro merged commit e788e55 into jaegertracing:master Jan 7, 2021
@mx-psi mx-psi deleted the mx-psi/add-queue-factory branch January 7, 2021 17:04
bhiravabhatla pushed a commit to bhiravabhatla/jaeger that referenced this pull request Jan 25, 2021
)

* [pkg/queue] Add `StartConsumersWithFactory` function

This provides a way to keep state for each consumer of a bounded queue,
which is useful in certain performance-critical setups. Fixes jaegertracing#2685.

Signed-off-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>

* Refactor bounded queue tests to extract common parts
The common part with the assertions was moved to a new `checkQueue`
function that is used by `TestBoundedQueue` and `TestBoundedQueueWithFactory`.

Signed-off-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>

* Use http.HandlerFunc pattern for getting a Consumer from a callback

Signed-off-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>

* Address review comment about unit tests

Refactor unit tests using a `helper` function that takes a function
to start consumers.

Signed-off-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[pkg/queue] Provide factory for consumers
3 participants