Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chore][exporter/elasticsearch] Use RunParallel in benchmarks #33621

Draft
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

carsonip
Copy link
Contributor

@carsonip carsonip commented Jun 18, 2024

Description:

  • [chore][exporter/elasticsearch] Use RunParallel in bench
  • Measure docs/s and bulkReqs/s
  • Explicitly disable queue

This will be needed to measure performance of #32632

@carsonip carsonip requested a review from lahsivjar June 25, 2024 17:39
Copy link
Contributor

@lahsivjar lahsivjar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! One small nit on the interface of counters.

Comment on lines +49 to +51
type counters struct {
observedBulkRequests atomic.Int64
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking nit: I would prefer an interface that allows overriding the bulk request counter rather than this similar to the data-provider.

So, something like func (es *esDataReceiver) SetBulkRequestCounter(c *atomic.Uint64) { es.bulkRequestCounter = c } with a default value set in newElasticsearchDataReceiver. The current interfaces kinda encourages adding more counters which I am a bit hesitant to do. Will leave it entirely up to you.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The data provider has SetLoadGeneratorCounters to satisfy the DataProvider interface, while there isn't a ElasticsearchDataReceiver interface to satisfy. There is only 1 concrete implementation and no interface in our case. Given I can't think of any more counters to add at the moment, the extra code of SetBulkRequestCounter isn't offering any obvious benefits imo. We can revisit this when we find the need to make changes to esDataReceiver again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The data provider has SetLoadGeneratorCounters to satisfy the DataProvider interface, while there isn't a ElasticsearchDataReceiver interface to satisfy. There is only 1 concrete implementation and no interface in our case

That was just an example and I am not suggesting introducing/satisfying any interface here. Introducing a method like SetBulkRequestsCounter keeps the counters required by users in their code and prevents the counters struct leaking/sharing counters. IMO, the current implementation encourages setting more counters which I am not a fan of but no strong feelings about it.

We can revisit this when we find the need to make changes to esDataReceiver again.

SGTM!

@carsonip carsonip marked this pull request as draft June 26, 2024 12:05
@carsonip
Copy link
Contributor Author

carsonip commented Jul 2, 2024

This PR only exists to show the code used to benchmark main and #32632. It is actually not useful to keep this modified benchmark in the codebase as #32632 will contain a better way to benchmark the exporter. Therefore, the plan is to close this PR after #32632 is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants