Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[APM] Improve performance of GET /internal/apm/has_data #154997

Closed
dgieselaar opened this issue Apr 16, 2023 · 6 comments · Fixed by #173382
Closed

[APM] Improve performance of GET /internal/apm/has_data #154997

dgieselaar opened this issue Apr 16, 2023 · 6 comments · Fixed by #173382
Assignees
@sorenlouv
Labels
apm:performance APM UI - Performance Work Team:APM All issues that need APM UI Team support v8.12.0

Comments

@dgieselaar
Copy link
Member

The GET /internal/apm/has_data returns whether there is any APM data at all. The search we execute here is has_historical_agent_data, which is an unbound search for any APM event in any APM index. We set terminate_after to 1, but this search will still hit all shards (terminate_after is a per-shard limit). Here's an example of this search taking significantly longer than another search that is time range bound:

CleanShot 2023-04-16 at 11 22 11@2x

We should investigate whether there are alternatives that do not hit all shards. Maybe another API, or perhaps first execute a time range bound search, and only if that returns no hits, execute a broader search.
@dgieselaar dgieselaar added the Team:APM All issues that need APM UI Team support label Apr 16, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/apm-ui (Team:APM)

@sorenlouv
Copy link
Member

We set terminate_after to 1, but this search will still hit all shards (terminate_after is a per-shard limit)

So the coordinating node will wait for all shards to return before returning the result? It feels like it should just wait for the first shard that returns a non-zero result and then ignore/cancel the requests to the remaining shards. Wouldn't that be faster?

@dgieselaar
Copy link
Member Author

@sqren yes, that's my understanding. perhaps it'll short circuit after the first batch, but not sure. I'll ask some of the search folks.

@gbamparop gbamparop added the apm:performance APM UI - Performance Work label Apr 17, 2023
@gbamparop gbamparop removed the v8.9.0 label Jul 3, 2023
@dgieselaar
Copy link
Member Author

@gbamparop can we add this as a candidate for 8.10?

@gbamparop
Copy link
Contributor

@sqren sqren added the 8.10 candidate label Jul 12, 2023

Got the answer :)

@sorenlouv sorenlouv added v8.12.0 good first issue low hanging fruit and removed 8.10 candidate good first issue low hanging fruit labels Oct 31, 2023
@crespocarlos crespocarlos self-assigned this Dec 4, 2023
@crespocarlos crespocarlos removed their assignment Dec 14, 2023
@sorenlouv sorenlouv self-assigned this Dec 14, 2023
sorenlouv added a commit to sorenlouv/kibana that referenced this issue Dec 14, 2023
@sorenlouv
[APM] Add timerange to /has_data api
ad8428d 
Closes elastic#154997
@sorenlouv sorenlouv mentioned this issue Dec 14, 2023
Merged
@achyutjhunjhunwala
Copy link
Contributor

I hope this solves this SDH - https://github.com/elastic/sdh-kibana/issues/4256#issuecomment-1853747899

sorenlouv added a commit to sorenlouv/kibana that referenced this issue Dec 14, 2023
@sorenlouv
[APM] Add data_tier filter to /has_data api
222099d 
Closes elastic#154997

Remove console.log

Fix

Use data_tier instead of timestamp

Improve naming

Undo change
sorenlouv added a commit to sorenlouv/kibana that referenced this issue Dec 14, 2023
@sorenlouv
[APM] Add data_tier filter to /has_data api
8678a39 
Closes elastic#154997

Remove console.log

Fix

Use data_tier instead of timestamp

Improve naming

Undo change
sorenlouv added a commit to sorenlouv/kibana that referenced this issue Dec 17, 2023
@sorenlouv
[APM] Add data_tier filter to /has_data api
f61fd16 
Closes elastic#154997

Remove console.log

Fix

Use data_tier instead of timestamp

Improve naming

Undo change
sorenlouv added a commit that referenced this issue Dec 18, 2023
@sorenlouv
[APM] Add filter to /has_data api (#173382)
e7593c0 
Closes #154997

This PR adds a data tier filter to the `/has_data` api, thus limitting
the number of shards being hit by the request.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Dec 18, 2023
@sorenlouv
[APM] Add filter to /has_data api (elastic#173382)
ddf4b1f 
Closes elastic#154997

This PR adds a data tier filter to the `/has_data` api, thus limitting
the number of shards being hit by the request.

(cherry picked from commit e7593c0)
kibanamachine added a commit that referenced this issue Dec 18, 2023
@kibanamachine @sorenlouv
[8.12] [APM] Add filter to /has_data api (#173382) (#173559)
23406d2 
# Backport

This will backport the following commits from `main` to `8.12`:
- [[APM] Add filter to `/has_data` api
(#173382)](#173382)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Søren
Louv-Jansen","email":"soren.louv@elastic.co"},"sourceCommit":{"committedDate":"2023-12-18T18:18:03Z","message":"[APM]
Add filter to `/has_data` api (#173382)\n\nCloses
#154997 PR adds a
data tier filter to the `/has_data` api, thus limitting\r\nthe number of
shards being hit by the
request.","sha":"e7593c0e46f1ce707c1b951f8d013e722ac79353","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v8.12.0","v8.10.5","Team:obs-ux-infra_services","v8.13.0","v8.11.4"],"number":173382,"url":"#173382
Add filter to `/has_data` api (#173382)\n\nCloses
#154997 PR adds a
data tier filter to the `/has_data` api, thus limitting\r\nthe number of
shards being hit by the
request.","sha":"e7593c0e46f1ce707c1b951f8d013e722ac79353"}},"sourceBranch":"main","suggestedTargetBranches":["8.12","8.10","8.11"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.10","label":"v8.10.5","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"#173382
Add filter to `/has_data` api (#173382)\n\nCloses
#154997 PR adds a
data tier filter to the `/has_data` api, thus limitting\r\nthe number of
shards being hit by the
request.","sha":"e7593c0e46f1ce707c1b951f8d013e722ac79353"}},{"branch":"8.11","label":"v8.11.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sorenlouv sorenlouv

Labels
apm:performance APM UI - Performance Work Team:APM All issues that need APM UI Team support v8.12.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[APM] Add filter to /has_data api sorenlouv/kibana
6 participants
@sorenlouv @dgieselaar @crespocarlos @gbamparop @achyutjhunjhunwala @elasticmachine