-
Notifications
You must be signed in to change notification settings - Fork 24.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Expose telemetry about search usage (#91528)
This is the continuation of #90176 which leverages #90425 to count query types. This PR adds search usage stats to the existing telemetry by counting sections being used as part of a search request, as well as query types. Each distinct query type is counted once per search request. The counting is performed while parsing, for the following REST search endpoints: - _search - _msearch - _async_search - _search/template - _msearch/template - _fleet/_fleet_search - _fleet/_fleet_msearch All other API using search internally, like reindex, ML transform, rank eval, sql etc. are not counted as part of these search usage stats. Such additional functionalities should have its own dedicated telemetry if needed. The counting of the search sections is not extensive, only the ones that are interesting to collect counts for are tracked. The following is the new section added to the cluster stats API response, including some sample stats: ``` "search" : { "total" : 63, "sections" : { "knn" : 42, "query" : 21, "aggs" : 46 }, "query" : { "match" : 58 } } ``` A big part of the change is actually the plumbing to make a common service class that holds the counters available to all the different callers of the parsing methods, especially plugins. Ideally, there would be a separate component that exposes the search parsing functionality rather than static methods, but changing that would require making the additional component available to the REST layer which is not trivial. I reused the existing UsageService which the RestController already holds, and is already used to count access to the different REST endpoints. Co-authored-by: Mayya Sharipova mayya.sharipova@elastic.co
- Loading branch information
Showing
43 changed files
with
1,424 additions
and
104 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| pr: 91528 | ||
| summary: Expose telemetry about search usage | ||
| area: Search | ||
| type: enhancement | ||
| issues: [] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
107 changes: 107 additions & 0 deletions
107
...he/src/internalClusterTest/java/org/elasticsearch/script/mustache/SearchUsageStatsIT.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0 and the Server Side Public License, v 1; you may not use this file except | ||
| * in compliance with, at your election, the Elastic License 2.0 or the Server | ||
| * Side Public License, v 1. | ||
| */ | ||
|
|
||
| package org.elasticsearch.script.mustache; | ||
|
|
||
| import org.elasticsearch.action.admin.cluster.stats.SearchUsageStats; | ||
| import org.elasticsearch.client.Request; | ||
| import org.elasticsearch.common.bytes.BytesArray; | ||
| import org.elasticsearch.plugins.Plugin; | ||
| import org.elasticsearch.test.ESIntegTestCase; | ||
| import org.elasticsearch.xcontent.XContentType; | ||
|
|
||
| import java.io.IOException; | ||
| import java.util.Collection; | ||
| import java.util.List; | ||
|
|
||
| import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; | ||
|
|
||
| @ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.TEST, numDataNodes = 1) | ||
| public class SearchUsageStatsIT extends ESIntegTestCase { | ||
|
|
||
| @Override | ||
| protected boolean addMockHttpTransport() { | ||
| return false; // enable http | ||
| } | ||
|
|
||
| @Override | ||
| protected Collection<Class<? extends Plugin>> nodePlugins() { | ||
| return List.of(MustachePlugin.class); | ||
| } | ||
|
|
||
| public void testSearchUsageStats() throws IOException { | ||
| { | ||
| SearchUsageStats stats = client().admin().cluster().prepareClusterStats().get().getIndicesStats().getSearchUsageStats(); | ||
| assertEquals(0, stats.getTotalSearchCount()); | ||
| assertEquals(0, stats.getQueryUsage().size()); | ||
| assertEquals(0, stats.getSectionsUsage().size()); | ||
| } | ||
|
|
||
| { | ||
| Request request = new Request("GET", "/_search/template"); | ||
| request.setJsonEntity(""" | ||
| { | ||
| "source": { | ||
| "query": { | ||
| "{{query_type}}": { | ||
| "message" : "{{query_string}}" | ||
| } | ||
| } | ||
| }, | ||
| "params": { | ||
| "query_string": "text query", | ||
| "query_type" : "match" | ||
| } | ||
| } | ||
| """); | ||
| getRestClient().performRequest(request); | ||
| } | ||
| { | ||
| assertAcked(client().admin().cluster().preparePutStoredScript().setId("testTemplate").setContent(new BytesArray(""" | ||
| { | ||
| "script": { | ||
| "lang": "mustache", | ||
| "source": { | ||
| "query": { | ||
| "match": { | ||
| "theField": "{{fieldParam}}" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }"""), XContentType.JSON)); | ||
| Request request = new Request("GET", "/_search/template"); | ||
| request.setJsonEntity(""" | ||
| { | ||
| "id": "testTemplate", | ||
| "params": { | ||
| "fieldParam": "text query" | ||
| } | ||
| } | ||
| """); | ||
| getRestClient().performRequest(request); | ||
| } | ||
| { | ||
| Request request = new Request("POST", "/_msearch/template"); | ||
| request.setJsonEntity(""" | ||
| {} | ||
| {"id": "testTemplate", "params": {"fieldParam":"text query"}} | ||
| {} | ||
| {"source": {"query": { "match": {"message" : "{{query_string}}"}}},"params": {"query_string": "text query"}} | ||
| """); | ||
| getRestClient().performRequest(request); | ||
| } | ||
|
|
||
| SearchUsageStats stats = client().admin().cluster().prepareClusterStats().get().getIndicesStats().getSearchUsageStats(); | ||
| assertEquals(4, stats.getTotalSearchCount()); | ||
| assertEquals(1, stats.getQueryUsage().size()); | ||
| assertEquals(4, stats.getQueryUsage().get("match").longValue()); | ||
| assertEquals(1, stats.getSectionsUsage().size()); | ||
| assertEquals(4, stats.getSectionsUsage().get("query").longValue()); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.