Skip to content

Additional trace logging for ESSingleNodeTestCase failures #110790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Additional trace logging for ESSingleNodeTestCase failures #110790

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a02cd03
updating logging
pmpailis Jul 11, 2024
23ad1bc
iter
pmpailis Jul 11, 2024
f0ad456
Merge branch 'main' into additional_logging_for_essinglenodetestcase
elasticmachine Jul 11, 2024
c728c08
adding more tests
pmpailis Jul 19, 2024
0d5b74d
enabling trace logging for all essinglenodetests
pmpailis Jul 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...ernalClusterTest/java/org/elasticsearch/action/search/SearchProgressActionListenerIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
import static org.hamcrest.Matchers.lessThan;

@TestIssueLogging(
issueUrl = "https://github.com/elastic/elasticsearch/issues/109830",
issueUrl = "https://github.com/elastic/elasticsearch/issues/110790",
value = "org.elasticsearch.action.search:TRACE," + "org.elasticsearch.search.SearchService:TRACE"
)
public class SearchProgressActionListenerIT extends ESSingleNodeTestCase {
Expand Down
6 changes: 5 additions & 1 deletion ...ernalClusterTest/java/org/elasticsearch/search/aggregations/bucket/terms/RareTermsIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import org.elasticsearch.search.aggregations.metrics.InternalTopHits;
import org.elasticsearch.search.aggregations.metrics.TopHitsAggregationBuilder;
import org.elasticsearch.test.ESSingleNodeTestCase;
import org.elasticsearch.test.junit.annotations.TestIssueLogging;
import org.elasticsearch.xcontent.XContentType;
import org.hamcrest.Matchers;

Expand All @@ -32,7 +33,10 @@
/**
* Test that index enough data to trigger the creation of Cuckoo filters.
*/

@TestIssueLogging(
issueUrl = "https://github.com/elastic/elasticsearch/issues/110790",
value = "org.elasticsearch.action.search:TRACE," + "org.elasticsearch.search.SearchService:TRACE"
)
public class RareTermsIT extends ESSingleNodeTestCase {

private static final String index = "idx";
Expand Down
6 changes: 6 additions & 0 deletions server/src/main/java/org/elasticsearch/search/SearchService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
Expand Down Expand Up @@ -463,6 +464,7 @@ public void beforeIndexShardCreated(ShardRouting routing, Settings indexSettings
}

protected void putReaderContext(ReaderContext context) {
logger.trace("adding reader context [{}]", context.id());
Copy link
Member

@benwtrent benwtrent Jul 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to accidentally merge this into main. The test seems to fail enough that it should be reproducible locally or with this PR if the test is unmuted.

Copy link
Contributor Author

@pmpailis pmpailis Jul 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm fair enough. Would you be worried of potential impact with this addition?
Tbh, I haven't been lucky locally even after a huge number of runs. Will try to reproduce within this PR :)

final ReaderContext previous = activeReaders.put(context.id().getId(), context);
assert previous == null;
// ensure that if we race against afterIndexRemoved, we remove the context from the active list.
Expand Down Expand Up @@ -1567,6 +1569,10 @@ public int getActiveContexts() {
return this.activeReaders.size();
}

public Set<Long> getActiveContextIDs() {
return new HashSet<>(this.activeReaders.keySet());
}

/**
* Returns the number of scroll contexts opened on the node
*/
Expand Down
5 changes: 5 additions & 0 deletions server/src/test/java/org/elasticsearch/action/search/KnnSearchSingleNodeTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import org.elasticsearch.search.vectors.KnnSearchBuilder;
import org.elasticsearch.search.vectors.KnnVectorQueryBuilder;
import org.elasticsearch.test.ESSingleNodeTestCase;
import org.elasticsearch.test.junit.annotations.TestIssueLogging;
import org.elasticsearch.xcontent.XContentBuilder;
import org.elasticsearch.xcontent.XContentFactory;

Expand All @@ -30,6 +31,10 @@
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.greaterThan;

@TestIssueLogging(
issueUrl = "https://github.com/elastic/elasticsearch/issues/110790",
value = "org.elasticsearch.action.search:TRACE," + "org.elasticsearch.search.SearchService:TRACE"
)
public class KnnSearchSingleNodeTests extends ESSingleNodeTestCase {
private static final int VECTOR_DIMENSION = 10;

Expand Down
9 changes: 8 additions & 1 deletion test/framework/src/main/java/org/elasticsearch/test/ESSingleNodeTestCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@
import org.elasticsearch.search.ConcurrentSearchTestPlugin;
import org.elasticsearch.search.SearchService;
import org.elasticsearch.search.internal.SearchContext;
import org.elasticsearch.test.junit.annotations.TestIssueLogging;
import org.elasticsearch.test.rest.ESRestTestCase;
import org.elasticsearch.transport.TransportSettings;
import org.elasticsearch.xcontent.NamedXContentRegistry;
Expand All @@ -64,6 +65,7 @@
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
Expand All @@ -81,6 +83,10 @@
* A test that keep a singleton node started for all tests that can be used to get
* references to Guice injectors in unit tests.
*/
@TestIssueLogging(
issueUrl = "https://github.com/elastic/elasticsearch/issues/110790",
value = "org.elasticsearch.action.search:TRACE," + "org.elasticsearch.search.SearchService:TRACE"
)
public abstract class ESSingleNodeTestCase extends ESTestCase {

private static Node NODE = null;
Expand Down Expand Up @@ -134,7 +140,8 @@ public void tearDown() throws Exception {
ensureAllFreeContextActionsAreConsumed();

SearchService searchService = getInstanceFromNode(SearchService.class);
assertThat(searchService.getActiveContexts(), equalTo(0));
Set<Long> activeContexts = searchService.getActiveContextIDs();
assertThat("contexts: {" + activeContexts + "} have not been closed yet", activeContexts.size(), equalTo(0));
assertThat(searchService.getOpenScrollContexts(), equalTo(0));
super.tearDown();
var deleteDataStreamsRequest = new DeleteDataStreamAction.Request("*");
Expand Down
5 changes: 5 additions & 0 deletions ...asticsearch/xpack/analytics/aggregations/metrics/HistogramPercentileAggregationTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import org.elasticsearch.search.aggregations.metrics.TDigestState;
import org.elasticsearch.tdigest.Centroid;
import org.elasticsearch.test.ESSingleNodeTestCase;
import org.elasticsearch.test.junit.annotations.TestIssueLogging;
import org.elasticsearch.xcontent.XContentBuilder;
import org.elasticsearch.xcontent.XContentFactory;
import org.elasticsearch.xpack.analytics.AnalyticsPlugin;
Expand All @@ -37,6 +38,10 @@
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertResponse;

@TestIssueLogging(
issueUrl = "https://github.com/elastic/elasticsearch/issues/110790",
value = "org.elasticsearch.action.search:TRACE," + "org.elasticsearch.search.SearchService:TRACE"
)
public class HistogramPercentileAggregationTests extends ESSingleNodeTestCase {

public void testHDRHistogram() throws Exception {
Expand Down
5 changes: 5 additions & 0 deletions ...t/java/org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import org.elasticsearch.search.rank.rerank.AbstractRerankerIT;
import org.elasticsearch.test.ESSingleNodeTestCase;
import org.elasticsearch.test.hamcrest.ElasticsearchAssertions;
import org.elasticsearch.test.junit.annotations.TestIssueLogging;
import org.elasticsearch.xpack.core.inference.action.InferenceAction;
import org.elasticsearch.xpack.inference.InferencePlugin;
import org.junit.Before;
Expand All @@ -30,6 +31,10 @@

import static org.hamcrest.Matchers.containsString;

@TestIssueLogging(
issueUrl = "https://github.com/elastic/elasticsearch/issues/110790",
value = "org.elasticsearch.action.search:TRACE," + "org.elasticsearch.search.SearchService:TRACE"
)
public class TextSimilarityRankTests extends ESSingleNodeTestCase {

/**
Expand Down
5 changes: 5 additions & 0 deletions ...ava/org/elasticsearch/xpack/security/authz/store/NativePrivilegeStoreSingleNodeTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.test.SecuritySingleNodeTestCase;
import org.elasticsearch.test.junit.annotations.TestIssueLogging;
import org.elasticsearch.xcontent.XContentType;
import org.elasticsearch.xpack.core.security.action.apikey.CreateApiKeyAction;
import org.elasticsearch.xpack.core.security.action.apikey.CreateApiKeyRequest;
Expand Down Expand Up @@ -54,6 +55,10 @@
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;

@TestIssueLogging(
issueUrl = "https://github.com/elastic/elasticsearch/issues/110790",
value = "org.elasticsearch.action.search:TRACE," + "org.elasticsearch.search.SearchService:TRACE"
)
public class NativePrivilegeStoreSingleNodeTests extends SecuritySingleNodeTestCase {

@Before
Expand Down