Skip to content

Add challenge tests for logsdb that utilize stored source #115606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 25, 2024
Merged

Add challenge tests for logsdb that utilize stored source #115606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
23ca087
Add challenge tests for logsdb that utilize stored source
lkts Oct 24, 2024
e0992d6
Fix comments
lkts Oct 25, 2024
e9d6ef4
fix another comment
lkts Oct 25, 2024
89ef9e9
Merge branch 'main' into logsdb_reindex_tests_new
elasticmachine Oct 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .../logsdb/src/javaRestTest/java/org/elasticsearch/xpack/logsdb/qa/DataGenerationHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ void standardMapping(XContentBuilder builder) throws IOException {
}

void logsDbSettings(Settings.Builder builder) {
builder.put("index.mode", "logsdb");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noticed this oversight, it only impacted reindex tests.

if (keepArraySource) {
builder.put(Mapper.SYNTHETIC_SOURCE_KEEP_INDEX_SETTING.getKey(), "arrays");
}
Expand Down
30 changes: 0 additions & 30 deletions ...ticsearch/xpack/logsdb/qa/LogsDbVersusLogsDbReindexedIntoStandardModeChallengeRestIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,10 @@

package org.elasticsearch.xpack.logsdb.qa;

import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.common.CheckedSupplier;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.xcontent.XContentBuilder;

import java.io.IOException;
import java.util.List;
import java.util.Locale;

import static org.hamcrest.Matchers.equalTo;

/**
* This test compares behavior of a logsdb data stream and a standard index mode data stream
Expand Down Expand Up @@ -52,27 +45,4 @@ public void baselineMappings(XContentBuilder builder) throws IOException {
public void contenderMappings(XContentBuilder builder) throws IOException {
dataGenerationHelper.standardMapping(builder);
}

@Override
public Response indexContenderDocuments(CheckedSupplier<List<XContentBuilder>, IOException> documentsSupplier) throws IOException {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method exists in a base class, just a clean up.

var reindexRequest = new Request("POST", "/_reindex?refresh=true");
reindexRequest.setJsonEntity(String.format(Locale.ROOT, """
{
"source": {
"index": "%s"
},
"dest": {
"index": "%s",
"op_type": "create"
}
}
""", getBaselineDataStreamName(), getContenderDataStreamName()));
var response = client.performRequest(reindexRequest);
assertOK(response);

var body = entityAsMap(response);
assertThat("encountered failures when performing reindex:\n " + body, body.get("failures"), equalTo(List.of()));

return response;
}
}
48 changes: 48 additions & 0 deletions ...g/elasticsearch/xpack/logsdb/qa/LogsDbVersusReindexedIntoStoredSourceChallengeRestIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

package org.elasticsearch.xpack.logsdb.qa;

import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.xcontent.XContentBuilder;

import java.io.IOException;

/**
* This test compares behavior of a standard mode data stream and a logsdb data stream using stored source.
* There should be no differences between such two data streams.
*/
public class LogsDbVersusReindexedIntoStoredSourceChallengeRestIT extends ReindexChallengeRestIT {
public String getBaselineDataStreamName() {
return "logs-apache-baseline";
}

public String getContenderDataStreamName() {
return "logs-apache-reindexed";
}

@Override
public void baselineSettings(Settings.Builder builder) {
dataGenerationHelper.logsDbSettings(builder);
}

@Override
public void contenderSettings(Settings.Builder builder) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if we should just create base classes like

SyntheticSourceAbstractChallengeRestTest
DisableSourceAbstractChallengeRestTest
StoredSourceAbstractChallengeRestTest

where the setting is applied and we can just inherit from them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thing is that sometimes baseline has stored source and sometimes the contender.

dataGenerationHelper.logsDbSettings(builder);
builder.put("index.mapping.source.mode", "stored");
}

@Override
public void baselineMappings(XContentBuilder builder) throws IOException {
dataGenerationHelper.logsDbMapping(builder);
}

@Override
public void contenderMappings(XContentBuilder builder) throws IOException {
dataGenerationHelper.logsDbMapping(builder);
}
}
30 changes: 0 additions & 30 deletions ...st/java/org/elasticsearch/xpack/logsdb/qa/LogsDbVersusReindexedLogsDbChallengeRestIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,10 @@

package org.elasticsearch.xpack.logsdb.qa;

import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.common.CheckedSupplier;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.xcontent.XContentBuilder;

import java.io.IOException;
import java.util.List;
import java.util.Locale;

import static org.hamcrest.Matchers.equalTo;

/**
* This test compares behavior of a logsdb data stream and a data stream containing
Expand Down Expand Up @@ -52,27 +45,4 @@ public void baselineMappings(XContentBuilder builder) throws IOException {
public void contenderMappings(XContentBuilder builder) throws IOException {
dataGenerationHelper.logsDbMapping(builder);
}

@Override
public Response indexContenderDocuments(CheckedSupplier<List<XContentBuilder>, IOException> documentsSupplier) throws IOException {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method exists in a base class, just a clean up.

var reindexRequest = new Request("POST", "/_reindex?refresh=true");
reindexRequest.setJsonEntity(String.format(Locale.ROOT, """
{
"source": {
"index": "%s"
},
"dest": {
"index": "%s",
"op_type": "create"
}
}
""", getBaselineDataStreamName(), getContenderDataStreamName()));
var response = client.performRequest(reindexRequest);
assertOK(response);

var body = entityAsMap(response);
assertThat("encountered failures when performing reindex:\n " + body, body.get("failures"), equalTo(List.of()));

return response;
}
}
22 changes: 22 additions & 0 deletions ...java/org/elasticsearch/xpack/logsdb/qa/StandardVersusLogsStoredSourceChallengeRestIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

package org.elasticsearch.xpack.logsdb.qa;

import org.elasticsearch.common.settings.Settings;

/**
* This test compares behavior of a standard mode data stream and a logsdb data stream using stored source.
* There should be no differences between such two data streams.
*/
public class StandardVersusLogsStoredSourceChallengeRestIT extends StandardVersusLogsIndexModeRandomDataChallengeRestIT {
@Override
public void contenderSettings(Settings.Builder builder) {
super.contenderSettings(builder);
builder.put("index.mapping.source.mode", "stored");
}
}
49 changes: 49 additions & 0 deletions ...elasticsearch/xpack/logsdb/qa/StoredSourceLogsDbVersusReindexedLogsDbChallengeRestIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

package org.elasticsearch.xpack.logsdb.qa;

import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.xcontent.XContentBuilder;

import java.io.IOException;

/**
* This test compares behavior of a logsdb data stream using stored source and a logsdb data stream
* containing data reindexed from initial data stream.
* There should be no differences between such two data streams.
*/
public class StoredSourceLogsDbVersusReindexedLogsDbChallengeRestIT extends ReindexChallengeRestIT {
public String getBaselineDataStreamName() {
return "logs-apache-baseline";
}

public String getContenderDataStreamName() {
return "logs-apache-reindexed";
}

@Override
public void baselineSettings(Settings.Builder builder) {
dataGenerationHelper.logsDbSettings(builder);
builder.put("index.mapping.source.mode", "stored");
}

@Override
public void contenderSettings(Settings.Builder builder) {
dataGenerationHelper.logsDbSettings(builder);
}

@Override
public void baselineMappings(XContentBuilder builder) throws IOException {
dataGenerationHelper.logsDbMapping(builder);
}

@Override
public void contenderMappings(XContentBuilder builder) throws IOException {
dataGenerationHelper.logsDbMapping(builder);
}
}