Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various RollupActionSingleNodeTests failing with IndexNotFoundException #69799

Closed
DaveCTurner opened this issue Mar 2, 2021 · 5 comments · Fixed by #85708 or #87333
Closed

Various RollupActionSingleNodeTests failing with IndexNotFoundException #69799

DaveCTurner opened this issue Mar 2, 2021 · 5 comments · Fixed by #85708 or #87333
Assignees
Labels
:StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

Build scan:

Repro line: ./gradlew ':x-pack:plugin:rollup:test' --tests "org.elasticsearch.xpack.rollup.v2.RollupActionSingleNodeTests.testMaxMetric" -Dtests.seed=AF76AE54E383FA3F -Dtests.security.manager=true -Dtests.locale=nb -Dtests.timezone=Asia/Pontianak -Druntime.java=11

Reproduces locally?: No

Applicable branches: master, 7.x and 7.12

Failure history:

https://build-stats.elastic.co/goto/de41e6b94aa2ae417cbe342ddc4d0741 indicates a few failures a day.

Failure excerpt:

  1> [2021-03-02T15:01:32,165][INFO ][o.e.x.r.v.RollupActionSingleNodeTests] [testAvgMetric] before test
  1> [2021-03-02T15:01:32,167][DEPRECATION][o.e.d.c.m.MetadataCreateIndexService] [node_s_0] index [jnmin] matches multiple legacy templates [one_shard_index_template, random-soft-deletes-template], composable templates will only match a single template
  1> [2021-03-02T15:01:32,173][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s_0] [jnmin] creating index, cause [api], templates [one_shard_index_template, random-soft-deletes-template], shards [1]/[0]
  1> [2021-03-02T15:01:32,232][DEPRECATION][o.e.d.c.m.MetadataCreateIndexService] [node_s_0] index [.rolluptmp-kglmlq] matches multiple legacy templates [one_shard_index_template, random-soft-deletes-template], composable templates will only match a single template
  1> [2021-03-02T15:01:32,233][DEPRECATION][o.e.d.c.m.MetadataCreateIndexService] [node_s_0] index name [.rolluptmp-kglmlq] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices
  1> [2021-03-02T15:01:32,236][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s_0] [.rolluptmp-kglmlq] creating index, cause [auto(bulk api)], templates [one_shard_index_template, random-soft-deletes-template], shards [1]/[0]
  1> [2021-03-02T15:01:32,362][INFO ][o.e.c.r.a.AllocationService] [node_s_0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.rolluptmp-kglmlq][0]]]).
  1> [2021-03-02T15:01:32,762][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s_0] [.rolluptmp-ecjbud] creating index, cause [api], templates [], shards [1]/[1]
  1> [2021-03-02T15:01:32,834][INFO ][o.e.c.m.MetadataMappingService] [node_s_0] [.rolluptmp-kglmlq/3YQHCIUPTW6YJhWTeug1Og] create_mapping [_doc]
  1> [2021-03-02T15:01:33,091][INFO ][o.e.x.r.v.RollupShardIndexer] [node_s_0] Successfully sent [262], indexed [262]
  1> [2021-03-02T15:01:33,149][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s_0] applying create index request using existing index [.rolluptmp-kglmlq] metadata
  1> [2021-03-02T15:01:33,153][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s_0] [kglmlq] creating index, cause [clone_index], templates [], shards [1]/[1]
  1> [2021-03-02T15:01:33,220][INFO ][o.e.x.r.v.RollupShardIndexer] [node_s_0] Successfully sent [10], indexed [10]
  1> [2021-03-02T15:01:33,328][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s_0] applying create index request using existing index [.rolluptmp-ecjbud] metadata
  1> [2021-03-02T15:01:33,330][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s_0] [ecjbud] creating index, cause [clone_index], templates [], shards [1]/[1]
  1> [2021-03-02T15:01:33,435][INFO ][o.e.c.m.MetadataMappingService] [node_s_0] [ecjbud/3VA-iINcTl6RCqNgz1jHxQ] create_mapping [_doc]
  1> [2021-03-02T15:01:33,509][INFO ][o.e.c.m.MetadataMappingService] [node_s_0] [kglmlq/g1PmmRXuQQ-6VUQ-duXhUA] create_mapping [_doc]
  1> [2021-03-02T15:01:33,633][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s_0] [.rolluptmp-ecjbud/EbxCDhmdS-SltrHrWilOOQ] deleting index
  1> [2021-03-02T15:01:33,768][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s_0] [.rolluptmp-kglmlq/3YQHCIUPTW6YJhWTeug1Og] deleting index
  1> [2021-03-02T15:01:33,780][INFO ][o.e.x.r.v.RollupActionSingleNodeTests] [testAvgMetric] after test
  2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:rollup:test' --tests "org.elasticsearch.xpack.rollup.v2.RollupActionSingleNodeTests.testAvgMetric" -Dtests.seed=5EC5B0A6CCAA9904 -Dtests.security.manager=true -Dtests.locale=is -Dtests.timezone=Africa/Lubumbashi -Druntime.java=8
  2> [.rolluptmp-kglmlq/3YQHCIUPTW6YJhWTeug1Og] IndexNotFoundException[no such index [.rolluptmp-kglmlq]]
        at __randomizedtesting.SeedInfo.seed([5EC5B0A6CCAA9904:73AACC3E1E9D7A9D]:0)
        at org.elasticsearch.cluster.metadata.Metadata.getIndexSafe(Metadata.java:719)
        at org.elasticsearch.cluster.metadata.MetadataDeleteIndexService.deleteIndices(MetadataDeleteIndexService.java:80)
        at org.elasticsearch.cluster.metadata.MetadataDeleteIndexService$1.execute(MetadataDeleteIndexService.java:67)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:48)
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:691)
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:313)
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:208)
        at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:62)
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:140)
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:139)
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:177)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:241)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
@DaveCTurner DaveCTurner added >test-failure Triaged test failures from CI :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data labels Mar 2, 2021
@elasticmachine elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Mar 2, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

DaveCTurner added a commit that referenced this issue Mar 2, 2021
DaveCTurner added a commit that referenced this issue Mar 2, 2021
@DaveCTurner
Copy link
Contributor Author

Muted in master, 7.x and 7.12.

DaveCTurner added a commit that referenced this issue Mar 2, 2021
talevy added a commit to talevy/elasticsearch that referenced this issue Mar 17, 2021
this commit unmutes the tests so that more CI runs with
new code that includes better error messages will highlight why
tests are flaky.

relates elastic#69506, elastic#69799, elastic#69733.
talevy added a commit to talevy/elasticsearch that referenced this issue Mar 17, 2021
this commit unmutes the tests so that more CI runs with
new code that includes better error messages will highlight why
tests are flaky.

relates elastic#69506, elastic#69799, elastic#69733.
talevy added a commit to talevy/elasticsearch that referenced this issue Mar 17, 2021
this commit unmutes the tests so that more CI runs with
new code that includes better error messages will highlight why
tests are flaky.

relates elastic#69506, elastic#69799, elastic#69733.
talevy added a commit that referenced this issue Mar 18, 2021
this commit unmutes the tests so that more CI runs with
new code that includes better error messages will highlight why
tests are flaky.

relates #69506, #69799, #69733.
@talevy
Copy link
Contributor

talevy commented Mar 18, 2021

Unmuted in master/8.0 for more debugging information. Difficult to reproduce and find out what is going on locally. #70519 unmuted. Will mute again if still causing issues and enough logging information was collected.

easyice pushed a commit to easyice/elasticsearch that referenced this issue Mar 25, 2021
mark-vieira pushed a commit to mark-vieira/elasticsearch that referenced this issue Mar 25, 2021
this commit unmutes the tests so that more CI runs with
new code that includes better error messages will highlight why
tests are flaky.

relates elastic#69506, elastic#69799, elastic#69733.
@csoulios csoulios self-assigned this May 19, 2022
csoulios added a commit that referenced this issue May 25, 2022
This PR implements downsampling operation on time series indices.

The PR creates a _rollup endpoint that allows users to downsample an index and can be
accessed by the following call:

POST /<source_index>/_rollup/<rollup_index>
{
    "fixed_interval": "1d"
}

Requirements

An index can be downsampled if all of the following requirements are met:

    Must be a time series index (have the index.mode: time_series index setting)
    Must not be writeable (have the index.blocks.write: true index setting)
    Must have dimension fields marked with mapping parameter time_series_dimension: true
    Must have metric fields marked with mapping parameter time_series_metric

Relates to #74660

Fixes #65769
Fixes #69799
Finally, this PR is based on the code written for #64900
salvatore-campagna pushed a commit to salvatore-campagna/elasticsearch that referenced this issue May 26, 2022
This PR implements downsampling operation on time series indices.

The PR creates a _rollup endpoint that allows users to downsample an index and can be
accessed by the following call:

POST /<source_index>/_rollup/<rollup_index>
{
    "fixed_interval": "1d"
}

Requirements

An index can be downsampled if all of the following requirements are met:

    Must be a time series index (have the index.mode: time_series index setting)
    Must not be writeable (have the index.blocks.write: true index setting)
    Must have dimension fields marked with mapping parameter time_series_dimension: true
    Must have metric fields marked with mapping parameter time_series_metric

Relates to elastic#74660

Fixes elastic#65769
Fixes elastic#69799
Finally, this PR is based on the code written for elastic#64900
@jtibshirani
Copy link
Contributor

Reopening since this just failed with a very similar error message: https://gradle-enterprise.elastic.co/s/fklmbfwam4q2w.

Failure excerpt:

REPRODUCE WITH: ./gradlew ':x-pack:plugin:rollup:test' --tests "org.elasticsearch.xpack.rollup.v2.RollupActionSingleNodeTests.testCannotRollupIndexWithNoMetrics" -Dtests.seed=92F5C5552AE7A49E -Dtests.locale=en-SG -Dtests.timezone=Europe/Skopje -Druntime.java=17

org.elasticsearch.xpack.rollup.v2.RollupActionSingleNodeTests > testCannotRollupIndexWithNoMetrics FAILED
    [ptddxj/TpWo84wSSDGt9s7gy9YBLQ] org.elasticsearch.index.IndexNotFoundException: no such index [ptddxj]
        at __randomizedtesting.SeedInfo.seed([92F5C5552AE7A49E:F2B6321CAB25A368]:0)
        at app//org.elasticsearch.cluster.metadata.Metadata.getIndexSafe(Metadata.java:838)
        at app//org.elasticsearch.cluster.metadata.MetadataDeleteIndexService.deleteIndices(MetadataDeleteIndexService.java:89)
        at app//org.elasticsearch.cluster.metadata.MetadataDeleteIndexService$1.execute(MetadataDeleteIndexService.java:70)
        at app//org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:482)
        at app//org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:908)
        at app//org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:878)
        at app//org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:248)
        at app//org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:156)
        at app//org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:110)
        at app//org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:148)
        at app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:709)
        at app//org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:260)
        at app//org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:223)
        at java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base@17.0.2/java.lang.Thread.run(Thread.java:833)

@jtibshirani jtibshirani reopened this Jun 1, 2022
@csoulios
Copy link
Contributor

csoulios commented Jun 2, 2022

Please don't mute this test ^^^

I would like to monitor its behavior. Thanks

csoulios added a commit to csoulios/elasticsearch that referenced this issue Jun 2, 2022
csoulios added a commit that referenced this issue Jun 3, 2022
…tion (#87333)

Looks like the failure reported at #69799 (comment) happens because of
randomly generated index names conflict between two tests.

    Add more random numbers to test names, so that the probability of conflicts is negligible
    Generate source and rollup index names more carefully so that they are logged properly.
    Cleanup test code

Fixes #69799
csoulios added a commit to csoulios/elasticsearch that referenced this issue Jun 3, 2022
…tion (elastic#87333)

Looks like the failure reported at elastic#69799 (comment) happens because of
randomly generated index names conflict between two tests.

    Add more random numbers to test names, so that the probability of conflicts is negligible
    Generate source and rollup index names more carefully so that they are logged properly.
    Cleanup test code

Fixes elastic#69799
elasticsearchmachine pushed a commit that referenced this issue Jun 3, 2022
…tion (#87333) (#87384)

Looks like the failure reported at #69799 (comment) happens because of
randomly generated index names conflict between two tests.

    Add more random numbers to test names, so that the probability of conflicts is negligible
    Generate source and rollup index names more carefully so that they are logged properly.
    Cleanup test code

Fixes #69799
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) >test-failure Triaged test failures from CI
Projects
None yet
5 participants