Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - when splitting .kibana into multiple indices and one clone fails after resolving the problem and retrying the migration completes successfully #163253

kibanamachine · 2023-08-07T08:51:18Z

A test failed on a tracked branch

Jest encountered an unexpected token

Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.

Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.

By default "node_modules" folder is ignored by transformers.

Here's what you can do:
 • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
 • If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript
 • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
 • If you need a custom transformation specify a "transform" option in your config.
 • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

You'll find more details and examples of these config options in the docs:
https://jestjs.io/docs/configuration
For information about custom transformations, see:
https://jestjs.io/docs/code-transformation

Details:

SyntaxError: /var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/x-pack/plugins/cases/common/ui/types.ts: Identifier 'CaseMetricsFeature' has already been declared. (39:2)

  37 |   UserActionFindRequestTypes,
  38 |   UserActionFindResponse,
> 39 |   CaseMetricsFeature,
     |   ^
  40 |   CasesMetricsResponse,
  41 |   SingleCaseMetricsResponse,
  42 | } from '../types/api';
    at instantiate (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parse-error/credentials.ts:62:21)
    at instantiate (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parse-error.ts:60:12)
    at TypeScriptParserMixin.toParseError [as raise] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/tokenizer/index.ts:1464:19)
    at TypeScriptScopeHandler.raise [as declareName] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/plugins/typescript/scope.ts:83:21)
    at TypeScriptParserMixin.declareName [as declareNameFromIdentifier] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/lval.ts:705:16)
    at TypeScriptParserMixin.declareNameFromIdentifier [as checkIdentifier] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/lval.ts:700:12)
    at TypeScriptParserMixin.checkIdentifier [as checkLVal] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/lval.ts:615:12)
    at TypeScriptParserMixin.checkLVal [as finishImportSpecifier] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:3000:10)
    at TypeScriptParserMixin.finishImportSpecifier (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:3208:17)
    at TypeScriptParserMixin.parseImportSpecifier (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/plugins/typescript/index.ts:4024:20)
    at TypeScriptParserMixin.parseImportSpecifier [as parseNamedImportSpecifiers] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:3167:36)
    at TypeScriptParserMixin.parseNamedImportSpecifiers (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:2946:39)
    at TypeScriptParserMixin.parseImport (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/plugins/typescript/index.ts:2702:32)
    at TypeScriptParserMixin.parseImport (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:571:25)
    at TypeScriptParserMixin.parseStatementContent (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/plugins/typescript/index.ts:2857:20)
    at TypeScriptParserMixin.parseStatementContent [as parseStatementLike] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:416:17)
    at TypeScriptParserMixin.parseStatementLike [as parseModuleItem] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:353:17)
    at TypeScriptParserMixin.parseModuleItem [as parseBlockOrModuleBlockBody] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:1362:16)
    at TypeScriptParserMixin.parseBlockOrModuleBlockBody [as parseBlockBody] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:1336:10)
    at TypeScriptParserMixin.parseBlockBody [as parseProgram] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:226:10)
    at TypeScriptParserMixin.parseProgram [as parseTopLevel] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/statement.ts:208:25)
    at TypeScriptParserMixin.parseTopLevel (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/parser/index.ts:45:10)
    at TypeScriptParserMixin.parse (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/plugins/typescript/index.ts:3974:20)
    at parse (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/parser/src/index.ts:67:38)
    at parser (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/core/src/parser/index.ts:28:19)
    at parser.next (<anonymous>)
    at normalizeFile (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/core/src/transformation/normalize-file.ts:51:24)
    at normalizeFile.next (<anonymous>)
    at run (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/core/src/transformation/index.ts:38:36)
    at run.next (<anonymous>)
    at transform (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/core/src/transform.ts:29:20)
    at transform.next (<anonymous>)
    at evaluateSync (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/gensync/index.js:251:28)
    at sync (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/gensync/index.js:89:14)
    at fn (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/core/src/errors/rewrite-stack-trace.ts:97:14)
    at transformSync (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/@babel/core/src/transform.ts:66:52)
    at Object.process (/var/lib/buildkite-agent/builds/kb-n2-4-spot-f484cd9f26dcc6f4/elastic/kibana-on-merge/kibana/node_modules/babel-j
[report_failure] output truncated to 8192 characters

First failure: CI Build - main

The text was updated successfully, but these errors were encountered:

elasticmachine · 2023-08-07T08:51:43Z

Pinging @elastic/kibana-core (Team:Core)

afharo · 2023-08-24T11:22:23Z

Caused by the same reason as #163273. Closing.

kibanamachine · 2023-09-29T16:20:30Z

New failure: CI Build - main

kibanamachine · 2023-09-29T16:21:13Z

New failure: CI Build - main

kibanamachine · 2023-09-29T16:21:57Z

New failure: CI Build - main

kibanamachine · 2023-09-29T16:27:10Z

New failure: CI Build - main

kibanamachine · 2023-12-01T12:46:56Z

New failure: CI Build - main

kibanamachine · 2023-12-05T18:16:59Z

New failure: CI Build - main

kibanamachine · 2023-12-11T17:28:26Z

New failure: CI Build - main

pgayvallet · 2023-12-13T07:11:48Z

Actual error from the last CI build:

FAIL  src/core/server/integration_tests/saved_objects/migrations/group3/split_failed_to_clone.test.ts (63.424 s)
--
  | ● when splitting .kibana into multiple indices and one clone fails › after resolving the problem and retrying the migration completes successfully
  |  
  | expect(received).rejects.toThrowError(expected)
  |  
  | Expected pattern: /cluster_shard_limit_exceeded/
  | Received message: "Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 404, method: GET, url: /_tasks/VpXIdVnGRjishpLCm0CrzA%3A431?wait_for_completion=true&timeout=120s error: [resource_not_found_exception]: task [VpXIdVnGRjishpLCm0CrzA:431] isn't running and hasn't stored its results,"

kibanamachine · 2023-12-13T17:02:42Z

New failure: CI Build - main

kibanamachine · 2023-12-13T17:18:40Z

New failure: CI Build - main

delanni · 2023-12-14T10:00:13Z

/skip

delanni · 2023-12-14T10:00:40Z

Started to break on PRs and main:
960b1a1

kibanamachine · 2024-05-13T12:43:33Z

New failure: kibana-on-merge - main

kibanamachine · 2024-05-13T18:10:50Z

New failure: kibana-on-merge - main

mistic · 2024-05-14T03:41:52Z

Skipped.

main: b122b46

afharo · 2024-05-14T07:59:06Z

This is a new error:

Error: expect(received).rejects.toThrowError(expected)

Expected pattern: /cluster_shard_limit_exceeded/
Received message: "Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 404, method: GET, url: /_tasks/VpXIdVnGRjishpLCm0CrzA%3A458?wait_for_completion=true&timeout=120s error: [resource_not_found_exception]: task [VpXIdVnGRjishpLCm0CrzA:458] isn't running and hasn't stored its results,"

I'll take a look at what could cause them.

UPDATE: It looks like the same error as reported in #163253 (comment)

gsoldevila · 2024-06-07T10:42:45Z

I have identified the place where the error happens, it's the CLEANUP_UNKNOWN_AND_EXCLUDED_WAIT_FOR_TASK.
When performing a compatible migration, we update documents in place, but we also cleanup those that are unknown or excluded.

First, a deleteByQuery is issued as part of the CLEANUP_UNKNOWN_AND_EXCLUDED step.
- This is deferred and we get a taskId.
We then query this task "waiting for it", with a timeout of 120s. This is the request logged in the error as:
- GET, url: /_tasks/VpXIdVnGRjishpLCm0CrzA%3A458?wait_for_completion=true&timeout=120s
- error: [resource_not_found_exception]: task [VpXIdVnGRjishpLCm0CrzA:458] isn't running and hasn't stored its results.

Even though we have a retry mechanism to keep trying until task completes, this time we're getting a 404 Not Found.
I've searched for the error message, and ES is throwing this error in 2 places in this file:

A) the task index has not yet been created.
B) the task was not found (false == response.isExists()).

If we take a step back and look at the test, Rudolf is trying to make migrations fail on purpose, by doing:

  // cause a failure when cloning .kibana_slow_clone_* indices 
  await client.cluster.putSettings({ persistent: { 'cluster.max_shards_per_node': 15 } });

after which, we expect

  await expect(runMigrationsWhichFailsWhenCloning()).rejects.toThrowError(
    /cluster_shard_limit_exceeded/
  );

However, it seems that we might be failing before we attempt to create enough SO indices to cause the expected failure.
My theory is that this max_shards_per_node can be messing up with ES, not allowing it to create its internal _tasks index.

## Summary Addresses #163253 (see [comment](#163253 (comment))) TLDR: The PR renames the task manager SO index, forcing a reindex, and preventing the `deleteByQuery`. **Long story** * The fact that `.kibana_task_manager` SO index is compatible since 7.14.0 triggers a compatible migration path for that migrator. * This includes a cleanup step for old and excluded SO types. * This translates to a `deleteByQuery()`, which creates an async task on ES. * ES stores this task internally in an index. --- * The test at stakes is carefully limiting **shards_per_node**, effectively limiting amount of indices that we can create. * All migrators run in parallel, so whilst some of them are cloning (to the expected failure), the `.kibana_task_manager` migrator is/was running a compatible migration, which also requires an extra index. * This creates a race condition, where either the _clone_ operations (expected!) or the _deleteByQuery_ (flaky!) can fail arbitrarily, depending on who attempts to create the index first, hence the flakiness (well, this is my theory).

gsoldevila · 2024-06-18T13:26:40Z

Fixed by #185005

kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Aug 7, 2023

botelastic bot added the needs-team Issues missing a team label label Aug 7, 2023

kibanamachine added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Aug 7, 2023

botelastic bot removed the needs-team Issues missing a team label label Aug 7, 2023

afharo closed this as completed Aug 24, 2023

kibanamachine reopened this Sep 29, 2023

pgayvallet closed this as completed Oct 2, 2023

kibanamachine reopened this Dec 1, 2023

pgayvallet closed this as completed Dec 5, 2023

kibanamachine reopened this Dec 5, 2023

pgayvallet closed this as completed Dec 6, 2023

kibanamachine reopened this Dec 11, 2023

kibanamachine added blocker v8.13.0 labels Dec 14, 2023

delanni added a commit that referenced this issue Dec 14, 2023

skip flaky suite (#163253)

960b1a1

afharo added the skipped-test label May 8, 2024

afharo added a commit to afharo/kibana that referenced this issue May 8, 2024

Unskip elastic#163253

d949ab5

afharo mentioned this issue May 8, 2024

Unskip #163253 #182971

Merged

1 task

afharo self-assigned this May 8, 2024

afharo closed this as completed in #182971 May 13, 2024

afharo added a commit that referenced this issue May 13, 2024

Unskip #163253 (#182971)

1c3ae4c

kibanamachine removed skipped-test v8.13.0 blocker labels May 13, 2024

kibanamachine reopened this May 13, 2024

mistic added a commit that referenced this issue May 14, 2024

skip flaky suite (#163253)

b122b46

mistic added blocker skipped-test v8.15.0 labels May 14, 2024

gsoldevila self-assigned this Jun 7, 2024

gsoldevila mentioned this issue Jun 7, 2024

Force .kibana_task_manager reindex by specifying a new name #185005

Merged

gsoldevila closed this as completed Jun 18, 2024

kibanamachine removed skipped-test blocker v8.15.0 labels Jun 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - when splitting .kibana into multiple indices and one clone fails after resolving the problem and retrying the migration completes successfully #163253

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - when splitting .kibana into multiple indices and one clone fails after resolving the problem and retrying the migration completes successfully #163253

kibanamachine commented Aug 7, 2023 •

edited

Loading

elasticmachine commented Aug 7, 2023

afharo commented Aug 24, 2023

kibanamachine commented Sep 29, 2023

kibanamachine commented Sep 29, 2023

kibanamachine commented Sep 29, 2023

kibanamachine commented Sep 29, 2023

kibanamachine commented Dec 1, 2023

kibanamachine commented Dec 5, 2023

kibanamachine commented Dec 11, 2023

pgayvallet commented Dec 13, 2023

kibanamachine commented Dec 13, 2023

kibanamachine commented Dec 13, 2023

delanni commented Dec 14, 2023

delanni commented Dec 14, 2023 •

edited

Loading

kibanamachine commented May 13, 2024

kibanamachine commented May 13, 2024

mistic commented May 14, 2024

afharo commented May 14, 2024 •

edited

Loading

gsoldevila commented Jun 7, 2024

gsoldevila commented Jun 18, 2024

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - when splitting .kibana into multiple indices and one clone fails after resolving the problem and retrying the migration completes successfully #163253

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - when splitting .kibana into multiple indices and one clone fails after resolving the problem and retrying the migration completes successfully #163253

Comments

kibanamachine commented Aug 7, 2023 • edited Loading

elasticmachine commented Aug 7, 2023

afharo commented Aug 24, 2023

kibanamachine commented Sep 29, 2023

kibanamachine commented Sep 29, 2023

kibanamachine commented Sep 29, 2023

kibanamachine commented Sep 29, 2023

kibanamachine commented Dec 1, 2023

kibanamachine commented Dec 5, 2023

kibanamachine commented Dec 11, 2023

pgayvallet commented Dec 13, 2023

kibanamachine commented Dec 13, 2023

kibanamachine commented Dec 13, 2023

delanni commented Dec 14, 2023

delanni commented Dec 14, 2023 • edited Loading

kibanamachine commented May 13, 2024

kibanamachine commented May 13, 2024

mistic commented May 14, 2024

afharo commented May 14, 2024 • edited Loading

gsoldevila commented Jun 7, 2024

gsoldevila commented Jun 18, 2024

kibanamachine commented Aug 7, 2023 •

edited

Loading

delanni commented Dec 14, 2023 •

edited

Loading

afharo commented May 14, 2024 •

edited

Loading