Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: X-Pack API Integration Tests.x-pack/test/api_integration/apis/ml/indices/field_caps·ts - apis Machine Learning system field_caps all fields in index #182837

Closed
kibanamachine opened this issue May 7, 2024 · 6 comments
Closed

Failing test: X-Pack API Integration Tests.x-pack/test/api_integration/apis/ml/indices/field_caps·ts - apis Machine Learning system field_caps all fields in index #182837

kibanamachine opened this issue May 7, 2024 · 6 comments
Assignees
@walterra
Labels
failed-test A test failure on a tracked branch, potentially flaky-test :ml

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented May 7, 2024 •
edited
Loading

A test failed on a tracked branch

Error: Expected number of fields to be 21, but got 22
    at Assertion.assert (/opt/local-ssd/buildkite/builds/kb-n2-4-0407b1859f44662b/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.eql (/opt/local-ssd/buildkite/builds/kb-n2-4-0407b1859f44662b/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility/kibana/node_modules/@kbn/expect/expect.js:244:8)
    at Context.<anonymous> (test/api_integration/apis/ml/indices/field_caps.ts:66:31)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (/opt/local-ssd/buildkite/builds/kb-n2-4-0407b1859f44662b/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:78:16) {
  actual: '22',
  expected: '21',
  showDiff: true
}

First failure: kibana-7-dot-17-es-8-dot-15-forward-compatibility - 7.17
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label May 7, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label May 7, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@botelastic botelastic bot removed the needs-team Issues missing a team label label May 7, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-7-dot-17-es-8-dot-15-forward-compatibility - 7.17

@kibanamachine
Copy link
Contributor Author

New failure: kibana-7-dot-17-es-8-dot-15-forward-compatibility - 7.17

@kibanamachine
Copy link
Contributor Author

New failure: kibana-7-dot-17-es-8-dot-15-forward-compatibility - 7.17

kibanamachine added a commit that referenced this issue May 8, 2024
@kibanamachine
skip failing test suite (#182837)
f7065bd
@kibanamachine
Copy link
Contributor Author

Skipped

7.17: f7065bd

@walterra walterra mentioned this issue May 10, 2024
Merged
2 tasks
walterra added a commit that referenced this issue May 10, 2024
@walterra
[ML] Anomaly Detection: Fix API integration tests for field caps. (#1…
8f04ab6 
…83110)

## Summary

Fixes #182837.
Fixes #182514.

The number of fields returned by the field caps API is different across
ES versions in forward compatibility tests. In ES 8.15.0, the
`_ignored_source` field was added
(elastic/elasticsearch#107567). This fixes the
API integration test for field caps to assert the correct number of
fields across versions. Note that in Kibana `8.15` we refactored away
from using fields caps directly in this way and removed the
corresponding API endpoint and tests
(#182588), that's why there's this
dedicated `7.17` PR to just fix the assertions on the existing test.

To test this locally, the following commands for the functional tests
server and runner can be used to run the tests in different forward
compatibility scenarios:

```
# 7.17 tests server
node scripts/functional_tests_server.js --config x-pack/test/api_integration/config.ts
# 7.17 tests runner
node scripts/functional_test_runner --config x-pack/test/api_integration/config.ts

# 8.14 tests server
ES_SNAPSHOT_MANIFEST="https://storage.googleapis.com/kibana-ci-es-snapshots-daily/8.14.0/manifest-latest-verified.json" node scripts/functional_tests_server.js
# 8.14 tests runner
node scripts/functional_test_runner --config x-pack/test/api_integration/config.ts --es-version=8.14.0-SNAPSHOT

# 8.15 tests server
ES_SNAPSHOT_MANIFEST="https://storage.googleapis.com/kibana-ci-es-snapshots-daily/8.15.0/manifest-latest-verified.json" node scripts/functional_tests_server.js
# 8.15 tests runner
node scripts/functional_test_runner --config x-pack/test/api_integration/config.ts --es-version=8.15.0-SNAPSHOT
```

Note in `7.17` the API integration tests are not split up yet into
several configs so the commands above will run ALL Kibaan API
integration tests.

The command to run the tests server for a specific ES version is also
shared in the buildkite reports, for example:
https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility/builds/20#annotation-es-snapshot-manifest

The versions the compatibility tests will currently run against can be
found here: https://github.com/elastic/kibana/blob/main/versions.json

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
@walterra
Copy link
Contributor

Closing, fixed by #183110

adelisle pushed a commit to Makila-AI/kibana that referenced this issue Aug 5, 2024
@kibanamachine @adelisle
skip failing test suite (elastic#182837)
1d13195
adelisle pushed a commit to Makila-AI/kibana that referenced this issue Aug 5, 2024
@walterra @adelisle
[ML] Anomaly Detection: Fix API integration tests for field caps. (el…
cd960de 
…astic#183110)

## Summary

Fixes elastic#182837.
Fixes elastic#182514.

The number of fields returned by the field caps API is different across
ES versions in forward compatibility tests. In ES 8.15.0, the
`_ignored_source` field was added
(elastic/elasticsearch#107567). This fixes the
API integration test for field caps to assert the correct number of
fields across versions. Note that in Kibana `8.15` we refactored away
from using fields caps directly in this way and removed the
corresponding API endpoint and tests
(elastic#182588), that's why there's this
dedicated `7.17` PR to just fix the assertions on the existing test.

To test this locally, the following commands for the functional tests
server and runner can be used to run the tests in different forward
compatibility scenarios:

```
# 7.17 tests server
node scripts/functional_tests_server.js --config x-pack/test/api_integration/config.ts
# 7.17 tests runner
node scripts/functional_test_runner --config x-pack/test/api_integration/config.ts

# 8.14 tests server
ES_SNAPSHOT_MANIFEST="https://storage.googleapis.com/kibana-ci-es-snapshots-daily/8.14.0/manifest-latest-verified.json" node scripts/functional_tests_server.js
# 8.14 tests runner
node scripts/functional_test_runner --config x-pack/test/api_integration/config.ts --es-version=8.14.0-SNAPSHOT

# 8.15 tests server
ES_SNAPSHOT_MANIFEST="https://storage.googleapis.com/kibana-ci-es-snapshots-daily/8.15.0/manifest-latest-verified.json" node scripts/functional_tests_server.js
# 8.15 tests runner
node scripts/functional_test_runner --config x-pack/test/api_integration/config.ts --es-version=8.15.0-SNAPSHOT
```

Note in `7.17` the API integration tests are not split up yet into
several configs so the commands above will run ALL Kibaan API
integration tests.

The command to run the tests server for a specific ES version is also
shared in the buildkite reports, for example:
https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility/builds/20#annotation-es-snapshot-manifest

The versions the compatibility tests will currently run against can be
found here: https://github.com/elastic/kibana/blob/main/versions.json

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@walterra walterra

Labels
failed-test A test failure on a tracked branch, potentially flaky-test :ml
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@walterra @peteharverson @elasticmachine @kibanamachine