Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Obs AI Assistant] Fix flaky e2e test (contextual insights for APM errors) #184642

Merged
merged 11 commits into from
Jun 26, 2024
Merged

[Obs AI Assistant] Fix flaky e2e test (contextual insights for APM errors) #184642

merged 11 commits into from
Jun 26, 2024

Conversation

sorenlouv
Copy link
Member

@sorenlouv sorenlouv commented Jun 3, 2024 •
edited
Loading

Closes #184071 #184029

Fixes flaky e2e tests

  • set static error.grouping_key. This required a small change in synthtrace.
  • retry clicking on contextual insight component to ensure the accordion is open.

@sorenlouv sorenlouv requested review from a team as code owners June 3, 2024 10:51
@botelastic botelastic bot added ci:project-deploy-observability Create an Observability project Team:Obs AI Assistant Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Jun 3, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

@sorenlouv sorenlouv marked this pull request as draft June 3, 2024 10:51
@apmmachine
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@sorenlouv
Copy link
Member Author

sorenlouv commented Jun 3, 2024 •
edited
Loading

Started flaky test runner:

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6187

[❌] x-pack/test/observability_ai_assistant_functional/enterprise/config.ts: 24/100 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6188

[❌] x-pack/test/observability_ai_assistant_functional/enterprise/config.ts: 0/25 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6189

[❌] x-pack/test/observability_ai_assistant_functional/enterprise/config.ts: 0/25 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6191

[❌] x-pack/test/observability_ai_assistant_functional/enterprise/config.ts: 0/25 tests passed.

see run history

@sorenlouv
Copy link
Member Author

buildkite test this

1 similar comment
@sorenlouv
Copy link
Member Author

buildkite test this

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6202

[❌] x-pack/test/observability_ai_assistant_functional/enterprise/config.ts: 0/25 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6204

[✅] x-pack/test/observability_ai_assistant_functional/enterprise/config.ts: 25/25 tests passed.

see run history

@sorenlouv sorenlouv marked this pull request as ready for review June 3, 2024 20:43
@sorenlouv sorenlouv changed the title [Obs AI Assistant] Set static grouping key to avoid flaky e2e test [Obs AI Assistant] Fix flaky e2e test (contextual insights for APM errors) Jun 4, 2024
sorenlouv
sorenlouv commented Jun 4, 2024
View reviewed changes
x-pack/test/observability_ai_assistant_functional/tests/contextual_insights/index.spec.ts Outdated
Comment on lines 100 to 108
async function openContextualInsights() {
await retry.try(async () => {
await testSubjects.click(ui.pages.contextualInsights.button);
const isOpen =
(await (
await find.byCssSelector(`[aria-controls="${ui.pages.contextualInsights.container}"]`)
).getAttribute('aria-expanded')) === 'true';
expect(isOpen).to.be(true);
});
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the culprit of the failure. The accordion was clicked before it was ready/interactive. This is somewhat ugly but feels robust.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i always found retry.try to be slow as it takes the default config timeout which is of 30 sec making the test fails very late.
There is an alternate called

   await retry.tryForTime(TIME_IN_MILLISECONDS, async () => {});

With this, i limit my test to 5*1000ms which makes failures faster. Its not a hard rule, sometimes tests would need more time, but the default time of 30 seconds definitely makes me feel that something is wrong with the test.

@sorenlouv sorenlouv enabled auto-merge (squash) June 4, 2024 11:54
@sorenlouv sorenlouv disabled auto-merge June 5, 2024 09:14
@sorenlouv
Copy link
Member Author

buildkite test this

@sorenlouv
Fix tsc
0895673
@sorenlouv
Merge branch 'main' of github.com:elastic/kibana into fix-flaky-funct…
9f48bc6 
…ional-test

# Conflicts:
#	packages/kbn-apm-synthtrace-client/src/lib/apm/apm_error.ts
@sorenlouv
Remove unused sleep
1b1161a
@sorenlouv sorenlouv enabled auto-merge (squash) June 25, 2024 09:30
@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6389

[✅] x-pack/test/observability_ai_assistant_functional/enterprise/config.ts: 25/25 tests passed.

see run history

@kibana-ci
Copy link
Collaborator

kibana-ci commented Jun 26, 2024 •
edited
Loading

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/apm-synthtrace-client 191 192 +1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
observabilityAIAssistant 17.7KB 17.7KB +9.0B
Unknown metric groups

API count

id before after diff
@kbn/apm-synthtrace-client 191 192 +1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@sorenlouv sorenlouv merged commit cf47eb5 into elastic:main Jun 26, 2024
22 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Jun 26, 2024
@crespocarlos crespocarlos mentioned this pull request Jul 10, 2024
Merged
elasticmachine added a commit to nkhristinin/kibana that referenced this pull request Jul 11, 2024
@crespocarlos @elasticmachine
[APM] Unskip APM alerting tests (elastic#188006)
5df7bb8 
fixes [elastic#176948](elastic#176948)
fixes [elastic#177061](elastic#177061)
fixes [elastic#177104](elastic#177104)

## Summary

I managed to reproduce the problem when `error_grouping_key` was null.
It was probably fixed here elastic#184642


https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6512

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jul 11, 2024
@crespocarlos
[APM] Unskip APM alerting tests (elastic#188006)
1f89c50 
fixes [elastic#176948](elastic#176948)
fixes [elastic#177061](elastic#177061)
fixes [elastic#177104](elastic#177104)

## Summary

I managed to reproduce the problem when `error_grouping_key` was null.
It was probably fixed here elastic#184642

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6512

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit 5df7bb8)
kibanamachine added a commit that referenced this pull request Jul 11, 2024
@kibanamachine @crespocarlos
[8.15] [APM] Unskip APM alerting tests (#188006) (#188095)
ba8805c 
# Backport

This will backport the following commits from `main` to `8.15`:
- [[APM] Unskip APM alerting tests
(#188006)](#188006)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Carlos
Crespo","email":"crespocarlos@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-07-11T11:30:35Z","message":"[APM]
Unskip APM alerting tests (#188006)\n\nfixes
[#176948](#176948
[#177061](#177061
[#177104](#177104
Summary\r\n\r\nI managed to reproduce the problem when
`error_grouping_key` was null.\r\nIt was probably fixed here
#184642:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"5df7bb87c0a9475d7f860fb45437c4624a2bd0e5","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services","v8.16.0"],"title":"[APM]
Unskip APM alerting
tests","number":188006,"url":"#188006
Unskip APM alerting tests (#188006)\n\nfixes
[#176948](#176948
[#177061](#177061
[#177104](#177104
Summary\r\n\r\nI managed to reproduce the problem when
`error_grouping_key` was null.\r\nIt was probably fixed here
#184642:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"5df7bb87c0a9475d7f860fb45437c4624a2bd0e5"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"#188006
Unskip APM alerting tests (#188006)\n\nfixes
[#176948](#176948
[#177061](#177061
[#177104](#177104
Summary\r\n\r\nI managed to reproduce the problem when
`error_grouping_key` was null.\r\nIt was probably fixed here
#184642:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"5df7bb87c0a9475d7f860fb45437c4624a2bd0e5"}}]}]
BACKPORT-->

Co-authored-by: Carlos Crespo <crespocarlos@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dgieselaar dgieselaar dgieselaar approved these changes

@crespocarlos crespocarlos crespocarlos approved these changes

@achyutjhunjhunwala achyutjhunjhunwala achyutjhunjhunwala approved these changes

Assignees
No one assigned
Labels
apm:review backport:skip This commit does not require backporting ci:project-deploy-observability Create an Observability project release_note:fix Team:Obs AI Assistant Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team v8.15.0
Projects
None yet
Milestone
No milestone
9 participants
@sorenlouv @elasticmachine @apmmachine @kibanamachine @kibana-ci @dgieselaar @crespocarlos @achyutjhunjhunwala @smith