Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Fleet Serverless Security API Integration Tests.x-pack/test_serverless/api_integration/test_suites/security/fleet/fleet·ts - fleet rejects request to create a new fleet server hosts if host url is different from default #176399

Closed
kibanamachine opened this issue Feb 7, 2024 · 6 comments · Fixed by #176614 or #176808
Closed

Failing test: Fleet Serverless Security API Integration Tests.x-pack/test_serverless/api_integration/test_suites/security/fleet/fleet·ts - fleet rejects request to create a new fleet server hosts if host url is different from default #176399

kibanamachine opened this issue Feb 7, 2024 · 6 comments · Fixed by #176614 or #176808
Assignees
@jillguyonnet
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Feb 7, 2024 •
edited
Loading

A test failed on a tracked branch

JestAssertionError: expect(received).toEqual(expected) // deep equality

- Expected  - 1
+ Received  + 1

  Object {
    "error": "Forbidden",
-   "message": "Fleet server host must have default URL in serverless: https://localhost:8220",
+   "message": "Fleet server host must have default URL in serverless",
    "statusCode": 403,
  }
    at Context.<anonymous> (fleet.ts:26:20)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:73:16) {
  matcherResult: {
    actual: {
      statusCode: 403,
      error: 'Forbidden',
      message: 'Fleet server host must have default URL in serverless'
    },
    expected: {
      statusCode: 403,
      error: 'Forbidden',
      message: 'Fleet server host must have default URL in serverless: https://localhost:8220'
    },
    message: '\x1B[2mexpect(\x1B[22m\x1B[31mreceived\x1B[39m\x1B[2m).\x1B[22mtoEqual\x1B[2m(\x1B[22m\x1B[32mexpected\x1B[39m\x1B[2m) // deep equality\x1B[22m\n' +
      '\n' +
      '\x1B[32m- Expected  - 1\x1B[39m\n' +
      '\x1B[31m+ Received  + 1\x1B[39m\n' +
      '\n' +
      '\x1B[2m  Object {\x1B[22m\n' +
      '\x1B[2m    "error": "Forbidden",\x1B[22m\n' +
      '\x1B[32m-   "message": "Fleet server host must have default URL in serverless: https://localhost:8220",\x1B[39m\n' +
      '\x1B[31m+   "message": "Fleet server host must have default URL in serverless",\x1B[39m\n' +
      '\x1B[2m    "statusCode": 403,\x1B[22m\n' +
      '\x1B[2m  }\x1B[22m',
    name: 'toEqual',
    pass: false
  }
}

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Feb 7, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Feb 7, 2024
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@nickpeihl nickpeihl added the Team:Fleet Team label for Observability Data Collection Fleet team label Feb 7, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Feb 7, 2024
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

mistic added a commit that referenced this issue Feb 7, 2024
@mistic
skip flaky suite (#176399)
3a15094
@mistic
Copy link
Member

mistic commented Feb 7, 2024

Skipped.

main: 3a15094

fkanout pushed a commit to fkanout/kibana that referenced this issue Feb 8, 2024
@mistic @fkanout
skip flaky suite (elastic#176399)
b11d1d1
@jillguyonnet jillguyonnet self-assigned this Feb 12, 2024
@jillguyonnet jillguyonnet mentioned this issue Feb 12, 2024
Merged
2 tasks
@jillguyonnet
Copy link
Contributor

Reopening as sister issue #176352 was reopened.

@jillguyonnet jillguyonnet reopened this Feb 13, 2024
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@mistic @CoenWarmer
skip flaky suite (elastic#176399)
26c5f9b
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@jillguyonnet @CoenWarmer
[Fleet] Fix flaky serverless API integration tests (elastic#176614)
f50fd0c 
## Summary

Closes elastic#176352
Closes elastic#176399

elastic#175315 added the possibility to
configure new Fleet Server hosts in serverless, with the constraint that
the host URL must match the default URL. The API integration tests
written to test this have been flaky, probably due to request timeout
when fetching all Fleet Server hosts. This PR improves this by directly
retrieving the default Fleet Server host by id.

This fix has been tested using the Flaky Test Runner Pipeline, with 25
test runs for observability and security project types:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5140

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@mistic @CoenWarmer
skip flaky suite (elastic#176399)
5a9922a
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@jillguyonnet @CoenWarmer
[Fleet] Fix flaky serverless API integration tests (elastic#176614)
6d545c0 
## Summary

Closes elastic#176352
Closes elastic#176399

elastic#175315 added the possibility to
configure new Fleet Server hosts in serverless, with the constraint that
the host URL must match the default URL. The API integration tests
written to test this have been flaky, probably due to request timeout
when fetching all Fleet Server hosts. This PR improves this by directly
retrieving the default Fleet Server host by id.

This fix has been tested using the Flaky Test Runner Pipeline, with 25
test runs for observability and security project types:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5140

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
@jillguyonnet jillguyonnet mentioned this issue Feb 19, 2024
Merged
2 tasks
jillguyonnet added a commit that referenced this issue Feb 21, 2024
@jillguyonnet
[Fleet] Add retry logic to serverless API check (#176808)
83e64be 
## Summary

Closes #176352
Closes #176399

#175315 added the possibility to
configure new Fleet Server hosts in serverless, with the constraint that
the host URL must match the default URL. The API integration tests
written to test this have been flaky, due to failure retrieving the
default Fleet Server host or default Elasticsearch output from saved
objects. This PR adds retry in the tests.

Note: I have tried adding retry logic in the API handlers but kept
hitting test flakiness.

This fix has been tested using the Flaky Test Runner Pipeline, with
48/49 test runs for observability and security project types:
🟢
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5218
🟢
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5222
🟢
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5225

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@mistic @fkanout
skip flaky suite (elastic#176399)
389ad3c
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@jillguyonnet @fkanout
[Fleet] Fix flaky serverless API integration tests (elastic#176614)
5f96f66 
## Summary

Closes elastic#176352
Closes elastic#176399

elastic#175315 added the possibility to
configure new Fleet Server hosts in serverless, with the constraint that
the host URL must match the default URL. The API integration tests
written to test this have been flaky, probably due to request timeout
when fetching all Fleet Server hosts. This PR improves this by directly
retrieving the default Fleet Server host by id.

This fix has been tested using the Flaky Test Runner Pipeline, with 25
test runs for observability and security project types:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5140

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@jillguyonnet @fkanout
[Fleet] Add retry logic to serverless API check (elastic#176808)
c548765 
## Summary

Closes elastic#176352
Closes elastic#176399

elastic#175315 added the possibility to
configure new Fleet Server hosts in serverless, with the constraint that
the host URL must match the default URL. The API integration tests
written to test this have been flaky, due to failure retrieving the
default Fleet Server host or default Elasticsearch output from saved
objects. This PR adds retry in the tests.

Note: I have tried adding retry logic in the API handlers but kept
hitting test flakiness.

This fix has been tested using the Flaky Test Runner Pipeline, with
48/49 test runs for observability and security project types:
🟢
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5218
🟢
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5222
🟢
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5225

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
jillguyonnet added a commit to jillguyonnet/kibana that referenced this issue Mar 12, 2024
@jillguyonnet
[Fleet] Add retry logic to serverless API check (elastic#176808)
69d52e4 
## Summary

Closes elastic#176352
Closes elastic#176399

elastic#175315 added the possibility to
configure new Fleet Server hosts in serverless, with the constraint that
the host URL must match the default URL. The API integration tests
written to test this have been flaky, due to failure retrieving the
default Fleet Server host or default Elasticsearch output from saved
objects. This PR adds retry in the tests.

Note: I have tried adding retry logic in the API handlers but kept
hitting test flakiness.

This fix has been tested using the Flaky Test Runner Pipeline, with
48/49 test runs for observability and security project types:
🟢
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5218
🟢
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5222
🟢
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5225

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

(cherry picked from commit 83e64be)

# Conflicts:
#	x-pack/test_serverless/api_integration/test_suites/security/fleet/fleet.ts
jillguyonnet added a commit that referenced this issue Mar 12, 2024
@jillguyonnet
[8.13] [Fleet] Add retry logic to serverless API check (#176808) (#17…
4555304 
…8508)

# Backport

This will backport the following commits from `main` to `8.13`:
- [[Fleet] Add retry logic to serverless API check
(#176808)](#176808)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jill
Guyonnet","email":"jill.guyonnet@elastic.co"},"sourceCommit":{"committedDate":"2024-02-21T09:14:10Z","message":"[Fleet]
Add retry logic to serverless API check (#176808)\n\n##
Summary\r\n\r\nCloses
#176352
#176399
added the possibility to\r\nconfigure new Fleet Server hosts in
serverless, with the constraint that\r\nthe host URL must match the
default URL. The API integration tests\r\nwritten to test this have been
flaky, due to failure retrieving the\r\ndefault Fleet Server host or
default Elasticsearch output from saved\r\nobjects. This PR adds retry
in the tests.\r\n\r\nNote: I have tried adding retry logic in the API
handlers but kept\r\nhitting test flakiness.\r\n\r\nThis fix has been
tested using the Flaky Test Runner Pipeline, with\r\n48/49 test runs for
observability and security project
types:\r\n🟢\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5218\r\n🟢\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5222\r\n🟢\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5225\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests
changed","sha":"83e64be5d6730ec1f22ade806192bf1eab54751a","branchLabelMapping":{"^v8.14.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","backport:prev-minor","v8.14.0"],"number":176808,"url":"#176808
Add retry logic to serverless API check (#176808)\n\n##
Summary\r\n\r\nCloses
#176352
#176399
added the possibility to\r\nconfigure new Fleet Server hosts in
serverless, with the constraint that\r\nthe host URL must match the
default URL. The API integration tests\r\nwritten to test this have been
flaky, due to failure retrieving the\r\ndefault Fleet Server host or
default Elasticsearch output from saved\r\nobjects. This PR adds retry
in the tests.\r\n\r\nNote: I have tried adding retry logic in the API
handlers but kept\r\nhitting test flakiness.\r\n\r\nThis fix has been
tested using the Flaky Test Runner Pipeline, with\r\n48/49 test runs for
observability and security project
types:\r\n🟢\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5218\r\n🟢\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5222\r\n🟢\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5225\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests
changed","sha":"83e64be5d6730ec1f22ade806192bf1eab54751a"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.14.0","labelRegex":"^v8.14.0$","isSourceBranch":true,"state":"MERGED","url":"#176808
Add retry logic to serverless API check (#176808)\n\n##
Summary\r\n\r\nCloses
#176352
#176399
added the possibility to\r\nconfigure new Fleet Server hosts in
serverless, with the constraint that\r\nthe host URL must match the
default URL. The API integration tests\r\nwritten to test this have been
flaky, due to failure retrieving the\r\ndefault Fleet Server host or
default Elasticsearch output from saved\r\nobjects. This PR adds retry
in the tests.\r\n\r\nNote: I have tried adding retry logic in the API
handlers but kept\r\nhitting test flakiness.\r\n\r\nThis fix has been
tested using the Flaky Test Runner Pipeline, with\r\n48/49 test runs for
observability and security project
types:\r\n🟢\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5218\r\n🟢\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5222\r\n🟢\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5225\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests
changed","sha":"83e64be5d6730ec1f22ade806192bf1eab54751a"}}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jillguyonnet jillguyonnet

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
5 participants
@nickpeihl @mistic @elasticmachine @jillguyonnet @kibanamachine