[Fleet] Support output and Fleet server proxies on serverless #175315

jillguyonnet · 2024-01-23T14:20:30Z

Summary

This PR adds support for custom output and Fleet server hosts with proxies in serverless mode:

Proxies are re-enabled in serverless.
It is possible to add custom Fleet server hosts in serverless, with the constraint that the host URL must match the Elasticsearch URL of the default host.
New Elasticsearch outputs must also have the default host URL.

Testing

The below requirements should be tested in serverless mode for observability and security project types:

# elasticsearch
yarn es serverless --kill
# kibana: one of
yarn serverless-oblt
yarn serverless-security

⚠️ In addition, stateful mode should not be affected by any of these changes.

Config

In order to test this change, you will need the following configuration to mirror a serverless setup.

Create a serverless.dev.yml if you don't have one already and set a project id (this is required for Fleet's cloud.isServerlessEnabled to correctly be true):

xpack.cloud.serverless.project_id: test-123

In kibana.devl.yml, make sure the default Fleet server hosts and default output have the expected ids:

xpack.fleet.fleetServerHosts:
  - id: default-fleet-server
    name: Default Fleet server
    is_default: true
    host_urls: ['https://host.docker.internal:8220']
xpack.fleet.outputs:
  - id: es-default-output
    name: Default output
    type: elasticsearch
    is_default: true
    is_default_monitoring: true
    hosts: ['https://host.docker.internal:9200']

Requirements

Checklist

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

Screenshots

Fleet settings (proxies available):

Adding and editing a Fleet server host:

Adding and editing an Elasticsearch output:

apmmachine · 2024-01-23T14:20:50Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

x-pack/plugins/fleet/public/applications/fleet/sections/settings/index.tsx

...plications/fleet/components/fleet_server_instructions/components/fleet_server_hosts_form.tsx

juliaElastic · 2024-01-29T15:25:20Z

Testing this locally, I started a security project. I am able to create proxy but can't assign it to the default fleet server host (API call fails), and I can't create another fleet server host.

Also when I try to create a new ES output with a proxy, can't save it as it says host url is required. The field looks populated and disabled, but fails the form validation.

jillguyonnet · 2024-01-29T15:53:28Z

Thank you for testing @juliaElastic!

I have made some changes locally following @nchaulet's indications, I will check this before committing.

Also thanks for pointing out the different project types, I have only been using observability type so far. Correct me if I'm wrong: we should test this for security and observability types, correct? (as well as testing that there were no changes affecting stateful mode)

juliaElastic · 2024-01-29T15:56:49Z

Also thanks for pointing out the different project types, I have only been using observability type so far. Correct me if I'm wrong: we should test this for security and observability types, correct? (as well as testing that there were no changes affecting stateful mode)

Yes, security and observability project type have fleet enabled, the elasticsearch project type has fleet disabled.

jillguyonnet · 2024-01-31T11:35:10Z

/ci

juliaElastic · 2024-01-31T15:09:29Z

I'm testing with the latest changes, and still seeing a missing host url when trying to add a new elasticsearch output:

For the fleet server hosts, I can see the button now to add a new one, though it comes with an empty host url. Shouldn't it be pre-populated with the default url and not be changeable? I see there is a validation to prevent adding a different url when clicking on Continue.
Also I'm wondering if instead of reusing the Add Fleet Server flyout, we could add a simplified version, something like the Edit Fleet server flyout in create mode. Currently, there is a lot of options on the flyout that are not relevant for serverless (quickstart/advanced, enroll fleet server) and adding a proxy to a new host entry can't be done in one step.

jillguyonnet · 2024-01-31T15:19:16Z

@juliaElastic Thanks again for testing!

Regarding creating an output: does the default output in your setup have id es-default-output? I set the following in my kibana.dev.yml to make it work per the fleet controller config:

xpack.fleet.outputs:
  - id: es-default-output
    name: Default output
    type: elasticsearch
    is_default: true
    is_default_monitoring: true
    hosts: ['https://host.docker.internal:9200']

I guess the same question applies to the default Fleet server host:

xpack.fleet.fleetServerHosts:
  - id: default-fleet-server
    name: Default Fleet server
    is_default: true
    host_urls: ['https://host.docker.internal:8220']
If the ids are correct, can you please share your config?

I really like your suggestion about simplifying the flyout for creating a Fleet server, that's a great point 👍 Let me check it out, I don't think it would be very complicated.

juliaElastic · 2024-01-31T15:58:10Z

Regarding creating an output: does the default output in your setup have id es-default-output?

My bad, I don't have these ids. I'll add to the preconfig locally and test again. Sorry for the false alarm.

jillguyonnet · 2024-01-31T16:02:52Z

My bad, I don't have these ids. I'll add to the preconfig locally and test again. Sorry for the false alarm.

No worries, I understand these ids are fixed in serverless but please correct me if that's not the case. I will update the testing steps to make that clearer.

jillguyonnet · 2024-01-31T17:17:36Z

x-pack/test_serverless/api_integration/test_suites/security/config.ts

@@ -18,4 +18,23 @@ export default createTestConfig({
  // include settings from project controller
  // https://github.com/elastic/project-controller/blob/main/internal/project/security/config/elasticsearch.yml
  esServerArgs: ['xpack.ml.nlp.enabled=false'],
+
+  // Required for Fleet setup.


Maybe it would make sense to extract this to a Fleet-specific config file?

yes, perhaps Fleet related tests could move to test_suites/fleet?

After some thought, I added new config files for Fleet in the test_suites/observability/fleet and test_suites/security/fleet folders in order to preserve the existing structure by project type and avoid adding unnecessary config to other tests. I also added a paragraph in the README on how to run these. Let me know how that sounds.

Oh okay, I didn't realise it means the project type. I'm good with these changes 👍

jillguyonnet · 2024-01-31T17:39:14Z

/ci

nchaulet · 2024-01-31T21:30:30Z

Just did some tests, and it seems to work well for output.

For the fleet server I think the idea of @juliaElastic to reuse the edit fleet server flyout to allow to add one will make sense (I think the flyout almost support it)

nimarezainia · 2024-02-04T09:33:51Z

@nimarezainia Could you please give feedback on the flyouts? The latest screenshots are in the PR description.

@jillguyonnet thank you. Could we add a description why the user can't create a new fleet server - I think being descriptive is better here. So if I may suggest:

"You may create another Fleet Server definition reachable via a proxy. In context of the serverless project, Fleet Service is managed by Elastic. Creation of a new one is therefore not permitted."

jillguyonnet · 2024-02-05T10:55:39Z

@elasticmachine merge upstream

jillguyonnet · 2024-02-05T11:10:49Z

@jillguyonnet thank you. Could we add a description why the user can't create a new fleet server - I think being descriptive is better here. So if I may suggest:

"You may create another Fleet Server definition reachable via a proxy. In context of the serverless project, Fleet Service is managed by Elastic. Creation of a new one is therefore not permitted."

Thank you! I've updated the text and corresponding screenshot.

azasypkin

Change in test/plugin_functional/test_suites/core_plugins/rendering.ts, LGTM.

jillguyonnet · 2024-02-05T13:45:24Z

@elasticmachine merge upstream

nchaulet

just tested the fleet server hosts flyout and it's working as expected. LGTM 🚀

juliaElastic

LGTM

kibana-ci · 2024-02-05T14:56:24Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 2df7a75

Failed CI Steps

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.2MB	1.2MB	+1.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	151.7KB	151.8KB	+121.0B

History

💛 Build #191291 was flaky a3a5562
💚 Build #191127 succeeded 1d20fcc
💔 Build #191035 failed f12b764
💚 Build #190551 succeeded 034a198
💔 Build #190413 failed c2e769d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jillguyonnet

…c#175315) Closes elastic#165672 ### Summary This PR adds support for custom output and Fleet server hosts with proxies in serverless mode: - Proxies are re-enabled in serverless. - It is possible to add custom Fleet server hosts in serverless, with the constraint that the host URL must match the Elasticsearch URL of the default host. - New Elasticsearch outputs must also have the default host URL. ### Testing The below requirements should be tested in serverless mode for observability and security project types: ```sh # elasticsearch yarn es serverless --kill # kibana: one of yarn serverless-oblt yarn serverless-security ``` ⚠️ In addition, stateful mode should **not** be affected by any of these changes. #### Config In order to test this change, you will need the following configuration to mirror a serverless setup. Create a `serverless.dev.yml` if you don't have one already and set a project id (this is required for Fleet's `cloud.isServerlessEnabled` to correctly be `true`): ```yaml xpack.cloud.serverless.project_id: test-123 ``` In `kibana.devl.yml`, make sure the default Fleet server hosts and default output have the expected ids: ```yaml xpack.fleet.fleetServerHosts: - id: default-fleet-server name: Default Fleet server is_default: true host_urls: ['https://host.docker.internal:8220'] xpack.fleet.outputs: - id: es-default-output name: Default output type: elasticsearch is_default: true is_default_monitoring: true hosts: ['https://host.docker.internal:9200'] ``` #### Requirements - [ ] User can create proxy configurations in the Fleet UI and API. - [ ] User can create new Fleet server host via the UI - The Fleet Server Hosts dropdown is disabled and set to the default host URL - [ ] User can create new Fleet server host via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Fleet server host config - [ ] User can select a custom Fleet Server host configuration from the Agent policy settings page - [ ] User can create new Fleet Elasticsearch output via the UI - The Hosts dropdown is disabled and set to the default Elasticsearch URL - [ ] User can create new Fleet Elasticsearch output via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Elasticsearch output - [ ] User can select a custom Elasticsearch output configuration from the Agent policy settings page - [ ] User can create a custom Logstash output with proxy - [ ] User can create a custom Kafka output with proxy ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Screenshots Fleet settings (proxies available): ![Screenshot 2024-02-02 at 17 21 05](https://github.com/elastic/kibana/assets/23701614/e94d3dfa-0467-48d0-9c99-c0288dfbba92) ![Screenshot 2024-02-02 at 17 21 15](https://github.com/elastic/kibana/assets/23701614/eb815d25-0e4c-497f-899a-036a89a0d126) Adding and editing a Fleet server host: ![Screenshot 2024-02-05 at 11 54 04](https://github.com/elastic/kibana/assets/23701614/2ad713fd-7992-4285-8605-1841ab7f56ef) ![Screenshot 2024-02-02 at 17 22 00](https://github.com/elastic/kibana/assets/23701614/dae30411-b023-4a3c-bba2-427dd3b8cd2a) Adding and editing an Elasticsearch output: ![Screenshot 2024-02-02 at 17 22 23](https://github.com/elastic/kibana/assets/23701614/cd6bdb90-d68a-4cf8-a1aa-34cf0eddc978) ![Screenshot 2024-02-02 at 17 22 38](https://github.com/elastic/kibana/assets/23701614/3a9c8f04-2586-4e3a-afd4-7a192245bc15) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

## Summary Closes #176352 Closes #176399 #175315 added the possibility to configure new Fleet Server hosts in serverless, with the constraint that the host URL must match the default URL. The API integration tests written to test this have been flaky, probably due to request timeout when fetching all Fleet Server hosts. This PR improves this by directly retrieving the default Fleet Server host by id. This fix has been tested using the Flaky Test Runner Pipeline, with 25 test runs for observability and security project types: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5140 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed

…c#175315) Closes elastic#165672 ### Summary This PR adds support for custom output and Fleet server hosts with proxies in serverless mode: - Proxies are re-enabled in serverless. - It is possible to add custom Fleet server hosts in serverless, with the constraint that the host URL must match the Elasticsearch URL of the default host. - New Elasticsearch outputs must also have the default host URL. ### Testing The below requirements should be tested in serverless mode for observability and security project types: ```sh # elasticsearch yarn es serverless --kill # kibana: one of yarn serverless-oblt yarn serverless-security ``` ⚠️ In addition, stateful mode should **not** be affected by any of these changes. #### Config In order to test this change, you will need the following configuration to mirror a serverless setup. Create a `serverless.dev.yml` if you don't have one already and set a project id (this is required for Fleet's `cloud.isServerlessEnabled` to correctly be `true`): ```yaml xpack.cloud.serverless.project_id: test-123 ``` In `kibana.devl.yml`, make sure the default Fleet server hosts and default output have the expected ids: ```yaml xpack.fleet.fleetServerHosts: - id: default-fleet-server name: Default Fleet server is_default: true host_urls: ['https://host.docker.internal:8220'] xpack.fleet.outputs: - id: es-default-output name: Default output type: elasticsearch is_default: true is_default_monitoring: true hosts: ['https://host.docker.internal:9200'] ``` #### Requirements - [ ] User can create proxy configurations in the Fleet UI and API. - [ ] User can create new Fleet server host via the UI - The Fleet Server Hosts dropdown is disabled and set to the default host URL - [ ] User can create new Fleet server host via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Fleet server host config - [ ] User can select a custom Fleet Server host configuration from the Agent policy settings page - [ ] User can create new Fleet Elasticsearch output via the UI - The Hosts dropdown is disabled and set to the default Elasticsearch URL - [ ] User can create new Fleet Elasticsearch output via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Elasticsearch output - [ ] User can select a custom Elasticsearch output configuration from the Agent policy settings page - [ ] User can create a custom Logstash output with proxy - [ ] User can create a custom Kafka output with proxy ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Screenshots Fleet settings (proxies available): ![Screenshot 2024-02-02 at 17 21 05](https://github.com/elastic/kibana/assets/23701614/e94d3dfa-0467-48d0-9c99-c0288dfbba92) ![Screenshot 2024-02-02 at 17 21 15](https://github.com/elastic/kibana/assets/23701614/eb815d25-0e4c-497f-899a-036a89a0d126) Adding and editing a Fleet server host: ![Screenshot 2024-02-05 at 11 54 04](https://github.com/elastic/kibana/assets/23701614/2ad713fd-7992-4285-8605-1841ab7f56ef) ![Screenshot 2024-02-02 at 17 22 00](https://github.com/elastic/kibana/assets/23701614/dae30411-b023-4a3c-bba2-427dd3b8cd2a) Adding and editing an Elasticsearch output: ![Screenshot 2024-02-02 at 17 22 23](https://github.com/elastic/kibana/assets/23701614/cd6bdb90-d68a-4cf8-a1aa-34cf0eddc978) ![Screenshot 2024-02-02 at 17 22 38](https://github.com/elastic/kibana/assets/23701614/3a9c8f04-2586-4e3a-afd4-7a192245bc15) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

## Summary Closes elastic#176352 Closes elastic#176399 elastic#175315 added the possibility to configure new Fleet Server hosts in serverless, with the constraint that the host URL must match the default URL. The API integration tests written to test this have been flaky, probably due to request timeout when fetching all Fleet Server hosts. This PR improves this by directly retrieving the default Fleet Server host by id. This fix has been tested using the Flaky Test Runner Pipeline, with 25 test runs for observability and security project types: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5140 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed

…c#175315) Closes elastic#165672 ### Summary This PR adds support for custom output and Fleet server hosts with proxies in serverless mode: - Proxies are re-enabled in serverless. - It is possible to add custom Fleet server hosts in serverless, with the constraint that the host URL must match the Elasticsearch URL of the default host. - New Elasticsearch outputs must also have the default host URL. ### Testing The below requirements should be tested in serverless mode for observability and security project types: ```sh # elasticsearch yarn es serverless --kill # kibana: one of yarn serverless-oblt yarn serverless-security ``` ⚠️ In addition, stateful mode should **not** be affected by any of these changes. #### Config In order to test this change, you will need the following configuration to mirror a serverless setup. Create a `serverless.dev.yml` if you don't have one already and set a project id (this is required for Fleet's `cloud.isServerlessEnabled` to correctly be `true`): ```yaml xpack.cloud.serverless.project_id: test-123 ``` In `kibana.devl.yml`, make sure the default Fleet server hosts and default output have the expected ids: ```yaml xpack.fleet.fleetServerHosts: - id: default-fleet-server name: Default Fleet server is_default: true host_urls: ['https://host.docker.internal:8220'] xpack.fleet.outputs: - id: es-default-output name: Default output type: elasticsearch is_default: true is_default_monitoring: true hosts: ['https://host.docker.internal:9200'] ``` #### Requirements - [ ] User can create proxy configurations in the Fleet UI and API. - [ ] User can create new Fleet server host via the UI - The Fleet Server Hosts dropdown is disabled and set to the default host URL - [ ] User can create new Fleet server host via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Fleet server host config - [ ] User can select a custom Fleet Server host configuration from the Agent policy settings page - [ ] User can create new Fleet Elasticsearch output via the UI - The Hosts dropdown is disabled and set to the default Elasticsearch URL - [ ] User can create new Fleet Elasticsearch output via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Elasticsearch output - [ ] User can select a custom Elasticsearch output configuration from the Agent policy settings page - [ ] User can create a custom Logstash output with proxy - [ ] User can create a custom Kafka output with proxy ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Screenshots Fleet settings (proxies available): ![Screenshot 2024-02-02 at 17 21 05](https://github.com/elastic/kibana/assets/23701614/e94d3dfa-0467-48d0-9c99-c0288dfbba92) ![Screenshot 2024-02-02 at 17 21 15](https://github.com/elastic/kibana/assets/23701614/eb815d25-0e4c-497f-899a-036a89a0d126) Adding and editing a Fleet server host: ![Screenshot 2024-02-05 at 11 54 04](https://github.com/elastic/kibana/assets/23701614/2ad713fd-7992-4285-8605-1841ab7f56ef) ![Screenshot 2024-02-02 at 17 22 00](https://github.com/elastic/kibana/assets/23701614/dae30411-b023-4a3c-bba2-427dd3b8cd2a) Adding and editing an Elasticsearch output: ![Screenshot 2024-02-02 at 17 22 23](https://github.com/elastic/kibana/assets/23701614/cd6bdb90-d68a-4cf8-a1aa-34cf0eddc978) ![Screenshot 2024-02-02 at 17 22 38](https://github.com/elastic/kibana/assets/23701614/3a9c8f04-2586-4e3a-afd4-7a192245bc15) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

## Summary Closes elastic#176352 Closes elastic#176399 elastic#175315 added the possibility to configure new Fleet Server hosts in serverless, with the constraint that the host URL must match the default URL. The API integration tests written to test this have been flaky, probably due to request timeout when fetching all Fleet Server hosts. This PR improves this by directly retrieving the default Fleet Server host by id. This fix has been tested using the Flaky Test Runner Pipeline, with 25 test runs for observability and security project types: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5140 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed

## Summary Closes #176352 Closes #176399 #175315 added the possibility to configure new Fleet Server hosts in serverless, with the constraint that the host URL must match the default URL. The API integration tests written to test this have been flaky, due to failure retrieving the default Fleet Server host or default Elasticsearch output from saved objects. This PR adds retry in the tests. Note: I have tried adding retry logic in the API handlers but kept hitting test flakiness. This fix has been tested using the Flaky Test Runner Pipeline, with 48/49 test runs for observability and security project types: 🟢 https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5218 🟢 https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5222 🟢 https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5225 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed

…c#175315) Closes elastic#165672 ### Summary This PR adds support for custom output and Fleet server hosts with proxies in serverless mode: - Proxies are re-enabled in serverless. - It is possible to add custom Fleet server hosts in serverless, with the constraint that the host URL must match the Elasticsearch URL of the default host. - New Elasticsearch outputs must also have the default host URL. ### Testing The below requirements should be tested in serverless mode for observability and security project types: ```sh # elasticsearch yarn es serverless --kill # kibana: one of yarn serverless-oblt yarn serverless-security ``` ⚠️ In addition, stateful mode should **not** be affected by any of these changes. #### Config In order to test this change, you will need the following configuration to mirror a serverless setup. Create a `serverless.dev.yml` if you don't have one already and set a project id (this is required for Fleet's `cloud.isServerlessEnabled` to correctly be `true`): ```yaml xpack.cloud.serverless.project_id: test-123 ``` In `kibana.devl.yml`, make sure the default Fleet server hosts and default output have the expected ids: ```yaml xpack.fleet.fleetServerHosts: - id: default-fleet-server name: Default Fleet server is_default: true host_urls: ['https://host.docker.internal:8220'] xpack.fleet.outputs: - id: es-default-output name: Default output type: elasticsearch is_default: true is_default_monitoring: true hosts: ['https://host.docker.internal:9200'] ``` #### Requirements - [ ] User can create proxy configurations in the Fleet UI and API. - [ ] User can create new Fleet server host via the UI - The Fleet Server Hosts dropdown is disabled and set to the default host URL - [ ] User can create new Fleet server host via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Fleet server host config - [ ] User can select a custom Fleet Server host configuration from the Agent policy settings page - [ ] User can create new Fleet Elasticsearch output via the UI - The Hosts dropdown is disabled and set to the default Elasticsearch URL - [ ] User can create new Fleet Elasticsearch output via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Elasticsearch output - [ ] User can select a custom Elasticsearch output configuration from the Agent policy settings page - [ ] User can create a custom Logstash output with proxy - [ ] User can create a custom Kafka output with proxy ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Screenshots Fleet settings (proxies available): ![Screenshot 2024-02-02 at 17 21 05](https://github.com/elastic/kibana/assets/23701614/e94d3dfa-0467-48d0-9c99-c0288dfbba92) ![Screenshot 2024-02-02 at 17 21 15](https://github.com/elastic/kibana/assets/23701614/eb815d25-0e4c-497f-899a-036a89a0d126) Adding and editing a Fleet server host: ![Screenshot 2024-02-05 at 11 54 04](https://github.com/elastic/kibana/assets/23701614/2ad713fd-7992-4285-8605-1841ab7f56ef) ![Screenshot 2024-02-02 at 17 22 00](https://github.com/elastic/kibana/assets/23701614/dae30411-b023-4a3c-bba2-427dd3b8cd2a) Adding and editing an Elasticsearch output: ![Screenshot 2024-02-02 at 17 22 23](https://github.com/elastic/kibana/assets/23701614/cd6bdb90-d68a-4cf8-a1aa-34cf0eddc978) ![Screenshot 2024-02-02 at 17 22 38](https://github.com/elastic/kibana/assets/23701614/3a9c8f04-2586-4e3a-afd4-7a192245bc15) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

## Summary Closes elastic#176352 Closes elastic#176399 elastic#175315 added the possibility to configure new Fleet Server hosts in serverless, with the constraint that the host URL must match the default URL. The API integration tests written to test this have been flaky, probably due to request timeout when fetching all Fleet Server hosts. This PR improves this by directly retrieving the default Fleet Server host by id. This fix has been tested using the Flaky Test Runner Pipeline, with 25 test runs for observability and security project types: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5140 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed

## Summary Closes elastic#176352 Closes elastic#176399 elastic#175315 added the possibility to configure new Fleet Server hosts in serverless, with the constraint that the host URL must match the default URL. The API integration tests written to test this have been flaky, due to failure retrieving the default Fleet Server host or default Elasticsearch output from saved objects. This PR adds retry in the tests. Note: I have tried adding retry logic in the API handlers but kept hitting test flakiness. This fix has been tested using the Flaky Test Runner Pipeline, with 48/49 test runs for observability and security project types: 🟢 https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5218 🟢 https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5222 🟢 https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5225 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed

## Summary Closes elastic#176352 Closes elastic#176399 elastic#175315 added the possibility to configure new Fleet Server hosts in serverless, with the constraint that the host URL must match the default URL. The API integration tests written to test this have been flaky, due to failure retrieving the default Fleet Server host or default Elasticsearch output from saved objects. This PR adds retry in the tests. Note: I have tried adding retry logic in the API handlers but kept hitting test flakiness. This fix has been tested using the Flaky Test Runner Pipeline, with 48/49 test runs for observability and security project types: 🟢 https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5218 🟢 https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5222 🟢 https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5225 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed (cherry picked from commit 83e64be) # Conflicts: # x-pack/test_serverless/api_integration/test_suites/security/fleet/fleet.ts

jillguyonnet self-assigned this Jan 23, 2024

juliaElastic reviewed Jan 26, 2024

View reviewed changes

x-pack/plugins/fleet/public/applications/fleet/sections/settings/index.tsx Outdated Show resolved Hide resolved

jillguyonnet added 2 commits January 29, 2024 11:28

Support custom ES outputs on serverless

16ccc38

Support custom Fleet server hosts on serverless

3db1fa4

jillguyonnet force-pushed the fleet/165672-serverless-proxies-support branch from a15b3ca to 3db1fa4 Compare January 29, 2024 10:31

Use constant for default output id

5d6068d

jillguyonnet commented Jan 29, 2024

View reviewed changes

...plications/fleet/components/fleet_server_instructions/components/fleet_server_hosts_form.tsx Outdated Show resolved Hide resolved

Linting

a498338

jillguyonnet added 3 commits January 30, 2024 17:31

Fix output UI and API

d5b12fd

Fix fleet server host UI and API

5c30c2c

Add unit tests for API handlers

c2e769d

jillguyonnet added Team:Fleet Team label for Observability Data Collection Fleet team release_note:enhancement labels Jan 31, 2024

jillguyonnet mentioned this pull request Jan 31, 2024

[Fleet] Add support for output and Fleet Server proxies on Serverless #165672

Closed

Fix and add API integration tests

c475c6a

jillguyonnet commented Jan 31, 2024

View reviewed changes

jillguyonnet added 2 commits January 31, 2024 18:22

And also observability tests

417b519

Add README section about Fleet serverless FTR

034a198

nchaulet self-requested a review January 31, 2024 20:59

Update create fleet server flyout description

b66acb1

Merge branch 'main' into fleet/165672-serverless-proxies-support

a3a5562

azasypkin approved these changes Feb 5, 2024

View reviewed changes

Merge branch 'main' into fleet/165672-serverless-proxies-support

2df7a75

nchaulet approved these changes Feb 5, 2024

View reviewed changes

juliaElastic approved these changes Feb 5, 2024

View reviewed changes

jillguyonnet merged commit 11fe56b into elastic:main Feb 5, 2024
23 checks passed

jillguyonnet deleted the fleet/165672-serverless-proxies-support branch February 5, 2024 15:00

kibanamachine added v8.13.0 backport:skip This commit does not require backporting labels Feb 5, 2024

jillguyonnet mentioned this pull request Feb 5, 2024

[Fleet] Document serverless restrictions changes elastic/ingest-docs#887

Closed

This was referenced Feb 12, 2024

[Fleet] Fix flaky serverless API integration tests #176614

Merged

[Fleet] Fixes for proxies support in serverless #176728

Closed

jillguyonnet mentioned this pull request Feb 19, 2024

[Fleet] Add retry logic to serverless API check #176808

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Support output and Fleet server proxies on serverless #175315

[Fleet] Support output and Fleet server proxies on serverless #175315

jillguyonnet commented Jan 23, 2024 •

edited

Loading

apmmachine commented Jan 23, 2024

juliaElastic commented Jan 29, 2024 •

edited

Loading

jillguyonnet commented Jan 29, 2024

juliaElastic commented Jan 29, 2024 •

edited

Loading

jillguyonnet commented Jan 31, 2024

juliaElastic commented Jan 31, 2024 •

edited

Loading

jillguyonnet commented Jan 31, 2024 •

edited by juliaElastic

Loading

juliaElastic commented Jan 31, 2024

jillguyonnet commented Jan 31, 2024

jillguyonnet Jan 31, 2024

juliaElastic Feb 2, 2024

jillguyonnet Feb 2, 2024

juliaElastic Feb 2, 2024

jillguyonnet commented Jan 31, 2024

nchaulet commented Jan 31, 2024

nimarezainia commented Feb 4, 2024

jillguyonnet commented Feb 5, 2024

jillguyonnet commented Feb 5, 2024

azasypkin left a comment

jillguyonnet commented Feb 5, 2024

nchaulet left a comment

juliaElastic left a comment

kibana-ci commented Feb 5, 2024

[Fleet] Support output and Fleet server proxies on serverless #175315

[Fleet] Support output and Fleet server proxies on serverless #175315

Conversation

jillguyonnet commented Jan 23, 2024 • edited Loading

Summary

Testing

Config

Requirements

Checklist

Screenshots

apmmachine commented Jan 23, 2024

🤖 GitHub comments

juliaElastic commented Jan 29, 2024 • edited Loading

jillguyonnet commented Jan 29, 2024

juliaElastic commented Jan 29, 2024 • edited Loading

jillguyonnet commented Jan 31, 2024

juliaElastic commented Jan 31, 2024 • edited Loading

jillguyonnet commented Jan 31, 2024 • edited by juliaElastic Loading

juliaElastic commented Jan 31, 2024

jillguyonnet commented Jan 31, 2024

jillguyonnet Jan 31, 2024

Choose a reason for hiding this comment

juliaElastic Feb 2, 2024

Choose a reason for hiding this comment

jillguyonnet Feb 2, 2024

Choose a reason for hiding this comment

juliaElastic Feb 2, 2024

Choose a reason for hiding this comment

jillguyonnet commented Jan 31, 2024

nchaulet commented Jan 31, 2024

nimarezainia commented Feb 4, 2024

jillguyonnet commented Feb 5, 2024

jillguyonnet commented Feb 5, 2024

azasypkin left a comment

Choose a reason for hiding this comment

jillguyonnet commented Feb 5, 2024

nchaulet left a comment

Choose a reason for hiding this comment

juliaElastic left a comment

Choose a reason for hiding this comment

kibana-ci commented Feb 5, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

Page load bundle

History

jillguyonnet commented Jan 23, 2024 •

edited

Loading

juliaElastic commented Jan 29, 2024 •

edited

Loading

juliaElastic commented Jan 29, 2024 •

edited

Loading

juliaElastic commented Jan 31, 2024 •

edited

Loading

jillguyonnet commented Jan 31, 2024 •

edited by juliaElastic

Loading