[Fleet] Add support for output and Fleet Server proxies on Serverless #165672

joshdover · 2023-09-05T10:46:40Z

In https://github.com/elastic/ingest-dev/issues/1530 we disabled support for configuring proxies on output and Fleet Server connections for Serverless projects.

Now that we know more about the requirements, we need to bring this support back. We need to allow users to configure a proxy for Elastic Agent to communicate with both Elasticsearch and Fleet Server hosted in Serverless. We are not bring back support for on-premise Fleet Server hosts or custom URLs for Fleet Server or Elasticsearch.

Requirements

Users need to be able to create proxy configurations in the Fleet UI and API, just like they can with Stateful deployments
Users need to be able to create new Fleet Server host configs with proxy configurations
- Users should NOT be able to change the Fleet Server host URL. All new Fleet Server hosts that are created should have the URL provided by the Serverless project and cannot be edited via the UI or API.
- Users should be able to add a proxy configuration to a Fleet Server host configuration
- Users should be able to set a default Fleet Server host configuration that includes a proxy config
- Users should be able to select a custom Fleet Server host configuration from the Agent policy settings page
Users need to be able to create custom Elasticsearch output configs with proxy configurations
- Users should NOT be able to change the Elasticsearch URL. All new Elasticsearch outputs that are created should have the URL provided by the Serverless project and cannot be edited via the UI or API.
- Users should be able to add a proxy configuration to an Elasticsearch output configuration
- Users should be able to select a custom Elasticsearch output configuration from the Agent policy settings page
Users need to be able to create custom Logstash and Kafka output configs with proxy configurations
- This should work exactly the same as Stateful deployments, all settings are customizable.

How to handle preconfigured outputs and Fleet Server hosts

We current preconfigure an ES output and Fleet Server host in Kibana configuration. These are not editable today. To avoid any issues in the future, I think we should keep this behavior and not allow any edits to the preconfigured output or Fleet server host config.

So if a user wants to use a proxy they have to create a new output or FS host config. We should allow them to set this new output or FS host config as the default. The default setting is the only aspect that the user should be able to override about the preconfigured output and FS configs.

elasticmachine · 2023-09-05T10:46:43Z

Pinging @elastic/fleet (Team:Fleet)

joshdover · 2023-09-05T10:47:08Z

cc @nimarezainia please review

nimarezainia · 2023-09-11T01:14:54Z

Users should be able to set a default Fleet Server host configuration that includes a proxy config

I'm not following what this is exactly saying that? the user shouldn't be allowed to change the host settings, they should only be allowed to add a proxy definition.Which are already mentioned.

Users should be able to select a custom Fleet Server host configuration from the Agent policy settings page

I had this as feedback elsewhere for serverless. Why do we allow this? In serverless there will be a default Fleet Server/Service that is configured. I can't think of a use case where we would have multiple. If that is correct then the drop down in the policy to select a Fleet Server is also not required. In serverless all of this is fixed.

I also added Kafka to the logstash description above. it would need the same treatment.

thanks again.

nimarezainia · 2023-09-11T01:24:36Z

reading the description again. Just to clarify: is the suggestion that if the user wants to add a proxy to ES or Fleet Server, they should add a new output or FS and then be allowed t modify the proxy settings. the host setings are still locked. Is this correct?

joshdover · 2023-09-11T16:52:28Z

reading the description again. Just to clarify: is the suggestion that if the user wants to add a proxy to ES or Fleet Server, they should add a new output or FS and then be allowed t modify the proxy settings. the host setings are still locked. Is this correct?

Yes that's the suggestion. Today we couple these things together. A FS host config is a list of FS URLs + an optional proxy configuration. So it's a bit roundabout to choose a proxy, you have to create a new FS host config that has a proxy selected.

We could consider implementing this differently and more specifically for Serverless where instead we just allow choosing a proxy from the Agent policy for FS connections. I think the output UX makes sense still though.

nchaulet · 2024-01-02T10:46:52Z

We current preconfigure an ES output and Fleet Server host in Kibana configuration. These are not editable today. To avoid any issues in the future, I think we should keep this behavior and not allow any edits to the preconfigured output or Fleet server host config.

This is not true today in serverless, we allow to edit some fields for default preconfigured output with the allow_edit property of preconfiguration that allow to configure preset and config_yaml, we should probably allow to edit the proxy_url here too.

We could consider implementing this differently and more specifically for Serverless where instead we just allow choosing a proxy from the Agent policy for FS connections. I think the output UX makes sense still though.

Does it make sense to add some extra/code complexity for serverless for that feature? it seems to me that feature is already for advanced user, if we can reuse Fleet server hosts for that it will probably make things easier to implement maintain.

nchaulet · 2024-01-02T11:44:20Z

Proposed Implementation plan

We can have a special behavior on serverless based on the value of cloudSetup.isServerless

For serverless we should do the following changes:

Outputs

UI

When adding/editing a new output of type elasticsearch:

Prepopulate the hosts field with the same values as the default output (the default output could be retrieved with the following id es-default-output)
disable the host field

API

For non preconfigured output validate that hosts is equal to the default output hosts or throw a 400

Setup preconfiguration

If the default output hosts change we should update all non preconfigured outputs with the new hosts. (we should verify if it's something that could happen)
We should add proxy_id to the allow_edit list in the project controller https://github.com/elastic/project-controller/blob/6e35c0a1bb26f094765e9ad86649a10e544a739a/internal/application/kibana/fleet_config.go#L87

Fleet server hosts

API

we should remove the restrictions to create/edit fleet server hosts in serverless.
we should validate that the host is the same as default-fleet-server-host fleet server host.

UI

When adding/editing a new fleet server host:

Remove the restriction to create fleet server hosts in the UI
Prepopulate the hosts field with the same values as the default fleet server host (could be retrieved with the following id default-fleet-server-host)
disable the host field

Setup preconfiguration

If the default fleet server hosts hosts change we should update all non preconfigured fleet server hosts with the new hosts. (we should verify if it's something that could happen)
* We should probably create a new fleet server host for the agentless policy in the project controller https://github.com/elastic/project-controller/blob/6e35c0a1bb26f094765e9ad86649a10e544a739a/internal/application/kibana/fleet_config.go#L87 and use it in the agentless policy to, to avoid that policy using a fleet server host with proxy. already the case

nimarezainia · 2024-01-08T04:09:53Z

We could consider implementing this differently and more specifically for Serverless where instead we just allow choosing a proxy from the Agent policy for FS connections. I think the output UX makes sense still though.

Does it make sense to add some extra/code complexity for serverless for that feature? it seems to me that feature is already for advanced user, if we can reuse Fleet server hosts for that it will probably make things easier to implement maintain.

I would also like to avoid this complexity which sees the behavior divergence between serverless and non-serverless.

nimarezainia · 2024-01-08T04:24:30Z

UI

When adding/editing a new fleet server host:

* Remove the restriction to create fleet server hosts in the UI

* Prepopulate the hosts field with the same values as the default fleet server host (could be retrieved with the following id `default-fleet-server-host`)

* disable the host field

Do we have to make these changes? Today the Fleet Server shows as locked, so no changes can be made to it. However the user seems to be able to edit:

Note that the host is locked down - so can't be modified however I am able to add a proxy. I can't go further than this as I don;t have the ability to add the proxy definition itself (subject of this issue).

So if I am not mistaken we just need to make sure the user can add the proxy and save.

nimarezainia · 2024-01-08T04:25:52Z

For use cases that have multiple different networks and different unique proxies all forwarding to the same project, I suppose the above change as described is required.

jillguyonnet · 2024-01-31T12:01:44Z

Hi, the following implementation is ready for testing in #175315:

proxies enabled in serverless
Fleet server and output hosts constrained to default URL in serverless

Just looking at the preconfiguration now: is it known whether the default output/fleet server hosts can change (per @nchaulet's comment)?

jlind23 · 2024-02-05T08:40:42Z

Just looking at the preconfiguration now: is it known whether the default output/fleet server hosts can change (per @nchaulet's #165672 (comment))?

I don't see any reason why the default output/fleet server hosts would change in Serverless, at least this never was something we faced up until today.

jillguyonnet · 2024-02-05T11:15:00Z

Thank you @jlind23 👍 As agreed on the PR, preconfiguration change will not be handled.

Closes #165672 ### Summary This PR adds support for custom output and Fleet server hosts with proxies in serverless mode: - Proxies are re-enabled in serverless. - It is possible to add custom Fleet server hosts in serverless, with the constraint that the host URL must match the Elasticsearch URL of the default host. - New Elasticsearch outputs must also have the default host URL. ### Testing The below requirements should be tested in serverless mode for observability and security project types: ```sh # elasticsearch yarn es serverless --kill # kibana: one of yarn serverless-oblt yarn serverless-security ``` ⚠️ In addition, stateful mode should **not** be affected by any of these changes. #### Config In order to test this change, you will need the following configuration to mirror a serverless setup. Create a `serverless.dev.yml` if you don't have one already and set a project id (this is required for Fleet's `cloud.isServerlessEnabled` to correctly be `true`): ```yaml xpack.cloud.serverless.project_id: test-123 ``` In `kibana.devl.yml`, make sure the default Fleet server hosts and default output have the expected ids: ```yaml xpack.fleet.fleetServerHosts: - id: default-fleet-server name: Default Fleet server is_default: true host_urls: ['https://host.docker.internal:8220'] xpack.fleet.outputs: - id: es-default-output name: Default output type: elasticsearch is_default: true is_default_monitoring: true hosts: ['https://host.docker.internal:9200'] ``` #### Requirements - [ ] User can create proxy configurations in the Fleet UI and API. - [ ] User can create new Fleet server host via the UI - The Fleet Server Hosts dropdown is disabled and set to the default host URL - [ ] User can create new Fleet server host via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Fleet server host config - [ ] User can select a custom Fleet Server host configuration from the Agent policy settings page - [ ] User can create new Fleet Elasticsearch output via the UI - The Hosts dropdown is disabled and set to the default Elasticsearch URL - [ ] User can create new Fleet Elasticsearch output via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Elasticsearch output - [ ] User can select a custom Elasticsearch output configuration from the Agent policy settings page - [ ] User can create a custom Logstash output with proxy - [ ] User can create a custom Kafka output with proxy ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Screenshots Fleet settings (proxies available): ![Screenshot 2024-02-02 at 17 21 05](https://github.com/elastic/kibana/assets/23701614/e94d3dfa-0467-48d0-9c99-c0288dfbba92) ![Screenshot 2024-02-02 at 17 21 15](https://github.com/elastic/kibana/assets/23701614/eb815d25-0e4c-497f-899a-036a89a0d126) Adding and editing a Fleet server host: ![Screenshot 2024-02-05 at 11 54 04](https://github.com/elastic/kibana/assets/23701614/2ad713fd-7992-4285-8605-1841ab7f56ef) ![Screenshot 2024-02-02 at 17 22 00](https://github.com/elastic/kibana/assets/23701614/dae30411-b023-4a3c-bba2-427dd3b8cd2a) Adding and editing an Elasticsearch output: ![Screenshot 2024-02-02 at 17 22 23](https://github.com/elastic/kibana/assets/23701614/cd6bdb90-d68a-4cf8-a1aa-34cf0eddc978) ![Screenshot 2024-02-02 at 17 22 38](https://github.com/elastic/kibana/assets/23701614/3a9c8f04-2586-4e3a-afd4-7a192245bc15) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

jillguyonnet · 2024-02-05T15:18:55Z

@elastic/fleet-qasource-external As this is a serverless change, it should be part of the next weekly release.

amolnater-qasource · 2024-02-07T09:39:08Z

Thank you for the update @jillguyonnet
These changes are not available on the latest available serverless environment.

Further, we will test this once next serverless build is available.

…c#175315) Closes elastic#165672 ### Summary This PR adds support for custom output and Fleet server hosts with proxies in serverless mode: - Proxies are re-enabled in serverless. - It is possible to add custom Fleet server hosts in serverless, with the constraint that the host URL must match the Elasticsearch URL of the default host. - New Elasticsearch outputs must also have the default host URL. ### Testing The below requirements should be tested in serverless mode for observability and security project types: ```sh # elasticsearch yarn es serverless --kill # kibana: one of yarn serverless-oblt yarn serverless-security ``` ⚠️ In addition, stateful mode should **not** be affected by any of these changes. #### Config In order to test this change, you will need the following configuration to mirror a serverless setup. Create a `serverless.dev.yml` if you don't have one already and set a project id (this is required for Fleet's `cloud.isServerlessEnabled` to correctly be `true`): ```yaml xpack.cloud.serverless.project_id: test-123 ``` In `kibana.devl.yml`, make sure the default Fleet server hosts and default output have the expected ids: ```yaml xpack.fleet.fleetServerHosts: - id: default-fleet-server name: Default Fleet server is_default: true host_urls: ['https://host.docker.internal:8220'] xpack.fleet.outputs: - id: es-default-output name: Default output type: elasticsearch is_default: true is_default_monitoring: true hosts: ['https://host.docker.internal:9200'] ``` #### Requirements - [ ] User can create proxy configurations in the Fleet UI and API. - [ ] User can create new Fleet server host via the UI - The Fleet Server Hosts dropdown is disabled and set to the default host URL - [ ] User can create new Fleet server host via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Fleet server host config - [ ] User can select a custom Fleet Server host configuration from the Agent policy settings page - [ ] User can create new Fleet Elasticsearch output via the UI - The Hosts dropdown is disabled and set to the default Elasticsearch URL - [ ] User can create new Fleet Elasticsearch output via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Elasticsearch output - [ ] User can select a custom Elasticsearch output configuration from the Agent policy settings page - [ ] User can create a custom Logstash output with proxy - [ ] User can create a custom Kafka output with proxy ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Screenshots Fleet settings (proxies available): ![Screenshot 2024-02-02 at 17 21 05](https://github.com/elastic/kibana/assets/23701614/e94d3dfa-0467-48d0-9c99-c0288dfbba92) ![Screenshot 2024-02-02 at 17 21 15](https://github.com/elastic/kibana/assets/23701614/eb815d25-0e4c-497f-899a-036a89a0d126) Adding and editing a Fleet server host: ![Screenshot 2024-02-05 at 11 54 04](https://github.com/elastic/kibana/assets/23701614/2ad713fd-7992-4285-8605-1841ab7f56ef) ![Screenshot 2024-02-02 at 17 22 00](https://github.com/elastic/kibana/assets/23701614/dae30411-b023-4a3c-bba2-427dd3b8cd2a) Adding and editing an Elasticsearch output: ![Screenshot 2024-02-02 at 17 22 23](https://github.com/elastic/kibana/assets/23701614/cd6bdb90-d68a-4cf8-a1aa-34cf0eddc978) ![Screenshot 2024-02-02 at 17 22 38](https://github.com/elastic/kibana/assets/23701614/3a9c8f04-2586-4e3a-afd4-7a192245bc15) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

amolnater-qasource · 2024-02-13T12:03:13Z

Hi @jillguyonnet

We have revalidated these changes on latest 8.13.0 Serverless environment and had below observations:

Proxy Used:
Squid

Observations:

Agent successfully gets installed using proxy.
We are able to make the changes under the Proxy settings.
Agent continues to send data when added to:
- Custom Fleet Server with Proxy.
- Custom Elasticsearch output with Proxy.
- Kafka output with Proxy.
Able to delete the added Proxy and agent continues to send data.

Further as per discussion under elastic/elastic-agent#2992, we are not able to test our squid proxy with Logstash output.
So we are currently working upon setting up mitmproxy for testing logstash output.

Build details:
BUILD: 71526
COMMIT: 955dd69

Please let us know if we are missing any scenario to be covered.

Thanks!

jillguyonnet · 2024-02-13T12:11:29Z

Hi @amolnater-qasource

Thank you, that is awesome.

Did you test the API as well or only the UI?

For visibility, the two bugs I found on my end are addressed in #176728 and should be fixed in the next release.

amolnater-qasource · 2024-02-13T12:13:19Z

@jillguyonnet We tested while adding from the UI only, could you please confirm if we need to test adding through API?

jillguyonnet · 2024-02-13T12:30:16Z

@amolnater-qasource Yes, that'd be great, thank you.

amolnater-qasource · 2024-02-15T08:08:28Z

Hi @jillguyonnet

We have revalidated the Fleet Settings on serverless environment through API and had below observations:

Observations:

We are able to create, update and delete Fleet Server hosts.
Able to create, update and delete Elasticsearch output.
However, updating Fleet Server hosts and Elasticsearch output also requires to add URL, or else it fails.
We are able to create, update and delete Proxy, and updating proxy didn't require URL.

Please let us know if we are missing any scenario to be covered.

Thanks!

jillguyonnet · 2024-02-15T09:11:36Z

Hi @amolnater-qasource Thank you very much for testing!

However, updating Fleet Server hosts and Elasticsearch output also requires to add URL, or else it fails.

The output API was fixed in #176728. I just re-tested to Fleet Server host API and I am able to update without passing the host URLs:

PUT kbn:api/fleet/fleet_server_hosts/test-fleet-server
{
  "name": "Renamed API created Fleet Server"
}

Could you confirm if you observed anything different?

Thanks

amolnater-qasource · 2024-02-15T10:50:18Z

Hi @jillguyonnet

Thank you for the update, we were not aware about a new serverless build availability and covered the testing on the same build shared under #165672 (comment).

We have revalidated this issue on the latest serverless environment and found it fixed now.

We are able to update Fleet Server hosts and Elasticsearch output without URL.

Screenshots:

Build details:
VERSION: 8.13.0 Serverless
BUILD: 71526
COMMIT: 955dd69

Thanks!

…c#175315) Closes elastic#165672 ### Summary This PR adds support for custom output and Fleet server hosts with proxies in serverless mode: - Proxies are re-enabled in serverless. - It is possible to add custom Fleet server hosts in serverless, with the constraint that the host URL must match the Elasticsearch URL of the default host. - New Elasticsearch outputs must also have the default host URL. ### Testing The below requirements should be tested in serverless mode for observability and security project types: ```sh # elasticsearch yarn es serverless --kill # kibana: one of yarn serverless-oblt yarn serverless-security ``` ⚠️ In addition, stateful mode should **not** be affected by any of these changes. #### Config In order to test this change, you will need the following configuration to mirror a serverless setup. Create a `serverless.dev.yml` if you don't have one already and set a project id (this is required for Fleet's `cloud.isServerlessEnabled` to correctly be `true`): ```yaml xpack.cloud.serverless.project_id: test-123 ``` In `kibana.devl.yml`, make sure the default Fleet server hosts and default output have the expected ids: ```yaml xpack.fleet.fleetServerHosts: - id: default-fleet-server name: Default Fleet server is_default: true host_urls: ['https://host.docker.internal:8220'] xpack.fleet.outputs: - id: es-default-output name: Default output type: elasticsearch is_default: true is_default_monitoring: true hosts: ['https://host.docker.internal:9200'] ``` #### Requirements - [ ] User can create proxy configurations in the Fleet UI and API. - [ ] User can create new Fleet server host via the UI - The Fleet Server Hosts dropdown is disabled and set to the default host URL - [ ] User can create new Fleet server host via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Fleet server host config - [ ] User can select a custom Fleet Server host configuration from the Agent policy settings page - [ ] User can create new Fleet Elasticsearch output via the UI - The Hosts dropdown is disabled and set to the default Elasticsearch URL - [ ] User can create new Fleet Elasticsearch output via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Elasticsearch output - [ ] User can select a custom Elasticsearch output configuration from the Agent policy settings page - [ ] User can create a custom Logstash output with proxy - [ ] User can create a custom Kafka output with proxy ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Screenshots Fleet settings (proxies available): ![Screenshot 2024-02-02 at 17 21 05](https://github.com/elastic/kibana/assets/23701614/e94d3dfa-0467-48d0-9c99-c0288dfbba92) ![Screenshot 2024-02-02 at 17 21 15](https://github.com/elastic/kibana/assets/23701614/eb815d25-0e4c-497f-899a-036a89a0d126) Adding and editing a Fleet server host: ![Screenshot 2024-02-05 at 11 54 04](https://github.com/elastic/kibana/assets/23701614/2ad713fd-7992-4285-8605-1841ab7f56ef) ![Screenshot 2024-02-02 at 17 22 00](https://github.com/elastic/kibana/assets/23701614/dae30411-b023-4a3c-bba2-427dd3b8cd2a) Adding and editing an Elasticsearch output: ![Screenshot 2024-02-02 at 17 22 23](https://github.com/elastic/kibana/assets/23701614/cd6bdb90-d68a-4cf8-a1aa-34cf0eddc978) ![Screenshot 2024-02-02 at 17 22 38](https://github.com/elastic/kibana/assets/23701614/3a9c8f04-2586-4e3a-afd4-7a192245bc15) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

amolnater-qasource · 2024-02-19T12:03:36Z

Hi Team,

We have setup mitmproxy with logstash output on self-managed 8.13.0-SNAPSHOT:

We observed the same error for mitmproxy as observed in the logs of issue attached under New data not generating for logstash output with proxy server. elastic-agent#2992 (comment). The logs were from the Squid proxy.

Logs from the Squid proxy:

Screenshot:

Mitmproxy with logstash Logs:
elastic-agent-diagnostics-2024-02-19T11-20-24Z-00.zip

Build details:
VERSION: 8.13.0
BUILD: 71681
COMMIT: 1b797a5

Please let us know if we are missing anything here or anything else is required from our end.

Thanks

jillguyonnet · 2024-02-19T13:43:53Z

Hi @amolnater-qasource, thank you for trying out mitmproxy 👍

So it seems we cannot test Logstash outputs for the scope of this change, correct? If so, then I suppose there is nothing else to test, at least while this issue persist.

…c#175315) Closes elastic#165672 ### Summary This PR adds support for custom output and Fleet server hosts with proxies in serverless mode: - Proxies are re-enabled in serverless. - It is possible to add custom Fleet server hosts in serverless, with the constraint that the host URL must match the Elasticsearch URL of the default host. - New Elasticsearch outputs must also have the default host URL. ### Testing The below requirements should be tested in serverless mode for observability and security project types: ```sh # elasticsearch yarn es serverless --kill # kibana: one of yarn serverless-oblt yarn serverless-security ``` ⚠️ In addition, stateful mode should **not** be affected by any of these changes. #### Config In order to test this change, you will need the following configuration to mirror a serverless setup. Create a `serverless.dev.yml` if you don't have one already and set a project id (this is required for Fleet's `cloud.isServerlessEnabled` to correctly be `true`): ```yaml xpack.cloud.serverless.project_id: test-123 ``` In `kibana.devl.yml`, make sure the default Fleet server hosts and default output have the expected ids: ```yaml xpack.fleet.fleetServerHosts: - id: default-fleet-server name: Default Fleet server is_default: true host_urls: ['https://host.docker.internal:8220'] xpack.fleet.outputs: - id: es-default-output name: Default output type: elasticsearch is_default: true is_default_monitoring: true hosts: ['https://host.docker.internal:9200'] ``` #### Requirements - [ ] User can create proxy configurations in the Fleet UI and API. - [ ] User can create new Fleet server host via the UI - The Fleet Server Hosts dropdown is disabled and set to the default host URL - [ ] User can create new Fleet server host via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Fleet server host config - [ ] User can select a custom Fleet Server host configuration from the Agent policy settings page - [ ] User can create new Fleet Elasticsearch output via the UI - The Hosts dropdown is disabled and set to the default Elasticsearch URL - [ ] User can create new Fleet Elasticsearch output via the API - The request should succeed if the host URL is set the the default one - Otherwise the request should fail - [ ] User can add a proxy config to the Elasticsearch output - [ ] User can select a custom Elasticsearch output configuration from the Agent policy settings page - [ ] User can create a custom Logstash output with proxy - [ ] User can create a custom Kafka output with proxy ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Screenshots Fleet settings (proxies available): ![Screenshot 2024-02-02 at 17 21 05](https://github.com/elastic/kibana/assets/23701614/e94d3dfa-0467-48d0-9c99-c0288dfbba92) ![Screenshot 2024-02-02 at 17 21 15](https://github.com/elastic/kibana/assets/23701614/eb815d25-0e4c-497f-899a-036a89a0d126) Adding and editing a Fleet server host: ![Screenshot 2024-02-05 at 11 54 04](https://github.com/elastic/kibana/assets/23701614/2ad713fd-7992-4285-8605-1841ab7f56ef) ![Screenshot 2024-02-02 at 17 22 00](https://github.com/elastic/kibana/assets/23701614/dae30411-b023-4a3c-bba2-427dd3b8cd2a) Adding and editing an Elasticsearch output: ![Screenshot 2024-02-02 at 17 22 23](https://github.com/elastic/kibana/assets/23701614/cd6bdb90-d68a-4cf8-a1aa-34cf0eddc978) ![Screenshot 2024-02-02 at 17 22 38](https://github.com/elastic/kibana/assets/23701614/3a9c8f04-2586-4e3a-afd4-7a192245bc15) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

joshdover added Team:Fleet Team label for Observability Data Collection Fleet team >enhancement labels Sep 5, 2023

jen-huang assigned criamico Dec 12, 2023

jlind23 assigned nchaulet and unassigned criamico Dec 20, 2023

nchaulet removed their assignment Jan 2, 2024

jen-huang assigned jillguyonnet Jan 2, 2024

kpollich added QA:Needs Validation Issue needs to be validated by QA 8.13 candidate and removed 8.13 candidate labels Jan 17, 2024

jillguyonnet mentioned this issue Jan 23, 2024

[Fleet] Support output and Fleet server proxies on serverless #175315

Merged

13 tasks

jillguyonnet closed this as completed in #175315 Feb 5, 2024

jillguyonnet mentioned this issue Feb 5, 2024

[Fleet] Document serverless restrictions changes elastic/ingest-docs#887

Closed

sophiec20 added enhancement New value added to drive a business result and removed >enhancement labels Mar 27, 2024

amolnater-qasource removed the QA:Needs Validation Issue needs to be validated by QA label Mar 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Add support for output and Fleet Server proxies on Serverless #165672

[Fleet] Add support for output and Fleet Server proxies on Serverless #165672

joshdover commented Sep 5, 2023 •

edited by nimarezainia

Loading

elasticmachine commented Sep 5, 2023

joshdover commented Sep 5, 2023

nimarezainia commented Sep 11, 2023

nimarezainia commented Sep 11, 2023

joshdover commented Sep 11, 2023

nchaulet commented Jan 2, 2024

nchaulet commented Jan 2, 2024 •

edited

Loading

nimarezainia commented Jan 8, 2024

nimarezainia commented Jan 8, 2024

UI

nimarezainia commented Jan 8, 2024

jillguyonnet commented Jan 31, 2024

jlind23 commented Feb 5, 2024

jillguyonnet commented Feb 5, 2024

jillguyonnet commented Feb 5, 2024

amolnater-qasource commented Feb 7, 2024

amolnater-qasource commented Feb 13, 2024

jillguyonnet commented Feb 13, 2024

amolnater-qasource commented Feb 13, 2024

jillguyonnet commented Feb 13, 2024

amolnater-qasource commented Feb 15, 2024 •

edited

Loading

jillguyonnet commented Feb 15, 2024

amolnater-qasource commented Feb 15, 2024

amolnater-qasource commented Feb 19, 2024

jillguyonnet commented Feb 19, 2024

[Fleet] Add support for output and Fleet Server proxies on Serverless #165672

[Fleet] Add support for output and Fleet Server proxies on Serverless #165672

Comments

joshdover commented Sep 5, 2023 • edited by nimarezainia Loading

Requirements

How to handle preconfigured outputs and Fleet Server hosts

elasticmachine commented Sep 5, 2023

joshdover commented Sep 5, 2023

nimarezainia commented Sep 11, 2023

nimarezainia commented Sep 11, 2023

joshdover commented Sep 11, 2023

nchaulet commented Jan 2, 2024

nchaulet commented Jan 2, 2024 • edited Loading

Outputs

UI

API

Setup preconfiguration

Fleet server hosts

API

UI

Setup preconfiguration

nimarezainia commented Jan 8, 2024

nimarezainia commented Jan 8, 2024

UI

nimarezainia commented Jan 8, 2024

jillguyonnet commented Jan 31, 2024

jlind23 commented Feb 5, 2024

jillguyonnet commented Feb 5, 2024

jillguyonnet commented Feb 5, 2024

amolnater-qasource commented Feb 7, 2024

amolnater-qasource commented Feb 13, 2024

jillguyonnet commented Feb 13, 2024

amolnater-qasource commented Feb 13, 2024

jillguyonnet commented Feb 13, 2024

amolnater-qasource commented Feb 15, 2024 • edited Loading

jillguyonnet commented Feb 15, 2024

amolnater-qasource commented Feb 15, 2024

amolnater-qasource commented Feb 19, 2024

jillguyonnet commented Feb 19, 2024

joshdover commented Sep 5, 2023 •

edited by nimarezainia

Loading

nchaulet commented Jan 2, 2024 •

edited

Loading

amolnater-qasource commented Feb 15, 2024 •

edited

Loading