Add documentation to cover the container flow.

[merlixelastic]

Description

We discovered a scenario where deploying Elastic agent with Fleet server integration need to have certificate to connect to Kibana.
This is setup apparently with variable KIBANA_CA or KIBANA_FLEET_CA.
I believe the code (reference in resources below) is related to the installation of Elastic agent with Fleet server on Kubernetes or docker.

Resources

The code related to this is here

I'm not seeing mention of setting up certificate between Elastic agent to Kibana in our documentation here and there.

This was discovered when we investigated error:
Kibana fetch policy failed: http GET request to https://kibana-multi:5601/api/fleet/agent_policies fails: fail to execute the HTTP GET request: Get "https://kibana-multi:5601/api/fleet/agent_policies": x509: certificate signed by unknown authority.

Collaboration

TBD. The docs and product team will work together to determine the best path forward.

Point of contact.

Main contact: @merlixelastic

Stakeholders: @lucabelluccini

[lucabelluccini]

This item has been discussed on 3rd June with Julien & Pierre

[lucabelluccini]

Notable points:

• We do not cover at all the use case of deploying Fleet Server on plain Kubernetes in our docs at https://www.elastic.co/guide/en/fleet/current/fleet-deployment-models.html
  • Not exactly the same, but we do not link deploying Fleet Servers on ECK, which exists at https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-elastic-agent-fleet.html
• The page https://www.elastic.co/guide/en/fleet/current/add-fleet-server-mixed.html#default-port-assignments-mixed at least to my analysis shows wrong "data flows". Elastic Agents for example reach out Fleet Server (enroll, checkin, management), the outputs (e.g. Elasticsearch or Logstash), but never Kibana. Also what Elastic Agent → Fleet mean?
• It seems in some flows, Fleet Servers on containerized environments need to reach Kibana, while it isn't necessary on normal installations.

[kilfoyle]

Thanks for reporting this @merlixelastic and for the clarifications @lucabelluccini.

So I think we need to:

• 1. Update the Deployment models page to cover the steps for setting up Fleet Server on Kubernetes.

• 2. Mention on that same page that managed Elastic Agent can be set up in ECK, with a link to the relevant docs.
  -> Opened docs PR: Add ECK deployment type with link to docs #1176

• 3. Update the default port assignments. It seems the only change needed is to remove the "Elastic Agent -> Fleet" entry.
  -> Opened docs PR: Remove "Elastic Agent to Fleet" default port assignments #1177

@michel-laterman could you perhaps provide the steps for number 1? I have no idea about the K8s setup at all, so if you don't mind putting something in a draft document I'd be happy to port that into the Deployment models page.

@nimarezainia just for your awareness.

[kilfoyle]

Regarding requirement 1, @nimarezainia and I discussed this and are thinking that we can update the Elastic Agent install instruction pages for Kubernetes and for Docker, to add the additional settings required to run agent as a Fleet Server. I've opened a PR with the proposed changes: #1184

@michel-laterman I'm guessing a bit at what I think the settings are so please take a look and let us know if anything needs fixing. We would still need to document setting up certificates between Elastic Agent and Kibana, which I guess would best fit in Configure SSL/TLS for self-managed Fleet Servers.

[michel-laterman]

We have documentation for container env vars that fleet-server uses during bootstrapping; however these are under the elastic-agent install instructions: https://www.elastic.co/guide/en/fleet/current/agent-environment-variables.html#env-prepare-kibana-for-fleet

I think we can re-organize some pages to make it more clear that we expect an end user to only deploy a fleet-server as an agent component and better utilize the agent installation instructions.

Additionally we should go over the env vars, i'm not certain if KIBANA_FLEET_SETUP still applies to agents in containers anymore

[kilfoyle]

Thanks @michel-laterman. I'll wait for your suggestions on this, as well as for #1184 "Add Fleet Server install steps on K8s and Docker".

If/when the Elastic Agent install pages have everything required to install Fleet Server, we probably just need to mention that somewhere in the Deployment models section.

[lucabelluccini]

Regarding the KIBANA_FLEET* env vars, please note we had users (see private issue created by Xavier) where the enrollment was failing.

[kilfoyle]

Looks like we're nearing completion on this:

• 1. Add steps for setting up Fleet Server on Kubernetes:
  • a. Add the Fleet Server setup steps to the Docker and Kubernetes pages Run Elastic Agent in a container and Run Elastic Agent on Kubernetes managed by Fleet
  • See Add Fleet Server install steps on K8s and Docker #1184
  • b. Add a "Docker and Kubernetes" section to the the Deployment models page with links to the above.
  • See Add 'Docker and Kubernetes' to Deployment Models page #1334
• 2. Mention on the Deployment Models page that managed Elastic Agent can be set up in ECK, with a link to the relevant docs.
  • See Add ECK deployment type with link to docs #1176
• 3. From Michel: We should go over the env vars, i'm not certain if KIBANA_FLEET_SETUP still applies to agents in containers anymore
• 4. Update the default port assignments to remove the "Elastic Agent -> Fleet" entry.
  • This has been discussed separately in Slack, and I understand from that discussion that there is a scenario (maybe during initial setup?) where Agent talks directly to Fleet/Kibana. So I think we should leave this as is. Let me know anyone if you disagree.

---

@michel-laterman Can you please help with number 3, and also review my small PR for number 1?

@merlixelastic, @lucabelluccini, @nimarezainia Let me know if you think we're missing anything.

[michel-laterman]

KIBANA_FLEET_SETUP has been removed for a while: elastic/elastic-agent#2910

We're missing some env vars in our public docs (as well as from the output of the container help command):

• FLEET_FORCE - same as the --force flag
• ELASTIC_AGENT_CERT & ELASTIC_AGENT_CERT_KEY - paths to mtls cert and key the agent will use to connect to fleet-server. (corresponding var for passphrase is missing)
• FLEET_SERVER_CLIENT_AUTH - set fleet-server mtls settings, one of: none (default), optional, required
• FLEET_SERVER_ES_CERT & FLEET_SERVER_ES_CERT_KEY - fleet-server -> es mTLS cert paths. (corresponding var for passphrase is missing)
• FLEET_SERVER_INSECURE_HTTP - expose fleet-server as an http server
• FLEET_HEADER - I think this one is headers the agent will send to fleet-server
• FLEET_KIBANA_HEADER - headers used when contacting Kibana
• FLEET_SERVER_TIMEOUT - how long elastic-agent will wait for fleet-server to checkin as healthy
• FLEET_DAEMON_TIMEOUT - how long fleet-server will wait during the bootstrap process for elastic-agent

[kilfoyle]

Nice catch @michel-laterman! I'll add the above env vars to the docs page.

[kilfoyle]

@michel-laterman I've opened this PR to add the missing settings. For easier reviewing, I've opened this gdoc and everyone from this issue should have access.

@nimarezainia FYI

[kilfoyle]

@merlixelastic and @lucabelluccini I've just merged this PR to add the environment variables for container setup that hadn't already been documented. That completes the list of "To dos" in this comment, so I think this issue can be closed now.

If we've missed anything, please let us know and I can open a new docs issue. Thanks!

And thanks @michel-laterman for the help on this!

[lucabelluccini]

Thank you @kilfoyle

My only comment/follow up is @michel-laterman reports the code was removed elastic/elastic-agent#2910, but we had a user hitting the error below when enrolling an Elastic Agent / Fleet Server on 8.13 on a container/kubernetes installation:

Kibana fetch policy failed: http GET request to https://kibana-multi:5601/api/fleet/agent_policies fails: fail to execute the HTTP GET request: Get "https://kibana-multi:5601/api/fleet/agent_policies": x509: certificate signed by unknown authority.

At this point I wonder if we should open a separate issue to investigate.

[kilfoyle]

At this point I wonder if we should open a separate issue to investigate.

@lucabelluccini It's a great question but it's not something I would know how to investigate, so I'd really appreciate if this could go into a separate issue in the ingest-dev repo.