[Fleet] Infra work / new test creation, for coming Fleet Server component

[EricDavisX]

will add more notes as it is clearer... main tracking issue: integrations-dev/issues/389

What we know now:

• Any 'Agent' can be set up as a Fleet Server, so our Docker usage is acceptable, as is of course the standard Centos / Debian / UBI8 / other images
• Stand-alone Agent scenarios won't be modified to use Fleet Server, as they will communicate w ES direct still
  ... An Agent running as Fleet Server will or will not be still a regular 'Agent' as well?
• We could validate that Stand-Alone Agent cannot be setup as a Fleet Server
• I think we should build the expectation of using Fleet Server into the existing install scripts... need to think on this more.

[EricDavisX]

Discussion notes from meeting on setting up Fleet Server:

general steps:

step 1) Agent needs to be running as a daemon

step 2) install (or enroll) command, pass in option --fleet_server with url + uid/pwd

• no api key usage
• which writes into fleet.yml

then the running daemon is told to reexec (on linux and mac it injects into the same pid) - and it reads the yml and brings up the fleet-server subprocess

if Kibana is not running, Fleet-server will wait, polling ES for the policy it should be on and confirming the index exists. wait for 2 mins, then times out.

Fleet server will then allow enrollment to begin

waits for 'degraded' state, then it enrolls (locally). rewrites fleet.yml - then the bootstrap flag is not included (set to false). and reexecs again (2nd time).

must include the Fleet Server integration in Kibana policy, get that policy id from the enroll command

on Kibana side, just a feature flag setup. not merged yet, but coming soon. :)

• will need to visit the /setup endpoint, but what about actually clicking button in Kibana?

e2e-test steps / questions:

• process name is fleet-server running as a child of agent
• more tests: passing in bad PWD
• tests: using user with missing permissions

[EricDavisX]

FYI - the basic process above should be discussed in: https://github.com/elastic/fleet-server/blob/master/README.md
and it basically amounts to just calling:
sudo ./elastic-agent enroll --enrollment-token {enrollment-token} --fleet-server http://[elastic-user]:[elastic-user-creds]@[fleet-server-url]:9200 --fleet-server-policy {fleet-server-policy-id}

the {fleet-server-policy-id} will be generated by default when Kibana starts up with Fleet in 7.13 and beyond. so it's name / id can be queried like any other policy id.

[mdelapenya]

Is fleet-server-url --fleet-server URL flag the same as Elasticsearch URL?

[EricDavisX]

no, it is the URL/port of the Fleet Server process running on the edge (Agent). Any Agent CAN run a fleet server process, but only 1 is required. the journey of reading can continue if you want to dig into more architecture docs (which may be out of date) - https://github.com/elastic/obs-dc-team/issues/389

I'd like to pull in @blakerouse for expertise confirming what literal options we need to set up here. @nchaulet is the Engineer contributing on the Kibana transition side. Nicolas, it would help us to know which specific APIs are being moved out of Kibana to Fleet Server and what the new routes are. I don't know if we have that doc'ed already.

[blakerouse]

@mdelapenya I don't see a --fleet-server-url option currently listed in @EricDavisX example. Are you speaking about the --fleet-server parameter? That is the connection string that Fleet Server uses to communicate with elasticsearch.

[nchaulet]

@EricDavisX we will remove the agent endpoints (enroll, checkin and acks also the endpoint artifacts too) these endpoints will be move to fleet server.

[mdelapenya]

@mdelapenya I don't see a --fleet-server-url option currently listed in @EricDavisX example. Are you speaking about the --fleet-server parameter? That is the connection string that Fleet Server uses to communicate with elasticsearch.

No, I meant the value of that flag: --fleet-server http://[elastic-user]:[elastic-user-creds]@[fleet-server-url]:9200. Because it uses the same port as Elasticsearch, I though there was a mistake in the value. But no, that URL is the one used by the fleet-server (another agent acting as server)

[mdelapenya]

@EricDavisX we will remove the agent endpoints (enroll, checkin and acks also the endpoint artifacts too) these endpoints will be move to fleet server.

During the move, it would be great if we all contribute the changes to the e2e before the PRs are merged in upstream repos (Kibana/Beats) so that we do not chase the changes after they are pushed. I offer myself to guide through the code so that we can cooperate to achieve it

[mdelapenya]

I'm getting progress with this scenario (see #900), but I'd like to add more consistent verifications and use cases. @EricDavisX I'd appreciate your help here with the definition of the scenarios/steps.

@blakerouse I'd like to know more about this scenario I want to add:

@start-fleet-server
Scenario Outline: Deploying the <os> fleet-server agent
  Given a "<os>" agent is deployed to Fleet with "tar" installer in fleet-server mode
  When an "<os>" agent is deployed to Fleet with "tar" installer using the fleet-server
  Then the agent is listed in Fleet as "online"

How the second agent communicates to the first one? How does it discover the server (name and port)?

[blakerouse]

@mdelapenya You have to give the the second agent the IP/hostname of the first agent for it to communicate.

[EricDavisX]

I need to learn more before I can help outline scenarios, if we want to make progress sooner we can get Blake's input.

[mdelapenya]

@mdelapenya You have to give the the second agent the IP/hostname of the first agent for it to communicate.

Any port needed?

[nchaulet]

@mdelapenya by default the port for fleet server will be 8220 also you would have to configure the fleet Kibana url to be the fleet server url

[EricDavisX]

@nchaulet what is the api endpoint & params/body to use to make that call to set up the Fleet Server in Kibana?

[EricDavisX]

As noted prior in #900 we are discussing Fleet Server set up, let us try to keep the tech details just in one place to help solve the confusion / make-it-work.

I am making notes, and have not seen it work yet, but I'm close and I know others have. Docker usage has been confirmed. Using a standard Agent (.zip, .tar.gz, etc) has not been yet.

Things to NOTE that we need:

1. make sure Fleet Server URL is setup in Fleet settings (with port 8220)
2. to enroll, we get the enroll token from the current ‘deploy Agent’ flyout for the 'Default Fleet Server' policy
3. then we get the Fleet Server policy id from the Agent policies page / listing
4. then we can call one form or another of the enrollment with that info.

Note, I think the param 'fleet-server' is possibly mis-named. What the value for it is, however, is the connection to Elastic Search, which the Fleet server process needs (to pass down in policy to the other Agents).

The other Agents connect to the Fleet-Server with the Kibana side setting for 'Fleet Server' in the settings 'gear' icon in the UI, the same location where we are currently setting the Kibana / ES connection urls. This is where port 8220 is used. Indeed the only thing running on 9200 is the Elasticsearch.

for Docker usage, this is what I see as the most up to date info in the Fleet Server repo:
KIBANA_HOST=https://snapshot.kb.us-central1.gcp.foundit.no:9243 KIBANA_USERNAME=elastic KIBANA_PASSWORD=asdfasdfasd ELASTICSEARCH_HOST=https://snapshot.es.us-central1.gcp.foundit.no:9243 ELASTICSEARCH_USERNAME=elastic ELASTICSEARCH_PASSWORD=asdfasdfasdf111 KIBANA_FLEET_SETUP=1 FLEET_SERVER_ENABLE=1 ./elastic-agent container

For non-Docker usage, the call will be something more 'normal' looking like it does when you enroll an Agent currently. I will post back to this thread when I have confirmed what works. I think I'm close.

Update: TBD

[blakerouse]

You should NOT use the container command outside of the actual container. It expects items at a specific path and it will alter your /usr/share/elastic-agent that should not be touched unless its in a container.

The container command is not exposed over --help for this exact reason. It should only be called from an actual container.

[mdelapenya]

As I mentioned in the implementation PR (see #900 (comment))

We would need an example of the canonical command that does that:

When I want to bootstrap the fleet server in a host
Then I run the FOO command with FLAGS and VALUES

As a follow-up, once this is merged and passing, we'd like to add scenarios for another agent using the bootstrapped fleet-server. Something like:

Given a fleet-server already bootstrapped
When I enroll a second agent in Fleet using the fleet-server
Then the agent is "online" in Fleet

[jalvz]

1. make sure Fleet Server URL is setup in Fleet settings (with port 8220)

How do we do that programmatically?

2. to enroll, we get the enroll token from the current ‘deploy Agent’ flyout for the 'Default Fleet Server' policy
3. then we get the Fleet Server policy id from the Agent policies page / listing

Do we need both?

[blakerouse]

@jalvz You do not need 3 if you are enrolled the Agent with Fleet Server into the Default Fleet Server policy. You only need 2, the enrollment token needs to be to the Default Fleet Server policy.

[EricDavisX]

I posted a note in the other issue, hopefully providing canonical install steps.

[mdelapenya]

I think we can close this, as #1078 was merged. Please reopen if needed

Thanks!

[EricDavisX]

I added the above issue as a continuation of overall support we need / intend:
#1266

this was acceptably closed out as initial support and usage, but we never reviewed and determined test depth / coverage desires at full depth.