[Windows and Linux ARM64]: Unable to install agent with Fleet Server URL.

[amolnater-qasource]

Kibana version: 8.0 Snapshot Kibana cloud environment

Host OS and Browser version: Windows and Ubuntu ARM64, All

Build Details:

  [Windows]Artifact link used: https://snapshots.elastic.co/8.0.0-c711755c/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-windows-x86_64.zip
  [Linux]Artifact link used: https://snapshots.elastic.co/8.0.0-c711755c/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-linux-arm64.tar.gz
  Build: 41953
  Commit: cb3c4e3a212255a9e9b8c89e784e0e452b661233

Preconditions:

1. 8.0 Snapshot Kibana cloud environment should be available.
2. Fleet Server should be setup for Windows and Linux endpoints.

Steps to reproduce:

1. Login to Kibana environment.
2. Run install command for Agent installation:

.\elastic-agent.exe install -f --url=http://10.0.5.233:8220 --enrollment-token=VUJFWnozZ0I3Y3RGakhkRVRNcUI6ZDc1SXljaW9TaENndkYxNWc5ekxaQQ== --insecure

3. Observe repetitive "Remote server is not ready to accept connections, will retry in a moment" errors.

Expected Result:
User should be able to install agent with Fleet Server URL.

Screenshots:

[ghost]

Reviewed and assigned to @EricDavisX

[amolnater-qasource]

Hi @EricDavisX
We have observed few changes in Fleet Server setup command and are able to setup Fleet Server with updated command.

Command used for Fleet Server:

.\elastic-agent install --fleet-server-es=https://elastic:ZnSHL0mDC7S3LSqHC0kxxxxx@security-deployment-dxxxx.es.us-central1.gcp.foundit.no:9243

However, we are still not able to install secondary agent and getting same errors: Remote Server not ready to accept connections

Currently we are blocked for testing because of this issue. Please let us know if there are any changes in Secondary Agent installation process.

Command used for Secondary Agent:

.\elastic-agent.exe install -f --url=http://10.0.5.2xx:8220 --enrollment-token=Q0MtYzdYZ0I4elpXRl96UHlSdFA6Tl9qaFZxxxxxxtiWlJzRGVnOFJBUQ== --insecure

Thanks
QAS

[EricDavisX]

@amolnater-qasource hi. I can cite that we need to keep the Fleet Server install and the subsequent Agent installs in sync in terms of security usage. If you intend to use the --insecure option with the Agent install, then you must use the --fleet-server-insecure-http option when first installing the Fleet Server.
That may help

[amolnater-qasource]

Hi @EricDavisX

As per feedback we have attempted to setup Fleet Server on 7.13.0 Self managed Kibana environment.
We are able to setup fleet server, however getting same Remote Server not ready to accept connections errors for secondary agent.

Build details:

Artifact link used: https://snapshots.elastic.co/7.13.0-6d604626/downloads/beats/elastic-agent/elastic-agent-7.13.0-SNAPSHOT-windows-x86_64.zip
Build: 40515
Commit: 9a38c64bf26c986a40dab999d103e59e5484a093

Screenshots:

Please let us know if anything else is required.
Thanks
QAS

[blakerouse]

You cannot install 2 Elastic Agents on the same machine. You will need to either install an Elastic Agent with a Fleet Server on that machine (then add integrations to the Default Fleet Server policy) or install an Elastic Agent without Fleet Server connecting to a remote Elastic Agent that is running the Fleet Server.

Seems that the installation of the first one works correctly. The second one is trying to overwrite the first one, being that you are using the -f it is not warning you that it is overwriting the other Elastic Agent (if you remove the -f you would get a warning telling you that). Because of this the second one first connects to the Fleet Server running on your host, but then Fleet Server its stopped because Elastic Agent install proceeds to stop it and overwrite it and that is why your getting that error.

I am going to close this, as it a user error and not an actual bug with Elastic Agent or Fleet Server.

[amolnater-qasource]

Hi @blakerouse
We have reported this issue for Cloud environment. However as per Eric's feedback we have attempted the same way on self-managed Kibana and shared our observations in this comment: #235 (comment)

Further we have re-attempted this today on 7.13.0 Snapshot Kibana Cloud environment and still getting the same issue.
Fleet Server install command used:(as given on UI)

.\elastic-agent.exe install --fleet-server-es=https://a835b48ce99848bf9eb7b5a1cf03xxxx.us-central1.gcp.foundit.no:443 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MTkwNjgzOTI1Njc6YjM2QzlsS0lSTktDdS1XSEoyxxxxxx

Secondary Agent install command used:(as given on UI)

.\elastic-agent.exe install -f --url=https://10.0.x.xxx:8220 --enrollment-token=SC1LLTkzZ0JYeWtVLTBLdzJpOTE6bHc4dlIzbElTaVdibi1yNmU5RUxxxx==

Screenshot:

As we are getting:Remote Server not ready to accept connections errors, hence re-opening this.

Build details:

Build: 40528
Commit: 506291847da6aa959066f0ec7d5bc17aa092d709

cc: @EricDavisX
Thanks
QAS

[blakerouse]

@amolnater-qasource The second agent is on a different host correct? Seems that it cannot communicate with the first host? Have you confirmed that there is not a network issue for them to communicate? Make the first one has its firewall enabled?

I have confirmed that installation of Elastic Agent w/ Fleet Server on Windows is working correctly. This at the moment seems to be a communication issue.

[amolnater-qasource]

Hi @blakerouse
We have revalidated this on 7.13.0 BC-1 Kibana Cloud Environment and we are still facing this issue.

The second agent is on a different host correct?

Yes

Have you confirmed that there is not a network issue for them to communicate? Make the first one has its firewall enabled?

Internet on both the machines is working fine. Firewall is "ON" too.

Steps followed:
Fleet-Server agent install:

1. Under Fleet Settings we added the Fleet Server host url: https://1.2.3.4:8220.
2. Installed Fleet server agent with .\elastic-agent install --fleet-server-es=https://elastic:password@elasticsearch-url:9243
3. Fleet Server Agent enrolled successfully with "Elastic Agent on Cloud" policy.

Secondary Agent install:

1. From Kibana UI we used the command for windows available under Add Agent: .\elastic-agent.exe install -f --url=https://1.2.3.4:8220 --enrollment-token=token
2. Getting Remote Server not ready to accept connections error.

Can you please check if we are missing any step?

Build details:

Artifact link: https://staging.elastic.co/7.13.0-f3a33857/downloads/beats/elastic-agent/elastic-agent-7.13.0-windows-x86_64.zip
Build: 40542
Commit: a93a7fe471bdcf6c06e18c0312af2435e7613e44

Thanks
QAS

[ruflin]

Is there an easy way to debug the connection? I filed elastic/elastic-agent#115 as a follow up idea.

[EricDavisX]

I have not yet tested the exact syntax, but I confirmed a work-around of using the preferred service token enrollment, so the test team should be unblocked now that the BC2 is available and is available on cloud-prod, particularly. I sent an email and will follow up with Amol, and will try to test this scenario too, since I know the code is working in other usage. I will report back.

[EricDavisX]

I have now gotten to test the Windows case as was originally cited. it looks working to me with the latest 7.13 BC 2 - @amolnater-qasource I can cite the specific items I think you will want in your setup and terminal usage:

• use https for the Kibana Fleet Server url: https://10.0.1.1:8220

• then also use https for the fleet server install / enrollment (but no other 'insecure' flags for the FS enrollment cal), like this:
  .\elastic-agent.exe install -f --fleet-server-es=https://asdfasdfadsfasdf.europe-west1.gcp.cloud.es.io:443 --fleet-server-service-token=ASDFASDF1341324ADSFASDF12341234

• then for the non-primary Agent installs use https as well, but do add the --insecure flag to the end of the enrollment command, like this:
  .\elastic-agent.exe install -f --url=https://10.0.7.165:8220 --enrollment-token=ADFASDF12341234ADSFASDF== --insecure

I tried both of these from a win2012 host (for the 2nd test, the non-primary agent, i did install it against a linux host fleet-server, fyi)

[amolnater-qasource]

Hi @EricDavisX
Thanks for the update. However we aren't getting any service token on 7.13 BC-2 Kibana cloud environment to install the new fleet server with service token command.
So as an workaround we attempted to install Fleet Server agent with following command:
.\elastic-agent install --fleet-server-es=https://elastic:password@elasticsearch-url:443

We were able to install Fleet Server successfully with "Elastic agent on cloud policy"(picked it by default).

Further when we attempted to install secondary agent using the command shown at Add Agent flyout after updating the fleet server host url we observed same error, as reported above.

then for the non-primary Agent installs use https as well, but do add the --insecure flag to the end of the enrollment command, like this:

We have also attempted with --insecure flag along with https, still same error message was displayed.
We have tried on Windows 10 x64 and Ubuntu 18 for primary and secondary agents.

Screenshot:

Thanks
QAS

[EricDavisX]

Thanks for the update. However we aren't getting any service token on 7.13 BC-2 Kibana cloud environment to install the new fleet server with service token command.

@amolnater-qasource @dikshachauhan-qasource if you can reproduce this one part we should log it separately and we can confirm the repro steps, it sounds like a bug if we can still see it.

[dikshachauhan-qasource]

Hi @EricDavisX

Earlier, we were not using Fleet server integration on attempting on install secondary agent in agent policy, so we were not able to find service token. However, this issue is not reproducible now.

Thanks for confirmation around buggy areas.