[All OS's]: Agent goes unhealthy when some other output is set to Default output.

[amolnater-qasource]

Kibana version: 8.2 Snapshot Kibana cloud environment

Host OS: Windows

Build details:
VERSION: 8.2.0 Snapshot
BUILD: 51431
COMMIT: a743498436a863e142592cb535b43f44c448851a
Artifact link: https://artifacts-api.elastic.co/v1/search/8.2-SNAPSHOT

Preconditions:

1. 8.2 Snapshot Kibana cloud environment should be available.
2. Windows agent should be installed.

Integrations:
System and Endpoint Security.

Steps to reproduce:

1. Login to Kibana environment.
2. Navigate to Fleet>Agent Policy tab.
3. Set output as "default" for agent policy.
4. Create a new output under Fleet Settings and set it to Default.
5. Observe within 60 seconds Windows Agent goes Unhealthy.

Expected Result:
Agent should not go unhealthy on changing Default output to some other user created output.

Logs:
logs.zip

Screen Recording:

Windows.Agent.Policy.-.Agent.policies.-.Fleet.-.Elastic.-.Google.Chrome.2022-03-28.12-48-34.mp4

Windows Output:

Note:

• Issue is not observed on Linux agent.

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[ph]

@lykkin might be related to you changes?

[amolnater-qasource]

Hi Team
We have revalidated this issue on latest 8.2 BC-4 Kibana cloud environment and found it reproducible on all OS's.
We have updated the title to avoid any confusions.

• We have set another output under policy settings, and it gets Agents Unhealthy.

Integrations:
System and Endpoint Security

Platforms Validated:

• Windows
• Ubuntu
• Centos
• MAC

Build details:
BUILD: 52005
COMMIT: 9a5003d8cf0062bf24ef64d6712b44823888cc03
Artifact Link: https://staging.elastic.co/8.2.0-3b2b9b86/summary-8.2.0.html

Logs:
elastic-agent-diagnostics-2022-04-21T06-36-28Z-00.zip

Screenshot:

Please let us know if anything else is required from our end.
Thanks

[jlind23]

@lykkin any progress here?

[lykkin]

In the logs we see

{"log.level":"error","@timestamp":"2022-03-28T08:00:31.761Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-03-28T04:00:31-04:00 - message: Application: endpoint-security--8.2.0-SNAPSHOT[3674c80e-9c57-47c2-9f2e-1e967c53b755]: State changed to FAILED: failed to start connection credentials listener: listen tcp 127.0.0.1:6788: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}

repeated many times. Out of curiosity, if you remove endpoint from the agent does the issue still occur?

Also pinging @elastic/endpoint in case this is a known issue.

[amolnater-qasource]

Hi @lykkin
We have revalidated this issue on latest 8.2 BC-4 and yes we have observed that agent goes to unhealthy state only when Endpoint Security is added to it.
Issue is not reproducible without ES.

Build details:
BUILD: 52005
COMMIT: 9a5003d8cf0062bf24ef64d6712b44823888cc03
Artifact Link: https://staging.elastic.co/8.2.0-3b2b9b86/summary-8.2.0.html

Screenshot:

Logs:
elastic-agent-diagnostics-2022-04-29T11-31-28Z-00.zip
endpoint-000000.zip

Thanks

[ferullo]

Can you share elastic-endpoint.yaml?

[amolnater-qasource]

Hi @ferullo
Please find below elastic-endpoint.yaml from 8.2 BC-4 Endpoint:
elastic-endpoint.zip

Please let us know if anything else is required from our end.
Thanks

[intxgo]

The endpoint log looks fine. Endpoint connects successfully to Agent.

The former log is an Agent log:

{"log.level":"error","@timestamp":"2022-03-28T08:00:31.761Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-03-28T04:00:31-04:00 - message: Application: endpoint-security--8.2.0-SNAPSHOT[3674c80e-9c57-47c2-9f2e-1e967c53b755]: State changed to FAILED: failed to start connection credentials listener: listen tcp 127.0.0.1:6788: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}

It looks like the Agent is trying to start duplicate TCP server on localhost:6788. This localhost port is used by Endpoint as bootstrap connection port. It's really not related to any stack Output and since it's not possible to run several Endpoints on the same host it would make no sense to use any advanced socket options like SO_REUSEADDR on the TCP server side. Most likely it's a bug in Agent logic.

On a side note, I'm curious what is the purpose of the custom output, how should it work? Is it supposed to alter behavior of Endpoint in any way? The only documentation I was able to find it's rather shallow. Digging deeper from that page, I found the following snippet here

Only a single output may be defined.