[Ingest Manager]: Unhealthy Agents are displayed after seemingly successful agent install

[dikshachauhan-qasource]

Kibana version:
Kibana:7.11.0 BC1 on prem environment

Elasticsearch version:
Elasticsearch: 7.11.0 BC1 on prem environment

Host OS and Browser version:
Windows 10, All

Original install method (e.g. download page, yum, from source, etc.):
7.11.0 BC1 on prem environment

Description
[Ingest Manager]: Agents logs not getting displayed after successful agent installation

Build Details:

Version : 7.11.0 BC1
Platform : production
BUILD 37399
COMMIT f3abc08ac648f8b302733c5c22a39048314a027c

Artifact link:BC1

Preconditions:

1. Elasticsearch & Kibana should be running on 7.11 BC1 artifacts.
2. Elastic-Agent windows 64 bit ( Test-Signing ON) should be installed on machine.

Steps to reproduce:

1. Login to Kibana cloud environment.
2. Go to Agents tab. Enable 'Create user and enable central management'
3. RDP to endpoint. Install Elastic-agent using default policy Token.
4. Observe successful installation message is displayed.
5. Navigate back to Kibana, Go to Fleet> Agents.
6. Wait Agent shows up.
7. Observed that Agent is Installed but with "Unhealth" Status with no logs no data stream and due to this Endpoint Security Integration installation is also impacted.

Note: we have got the update of Elastic Agent package on BC1 artifact is not signed yet . so turned Test-Signing ON on the machine .Conversation link here

Reference ticket Id/Slack Conversion:
N/A

Actual Result

1. Agents logs not getting displayed after successful agent installation.
2. Agent is unhealthy in status.
3. No Data streams

Expected Result

1. Agents logs should be displayed after successful agent installation.
2. Agent should be Healthy in status
3. Data stream should have respective data of installed agents

What's working
N/A

What's not working

• Endpoint Security is not installing
• No Logs collected for Agent
• No Datastream for Agent

Screenshot:

Logs:
N/A

[dikshachauhan-qasource]

@karanbirsingh-qasource Please review

[karanbirsingh-qasource]

reviewed and assigned to @EricDavisX

[EricDavisX]

@ph I think this is potentially a blocker.

[ferullo]

As I understand it this is blocking all testing of Agent and Endpoint. If that's true, we should get an off schedule BC2 once this is resolved.

[nchaulet]

Not sure if it's the same error but with the latest 7.11 agent build I got this error enrolling an agent with endpoint (it's working without endpoint)

{"log.level":"error","@timestamp":"2020-12-17T10:40:42.319-0500","log.origin":{"file.name":"application/fleet_gateway.go","file.line":168},"message":"failed to dispatch actions, error: operator: failed to execute step sc-run, error: operation 'operation-verify' failed to verify endpoint-security.7.11.0: 2 errors occurred:\n\t* open /Library/Elastic/Agent/data/elastic-agent-fc48a3/downloads/endpoint-security-7.11.0-darwin-x86_64.tar.gz.sha512: no such file or directory\n\t* open /Library/Elastic/Agent/data/elastic-agent-fc48a3/downloads/endpoint-security-7.11.0-darwin-x86_64.tar.gz.sha512: no such file or directory\n\n: operation 'operation-verify' failed to verify endpoint-security.7.11.0: 2 errors occurred:\n\t* open /Library/Elastic/Agent/data/elastic-agent-fc48a3/downloads/endpoint-security-7.11.0-darwin-x86_64.tar.gz.sha512: no such file or directory\n\t* open /Library/Elastic/Agent/data/elastic-agent-fc48a3/downloads/endpoint-security-7.11.0-darwin-x86_64.tar.gz.sha512: no such file or directory\n\n","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-12-17T10:40:42.319-0500","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2020-12-17T10:40:42-05:00: type: 'ERROR': sub_type: 'FAILED' message: Application: endpoint-security--7.11.0[1111d390-407e-11eb-98ab-c7c54718bb5d]: State changed to FAILED: operation 'operation-verify' failed to verify endpoint-security.7.11.0: 2 errors occurred:\n\t* open /Library/Elastic/Agent/data/elastic-agent-fc48a3/downloads/endpoint-security-7.11.0-darwin-x86_64.tar.gz.sha512: no such file or directory\n\t* open /Library/Elastic/Agent/data/elastic-agent-fc48a3/downloads/endpoint-security-7.11.0-darwin-x86_64.tar.gz.sha512: no such file or directory\n\n","ecs.version":"1.5.0"}

[ph]

@blakerouse Can you take a look at this one?

[ph]

@EricDavisX @ferullo I've looked at the last BC and I think there is something wrong it, the endpoint and the asc files arent included in that package which would cause the issue that @nchaulet found.

Is there an issue that discuss any building problem?

[blakerouse]

@ph Only issue I know about was that BC1 was requiring that Agent was manually signed and that was going to be fixed in BC2. I would expect the endpoint files to still be present, even without the signing of Agent.

[EricDavisX]

we logged the missing Endpoint files in the artifact in Infra # 25562

• that may very well explain this as the use case includes Endpoint and the 'health' is correctly noting the problem. Right?

[ph]

Should be fixed in BC1.

[EricDavisX]

It seems fixed in BC1 (overwritten, in early afternoon EST as the same version) - tested on Linux. Note, I am seeing an 'unhealthy' problem state for Agent that seems related to Endpoint still, as testing with the newer artifact and it is logged separately as elastic/kibana#86097

closing this out.