-
Notifications
You must be signed in to change notification settings - Fork 130
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Agent with Elastic Defend remains unhealthy for long when invalid configuration is added for sometime and then updated with correct. #3721
Comments
Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane) |
@manishgupta-qasource Please review. |
Secondary review for this ticket is Done |
The degraded input I see is:
|
This is the same symptom as #3654 I suspect. I do see a period where endpoint transitioned from Healthy to Failed which is different: {"log.level":"warn","@timestamp":"2023-11-07T11:16:27.995Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":540},"message":"Component state changed endpoint-1438a4a0-7d56-11ee-b010-9371de552274 (STARTING->DEGRADED): Degraded: endpoint service missed 1 check-in","log":{"source":"elastic-agent"},"component":{"id":"endpoint-1438a4a0-7d56-11ee-b010-9371de552274","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:17:28.004Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":540},"message":"Component state changed endpoint-1438a4a0-7d56-11ee-b010-9371de552274 (DEGRADED->FAILED): Failed: endpoint service missed 3 check-ins","log":{"source":"elastic-agent"},"component":{"id":"endpoint-1438a4a0-7d56-11ee-b010-9371de552274","state":"FAILED","old_state":"DEGRADED"},"ecs.version":"1.6.0"} I also see a lot of the following message, but this might be due to frequent process restarts: {"log.level":"debug","@timestamp":"2023-11-07T11:16:23.019Z","log.origin":{"file.name":"runtime/manager.go","file.line":653},"message":"actions stream sent an invalid token; closing connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} |
Slightly before endpoint started missing check ins we got a policy change to remove endpoint from the policy: {"log.level":"debug","@timestamp":"2023-11-07T11:15:52.547Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":205},"message":"got action teardown for endpoint service","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:52.547Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":184},"message":"got teardown for endpoint service, tearingDown: false, tamperProtection: true","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:52.547Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":165},"message":"start teardown for endpoint service","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:52.547Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":172},"message":"set teardown timer 30s for endpoint service","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:52.547Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":178},"message":"process new comp config for endpoint service","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:52.547Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":408},"message":"update component configuration for endpoint service","ecs.version":"1.6.0"} |
OK I now see an attempted uninstall of endpoint with an invalid uninstall token: {"log.level":"debug","@timestamp":"2023-11-07T11:15:54.598Z","log.origin":{"file.name":"handlers/handler_action_policy_change.go","file.line":83},"message":"handlerPolicyChange: action 'action_id: policy:3f7ffbb0-7c64-11ee-a559-87f695dbf4e9:24:1, type: POLICY_CHANGE' received","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:54.598Z","log.origin":{"file.name":"handlers/handler_action_policy_change.go","file.line":107},"message":"handlerPolicyChange: emit configuration for action action_id: policy:3f7ffbb0-7c64-11ee-a559-87f695dbf4e9:24:1, type: POLICY_CHANGE","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:54.598Z","log.origin":{"file.name":"dispatcher/dispatcher.go","file.line":153},"message":"Successfully dispatched action: 'action_id: policy:3f7ffbb0-7c64-11ee-a559-87f695dbf4e9:24:1, type: POLICY_CHANGE'","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:55.068Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":244},"message":"got check-in for endpoint service, tearingDown: true, ignoreCheckins: false","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:55.068Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":150},"message":"stop check-in timer for endpoint service","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:55.068Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":154},"message":"stop connection info for endpoint service","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:55.068Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/conn_info_server.go","file.line":45},"message":"failed accept conn info connection: accept tcp 127.0.0.1:6788: use of closed network connection","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-07T11:15:55.068Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":332},"message":"stopping endpoint service runtime","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-07T11:15:55.068Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":348},"message":"endpoint service has checked in, send stopping state to service","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-07T11:15:55.068Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":356},"message":"uninstall endpoint service","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:55.068Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":660},"message":"uninstall endpoint-security service","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:55.789Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":156},"message":"FleetGateway calling Checkin API","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:55.789Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":186},"message":"Checking started","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:56.012Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":323},"message":"using previously saved ack token: 2c3382cb-5f4e-423a-909b-b84f85b2373b","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:56.012Z","log.origin":{"file.name":"remote/client.go","file.line":172},"message":"Request method: POST, path: /api/fleet/agents/1181c62e-7fb3-4aad-af1b-9b67fab6c3ee/checkin, reqID: 01HEMQY1FCWTA893APNH8THE9Q","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:56.012Z","log.origin":{"file.name":"remote/client.go","file.line":186},"message":"Creating new request to request URL https://07f9f186c0c94405a4900deb91931e25.fleet.europe-west1.gcp.cloud.es.io:443/api/fleet/agents/1181c62e-7fb3-4aad-af1b-9b67fab6c3ee/checkin?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.745Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: info: Main.cpp:299 Executing uninstall","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.745Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: info: Internal.cpp:51 Found config path [C:\\Program Files\\Elastic\\Endpoint\\elastic-endpoint.yaml]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.761Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: debug: Utilities.cpp:420 Tamper protection enabled","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.761Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: info: InstallLib.cpp:953 Checking installed uninstall protection artifacts","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.766Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: info: Internal.cpp:51 Found config path [C:\\Program Files\\Elastic\\Endpoint\\elastic-endpoint.yaml]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.768Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: info: InstallLib.cpp:712 No custom public key detected in Endpoint config","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.770Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: info: Crypto.cpp:1067 RSA signature verified","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.777Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: info: InstallLib.cpp:885 Failed to read os section of tamper-protection-config, continuing","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.777Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: info: InstallLib.cpp:975 Finished checking installed uninstall protection artifacts with result deny","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.777Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: info: InstallLib.cpp:1047 Finished checking command line provided uninstall resource result deny","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.777Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-07 11:15:56: error: InstallLib.cpp:1237 Invalid uninstall token","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-07T11:15:56.809Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":359},"message":"failed endpoint service uninstall, err: 2023-11-07 11:15:56: error: InstallLib.cpp:1237 Invalid uninstall token: exit status 284","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:56.809Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":364},"message":"set endpoint service runtime to stopped state","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:56.809Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":244},"message":"got check-in for endpoint service, tearingDown: false, ignoreCheckins: true","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-11-07T11:15:56.809Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":244},"message":"got check-in for endpoint service, tearingDown: false, ignoreCheckins: true","ecs.version":"1.6.0"} |
@amolnater-qasource it looks to me like there was a policy reassignment or that Elastic Defend was installed and then uninstalled, which is missing from the steps in the description. Did either of these two things happen? |
Hi @cmacknz Thank you for looking into this.
No, we didn't do this. However when we had the invalid output set we waited for endpoint to get Unhealthy. For confirmation, we have revalidated this issue on released 8.11.0 kibana cloud environment and found it still reproducible.
We are more frequently able to reproduce this issue when tamper protection settings are enabled. Sharing the latest logs and json below: Please let us know if anything else is required from our end. |
Hi Team, We have revalidated this issue on latest 8.12.0 BC5 kibana cloud environment and found it still reproducible. Observations:
Build details:
Artifact Link: https://staging.elastic.co/8.12.0-9f05a310/downloads/beats/elastic-agent/elastic-agent-8.12.0-windows-x86_64.zip Agent Logs: Agent.json: Hence, we are reopening this issue. Please let us know if anything else is required from our end. Thanks! |
@amolnater-qasource is it still happening in 8.14? |
We have revalidated this issue on 8.14.0 BC7 Kibana cloud environment and found it fixed now. Observations:
Agent Diagnostics: Build details: Hence, we are closing and marking this issue as QA:Validated. Thanks! |
Kibana Build details:
Host OS: Windows
Preconditions:
[Update]
Steps to reproduce:
NOTE:
Screenshot:
Expected Result:
Agent with Endpoint remains should get back Healthy when invalid configuration is added for sometime and then updated with correct.
Agent.json:
ec2amaz-8ag00sp-agent-details.zip
Logs:
elastic-agent-diagnostics-2023-11-07T11-28-06Z-00.zip
The text was updated successfully, but these errors were encountered: