You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, everyone.
I set up a cluster with three Linux servers using elastic version 8.11.1. The information for each server is as follows: Server 1 ip-10.150.3.12 elasticsearch+kibana, server 2 ip-10.150.3.15 elasticsearch+elastic agent, and server 3 ip 10.150.3.17 elasticsearch+fleetserver. They are self-managed clusters and are configured with TSL, using self-generated certificates. When installing fleetserver and agent, I didn't do any configuration for Elasti-agent.yml, just tar and install. fleetserver and agent are up and running, but in kibana, don't see logs. May I ask where I need to troubleshoot the problem next?
I uploaded my diagnostics log at the end, hoping to help with this problem. Thank you so much!
sudo ./elastic-agent install --url=https://10.150.3.17:8220 \
> --fleet-server-es=https://10.150.3.17:9200 \
> --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MTE5NTgxNDE1NjE6Zi1LN0hJQVZUSUdlMjJZejJ5QU9YQQ \
> --fleet-server-policy=5906c780-ec1c-11ee-9cd9-39245a862599 \
> --fleet-server-es-ca-trusted-fingerprint=BC8F64D445A43597BE161251826AAA1B302C5C76BDE744C479E36C8481E10704 \
> --certificate-authorities=/home/elastic/downloads/beats/elastic-agent/fleet-server/ca/ca.crt \
> --fleet-server-cert=/home/elastic/downloads/beats/elastic-agent/fleet-server/ca/fleet-server.crt \
> --fleet-server-cert-key=/home/elastic/downloads/beats/elastic-agent/fleet-server/ca/fleet-server.key \
> --fleet-server-port=8220 \
> --fleet-server-es-insecure
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
Copying files......................... DONE
Installing service... DONE
Starting service... DONE
Enrolling Elastic Agent with Fleet...................{"log.level":"info","@timestamp":"2024-04-01T15:56:31.046+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":774},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
...............{"log.level":"info","@timestamp":"2024-04-01T15:56:35.050+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":788},"message":"Fleet Server - Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T15:56:35.276+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":479},"message":"Starting enrollment to URL: https://10.150.3.17:8220/","ecs.version":"1.6.0"}
....{"log.level":"info","@timestamp":"2024-04-01T15:56:36.315+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":277},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
DONE
Elastic Agent has been successfully installed.
{"log.level":"info","@timestamp":"2024-04-01T07:56:32.352Z","message":"Listening on localhost:8221","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:32.352Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:32.878Z","message":"Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-04-01T07:56:32.878Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":558},"message":"Unit state changed fleet-server-default (STARTING->DEGRADED): Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-04-01T07:56:32.878Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":558},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->DEGRADED): Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:35.505Z","message":"Checking enrollment key from database j85c2GqpRw-x2JEQAmEHjQ","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"mod":"enroll","fleet.enroll.apikey.id":"oQQ9f44B6eQfthdqM8OM","ecs.version":"1.6.0","service.name":"fleet-server","http.request.id":"01HTCAHZ9966N7HWFVQZV7ZBD3","server.address":"","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:35.758Z","message":"Found enrollment key b1FROWY0NEI2ZVFmdGhkcU04T006ajg1YzJHcXBSdy14MkpFUUFtRUhqUQ==","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.type":"fleet-server","server.address":"","mod":"enroll","fleet.enroll.apikey.id":"oQQ9f44B6eQfthdqM8OM","service.name":"fleet-server","http.request.id":"01HTCAHZ9966N7HWFVQZV7ZBD3","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:36.051Z","message":"Elastic Agent successfully enrolled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"mod":"enroll","fleet.access.apikey.id":"7aGomI4B2evC5-sW_pEv","event.duration":549099726,"service.name":"fleet-server","http.request.id":"01HTCAHZ9966N7HWFVQZV7ZBD3","fleet.enroll.apikey.id":"oQQ9f44B6eQfthdqM8OM","fleet.agent.id":"2ca3cb94-9bb4-43a8-b4b5-44a312a45de6","service.type":"fleet-server","server.address":"","http.response.body.bytes":1134,"ecs.version":"1.6.0","fleet.policy.id":"5906c780-ec1c-11ee-9cd9-39245a862599","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:36.314Z","log.origin":{"file.name":"cmd/run.go","file.line":302},"message":"reexec shutdown channel triggered","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:36.315Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","sig":"terminated","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.817Z","message":"Uptime: 3.097177795s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.817Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":163,"file.name":"log/log.go"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.817Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":73,"file.name":"api/server.go"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"api","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.817Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":531,"file.name":"instance/beat.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.983Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"Total metrics","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":195,"file.name":"log/log.go"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":80}},"total":{"ticks":210,"time":{"ms":210},"value":210},"user":{"ticks":130,"time":{"ms":130}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":10},"info":{"ephemeral_id":"a7add003-1f25-4a1e-97bd-84a95e33eb20","name":"metricbeat","uptime":{"ms":3090},"version":"8.11.1"},"memstats":{"gc_next":47007672,"memory_alloc":37410120,"memory_sys":58586376,"memory_total":52203896,"rss":90165248},"runtime":{"goroutines":29}},"libbeat":{"config":{"module":{"running":2,"starts":2,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":1,"dropped":0,"failed":1,"filtered":0,"published":1,"retry":0,"total":2},"queue":{"acked":0,"max_events":4096}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"system":{"cpu":{"cores":8},"load":{"1":0.67,"15":0.63,"5":0.62,"norm":{"1":0.0838,"15":0.0788,"5":0.0775}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"Uptime: 3.092760384s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":163,"file.name":"log/log.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"api","log.origin":{"file.line":73,"file.name":"api/server.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.origin":{"file.line":531,"file.name":"instance/beat.go"},"ecs.version":"1.6.0"}
Hello, everyone.
I set up a cluster with three Linux servers using elastic version 8.11.1. The information for each server is as follows: Server 1 ip-10.150.3.12 elasticsearch+kibana, server 2 ip-10.150.3.15 elasticsearch+elastic agent, and server 3 ip 10.150.3.17 elasticsearch+fleetserver. They are self-managed clusters and are configured with TSL, using self-generated certificates. When installing fleetserver and agent, I didn't do any configuration for Elasti-agent.yml, just tar and install. fleetserver and agent are up and running, but in kibana, don't see logs. May I ask where I need to troubleshoot the problem next?
I uploaded my diagnostics log at the end, hoping to help with this problem. Thank you so much!
elastic-agent-diagnostics-2024-04-01T08-21-38Z-00.zip
The text was updated successfully, but these errors were encountered: