APM Server transport error (ERR_SOCKET_TIMEOUT): Socket timeout

[kizzuc25]

What should I do to solve the problems below?
I have no idea why the APM Server transport error (ERR_SOCKET_TIMEOUT): Socket timeout"}` error occurs.

{"log.level":"error","@timestamp":"2024-05-16T05:19:14.619Z","log":{"logger":"elastic-apm-node"},"dt.entity.host":"HOST-40CA2xxxxxxxAA","dt.entity.host_group":"HOST_GROUP-0C777xxxxxxxxA3A","dt.entity.process_group":"PROCESS_GROUP-0AC212xxxxxxxxDB3F8","dt.entity.process_group_instance":"PROCESS_GROUP_INSTANCE-xxxxxxx","dt.host_group.id":"NT","ecs":{"version":"1.6.0"},"message":"APM Server transport error: error fetching APM Server version: timeout (30000ms) fetching APM Server version"} {"log.level":"error","@timestamp":"2024-05-16T05:19:14.621Z","log":{"logger":"elastic-apm-node"},"dt.entity.host":"HOST-40CxxxxxA0AA","dt.entity.host_group":"HOST_GROUP-0C77xxxxxx3A","dt.entity.process_group":"PROCESS_GROUP-0AC21xxxxxx3F8","dt.entity.process_group_instance":"PROCESS_GROUP_INSTANCE-0A26xxxxxxx39ED16","dt.host_group.id":"T","ecs":{"version":"1.6.0"},"message":"APM Server transport error (ERR_SOCKET_TIMEOUT): Socket timeout"}

The container comes up and then you get the following error message:

{"log.level":"debug","@timestamp":"2024-05-16T06:14:59.145Z","log":{"logger":"elastic-apm-node"},"dt.entity.host":"HOST-40CA2xxxxxAA","dt.entity.host_group":"HOST_GROUP-0C7779xxxxxxA3A","dt.entity.process_group":"PROCESS_GROUP-0ACxxxxxx8x3F8","dt.entity.process_group_instance":"PROCESS_GROUP_INSTANCE-094A53xxxxxC472","dt.host_group.id":"NT","ecs":{"version":"1.6.0"},"message":"no active transaction found -cannot build new span"} {"log.level":"debug","@timestamp":"2024-05-16T06:14:59.145Z","log":{"logger":"elastic-apm-node"},"dt.entity.host":"HOST-40CA2xxxx0AA","dt.entity.host_group":"HOST_GROUP-0Cxxxx9DF145xxxA3A","dt.entity.process_group":"PROCESS_GROUP-0AC2xxxxDB3F8","dt.entity.process_group_instance":"PROCESS_GROUP_INSTANCE-094AxxxxxC472","dt.host_group.id":"T","ecs":{"version":"1.6.0"},"message":"intercepted call to http.request {\"id\":null}"}

[trentm]

The first log message:

"APM Server transport error: error fetching APM Server version: timeout (30000ms) fetching APM Server version"

is saying that the APM agent running in your app got a timeout talking to your configured APM Server URL.
The very first log message from the APM agent will include a dump of its configuration -- which will show what it things its serverUrl value is.

For some reason, that serverUrl is not responding to requests. It looks like it is connecting to that URL, otherwise the error would be ECONNREFUSED.

Are you able to manually run a command from the same machine/VM/container where the APM agent is running? If so, this should be the equivalent of the first request the APM agent is attempting:

curl -v $serverUrl/

A possibility could be that your APM server is down, but I'm not positive. Are you able to look at your APM server logs?

[kizzuc25]

hi
If you run the curl command in docker by specifying the IP and port of the APM server, you can see the response.
However, I cannot retrieve server information using the command you sent.
Do I need to specify a serverUrl value?

thank you

[trentm]

Do I need to specify a serverUrl value?

Yes, you need to fill that in with the URL for your APM server. For example, the APM server URL for a deployment of mine on cloud.elastic.co is https://trentm-deployment17.apm.us-west1.gcp.cloud.es.io. For me, running that command gives a response like this:

% curl -i https://trentm-deployment17.apm.us-west1.gcp.cloud.es.io/
HTTP/2 200
date: Tue, 21 May 2024 21:28:53 GMT
x-cloud-request-id: TqX8ENw4QQeycOWecjqMwg
x-content-type-options: nosniff
x-found-handling-cluster: e1271517748b49929a101be02c2966c3
x-found-handling-instance: instance-0000000000
content-length: 0

[kizzuc25]

My apm server doesn't have a specific url.
I am also making requests via IP from apm/client in a different environment.
However, other environments are normal.

# curl -i http://10.10x.x0.x35:8200
HTTP/1.1 200 OK
Content-Type: application/json
X-Content-Type-Options: nosniff
Date: Wed, 22 May 2024 01:59:49 GMT
Content-Length: 150

{
  "build_date": "2023-01-31T04:33:06Z",
  "build_sha": "71a8b4c241eb5b4609862c8354d2aa2270f6c568",
  "publish_ready": true,
  "version": "7.17.9"
}

[trentm]

If you run the curl command in docker by specifying the IP and port of the APM server, you can see the response.

Can you confirm that the IP and port that works for you is the same as the serverUrl you are using for the Node.js APM agent config in that Docker container?

I'm not sure what else to say. The APM agent is configured with a URL to talk to the APM server (https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuration.html#server-url). From within that container that is running your app, a simple HTTP GET request to that URL fails.