"Error: socket hang up" in Lambda #3670

Max101 · 2023-10-02T18:18:46Z

Hi I can see this issue has popped up a few times in the past but it seems like its been resolved so I am opening a new issue.

We are experiencing multiple Error: socket hang up in traces BUT not in logs. Our lambda finishes successfully, and there are no errors in the logs. However, where the issue is quite visible, is in APM. We have thousands of similar logs across most of our services.

We went to analyze our code and really cannot seem to find an issue. Additionally if this were an issue in our code, it would break, no?

We are on Lambda
using NodeJS nodejs16.x
Installed library version is dd-trace@4.4.0
Installed DD constructs "datadog-cdk-constructs-v2": "1.7.4",

We are using SST v2 (Serverless Stack) to deploy our lambda code

Our DD config looks like this

const dd = new Datadog(stack, `${stack.stackName}-datadog`, {
    nodeLayerVersion: 91, // Releases: https://github.com/DataDog/datadog-lambda-js/releases
    addLayers: true,
    extensionLayerVersion: 43, // Releases: https://github.com/DataDog/datadog-lambda-extension/releases
    captureLambdaPayload: true,
    enableColdStartTracing: true,
    apiKey: process.env.DATADOG_API_KEY,
    site: 'datadoghq.com',
    enableDatadogTracing: true,
    enableDatadogLogs: true,
    injectLogContext: true,
    env: process.env.NODE_ENV,
    service: stack.stackName,
    version: getDeploymentId(),
  } satisfies DatadogPropsV2);

The text was updated successfully, but these errors were encountered:

Harmonickey · 2023-11-20T15:20:28Z

We're seeing this too, we're just executing in a normal NodeJS context outside of lambda, so perhaps it's more widespread. However, it is throwing an exception to the caller as well so it crashes the executing context for us.

using NodeJS nodejs20.9.0
Installed library version is dd-trace@4.19.0

Error: socket hang up at connResetException (node:internal/errors:721:14) 
    at TLSSocket.socketOnEnd (node:_http_client:519:23) 
    at TLSSocket.emit (node:events:526:35) 
    at TLSSocket.emit (node:domain:488:12) 
    at TLSSocket.emit (/app/node_modules/dd-trace/packages/datadog-instrumentations/src/net.js:61:25) 
    at endReadableNT (node:internal/streams/readable:1408:12) 
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

tlhunter · 2023-12-21T21:03:05Z

@Harmonickey do you know what lead to exception? For example, an outgoing request?

viict · 2024-01-26T03:38:14Z

We are also suffering from this issue, although we are not hitting it only on lambdas.
Most of the time it is related to dynamodb calls.

We are using ddtrace-js v5.1.0

Error: socket hang up
    at connResetException (node:internal/errors:720:14)
    at TLSSocket.socketOnEnd (node:_http_client:525:23)
    at TLSSocket.emit (node:events:529:35)
    at TLSSocket.emit (/usr/src/api/node_modules/dd-trace/packages/datadog-instrumentations/src/net.js:69:25)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

luxion-lennart · 2024-01-26T14:00:55Z

We are also seeing this issue, we see it from lambda doing POST calls to DynamoDB.

Error: socket hang up
    at connResetException (node:internal/errors:720:14)
    at TLSSocket.socketCloseListener (node:_http_client:474:25)
    at TLSSocket.emit (node:events:529:35)
    at TLSSocket.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/net.js:61:25)
    at node:net:350:12
    at TCP.done (node:_tls_wrap:614:7)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)

CDK Construct: v1.8.0
Extension version: 48

carmargut · 2024-01-30T10:47:49Z

Same issue here. Not from a lambda, just when doing dynamodb calls.
Using 5.2.0

Error: socket hang up
    at connResetException (node:internal/errors:720:14)
    at TLSSocket.socketOnEnd (node:_http_client:525:23)
    at TLSSocket.emit (node:events:529:35)
    at TLSSocket.emit (/app/node_modules/dd-trace/packages/datadog-instrumentations/src/net.js:61:25)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

IgnacioFerreras · 2024-02-01T09:08:41Z

Hi we are suffering it too. using 3.33.0
Error: socket hang up at connResetException (node:internal/errors:705:14) at TLSSocket.socketCloseListener (node:_http_client:467:25) at TLSSocket.emit (node:events:525:35) at TLSSocket.emit (/var/task/node_modules/dd-trace/packages/datadog-instrumentations/src/net.js:61:25) at node:net:301:12 at TCP.done (node:_tls_wrap:588:7) at TCP.callbackTrampoline (node:internal/async_hooks:130:17)

astuyve · 2024-02-01T14:07:58Z

All from the AWS-SDK and all doing dynamo calls? Which version of the aws-sdk is everyone using?

viict · 2024-02-01T14:28:53Z

@astuyve here we have 2 different versions:

    "@aws-sdk/client-dynamodb": "^3.387.0",
    "@aws-sdk/lib-dynamodb": "^3.387.0",
    "@aws-sdk/smithy-client": "^3.374.0",

and

   "@aws-sdk/client-dynamodb": "=3.40.0",
    "@aws-sdk/lib-dynamodb": "=3.40.0",
    "@aws-sdk/smithy-client": "=3.40.0",

luxion-lennart · 2024-02-01T14:57:43Z

@astuyve We use version 3.362.0, that is provided by the lambda nodejs runtime.

carmargut · 2024-02-01T15:26:40Z

@astuyve Here you have:

"@aws-sdk/client-dynamodb": "3.474.0",
"@aws-sdk/util-dynamodb": "3.474.0",

astuyve · 2024-02-01T16:17:29Z

So far everyone is using the v3 sdk, has anyone reproduced this with v2?

viict · 2024-02-20T20:30:22Z

@astuyve can we do something for v3 meanwhile no one with v2 answers here? 🙏🏻

astuyve · 2024-02-20T22:07:34Z

Hi @viict - I'm not sure there's something specific we can do right now. I was hoping someone could replicate with AWS SDK v2 or demonstrate definitively that ddtrace is causing this issue.

Instead, it seems that ddtrace is recording that the tcp connection was closed by the server without a response. I noticed other users reporting the same issue. The aws-sdk author also closed this issue as something that can happen.

I could certainly be wrong here, but I'm still not sure what exactly we'd change in this project at this time.

Does anyone have a minimally reproducible example? Does removing dd-trace solve this definitively? Does this impact application code, or is it successful on retries?

Thanks!

viict · 2024-02-22T21:04:30Z

@astuyve oh I understand that of course. I'll see what I can do to improve and share here as well if I'm able to answer any of these questions.

Harmonickey · 2024-02-27T05:05:59Z

@Harmonickey do you know what lead to exception? For example, an outgoing request?

It was an outgoing request from the dd-trace library to DataDog sending an 'info' message.

Here is my initial configuration in case that helps.

const httpTransportOptions = {
	host: 'http-intake.logs.datadoghq.com',
	path: `/v1/input/${environment.datadog.apiKey}?ddsource=nodejs&service=${service}`
		+ `&env=${environment.name}&envType=${isWorkerEnv ? 'work' : 'web'}`,
	ssl: true,
};

const logger = createLogger({
	level: 'info',
	exitOnError: false,
	format: format.json(),
	transports: [
		new transports.Http(httpTransportOptions),
	],
});

Then during runtime calling logger.info('some string message') is when it threw the exception. the message is a static string and it does not always throw.

Because I haven't seen this error in a while, I suspect it was due to DataDog intake servers just being overloaded? So the connection wasn't responded to quickly enough and threw the socket hang up error. Perhaps DataDog has fixed it since then and improved their response times.

atif-saddique-deel · 2024-04-30T07:55:43Z

@tlhunter any updates here?
We are getting a lot socket hang up recently, we are using 4.34.0 version of dd-trace.

[HPM] ECONNRESET: Error: socket hang up
    at connResetException (node:internal/errors:720:14)
    at Socket.socketCloseListener (node:_http_client:474:25)
    at Socket.emit (node:events:529:35)
    at Socket.emit (node:domain:552:15)
    at Socket.emit (/usr/src/app/node_modules/@letsdeel/init/node_modules/dd-trace/packages/datadog-instrumentations/src/net.js:69:25)
    at TCP.<anonymous> (node:net:350:12)
    at TCP.callbackTrampoline (node:internal/async_hooks:128:17) {
  code: 'ECONNRESET'
}

antamb · 2024-04-30T15:09:23Z

We are also experiencing this issue using:
"dd-trace": "^5.6.0"

Error: socket hang up
    at Socket.socketOnEnd (node:_http_client:524:23)
    at Socket.emit (node:events:530:35)
    at Socket.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/net.js:69:25)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

atif-saddique-deel · 2024-05-06T08:02:10Z

we are having the same issue with latest version of dd-trace v4.36.0

dvictory · 2024-05-20T04:26:31Z

Did you switch from node18 to node20? in Node 19 they changed the keep alive default - https://nodejs.org/en/blog/announcements/v19-release-announce#https11-keepalive-by-default
Leading to a number of issues: Some outlined here: nodejs/node#47130

we see this around calls to AWS services, sns, sqs, etc. (all self heal with the SDK retry logic). What it unclear to me is if this is an from dd-trace error or is dd-trace just logging the issue from the aws call?

Error: socket hang up
    at TLSSocket.socketOnEnd (node:_http_client:524:23)
    at TLSSocket.emit (node:events:530:35)
    at TLSSocket.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/net.js:69:25)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Here is the info tab from this same raw error:

saibotsivad · 2024-05-23T17:31:14Z

@astuyve we are experiencing the same problem but not related to an AWS SDK issue, and I've been able to track it down to a timeout on an API call.

We are using axios for requests, so the package.json file has:

{
  "dependencies": {
    // ...
    "axios": "^1.6.7",
    // ...
    "datadog-lambda-js": "^7.96.0",
    "dd-trace": "^4.26.0",
    // ...
    "serverless": "^3.38.0",
    // ...
  },
  "devDependencies": {
    // ...
    "serverless-plugin-datadog": "^5.56.0",
    // ...
  },
  // ...
}

We deploy with serverless which has:

# ...
frameworkVersion: '3'
plugins:
  - serverless-plugin-datadog
provider:
  name: aws
  architecture: arm64
  runtime: nodejs16.x
custom:
  version: '1'
  datadog:
    addExtension: true
    apiKey: ${env:DD_API_KEY, ''}
    service: public-charging-api
    env: ${opt:stage}
    version: ${env:DD_VERSION, ''}
    enableDDTracing: true
# ...

We have some API call that uses axios in a pretty normal way, like this:

const response: AxiosResponse = await axios.request({
  method: 'GET',
  url,
  headers: { authorization },
  timeout: 20000,
});

(That's wrapped in a try/catch, so we know exactly what we are logging in any case.)

Functionally: we have a Lambda that makes ~50 HTTP requests in a very amount of time, and sometimes a dozen of them will take too long to resolve, so in that Lambda execution we are timing out those requests.

For every request that is aborted by axios due to timeout, we are getting this "Error: socket hang up" log.

The "third party frames" makes me suspect that it's the DataDog layer adding these.

astuyve · 2024-05-23T17:44:13Z

Thanks Tobias!! that's a great clue, @tlhunter any thoughts here?

Max101 added the bug Something isn't working label Oct 2, 2023

tlhunter added the serverless label Dec 18, 2023

tlhunter assigned astuyve Dec 19, 2023

tlhunter removed the serverless label Dec 21, 2023

alyanm mentioned this issue May 10, 2024

Connection error on auto-flush within serverless lambda dbader/node-datadog-metrics#116

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Error: socket hang up" in Lambda #3670

"Error: socket hang up" in Lambda #3670

Max101 commented Oct 2, 2023 •

edited

Loading

Harmonickey commented Nov 20, 2023 •

edited

Loading

tlhunter commented Dec 21, 2023

viict commented Jan 26, 2024

luxion-lennart commented Jan 26, 2024 •

edited

Loading

carmargut commented Jan 30, 2024

IgnacioFerreras commented Feb 1, 2024

astuyve commented Feb 1, 2024

viict commented Feb 1, 2024

luxion-lennart commented Feb 1, 2024

carmargut commented Feb 1, 2024

astuyve commented Feb 1, 2024

viict commented Feb 20, 2024

astuyve commented Feb 20, 2024

viict commented Feb 22, 2024

Harmonickey commented Feb 27, 2024 •

edited

Loading

atif-saddique-deel commented Apr 30, 2024

antamb commented Apr 30, 2024

atif-saddique-deel commented May 6, 2024

dvictory commented May 20, 2024

saibotsivad commented May 23, 2024

astuyve commented May 23, 2024

"Error: socket hang up" in Lambda #3670

"Error: socket hang up" in Lambda #3670

Comments

Max101 commented Oct 2, 2023 • edited Loading

Harmonickey commented Nov 20, 2023 • edited Loading

tlhunter commented Dec 21, 2023

viict commented Jan 26, 2024

luxion-lennart commented Jan 26, 2024 • edited Loading

carmargut commented Jan 30, 2024

IgnacioFerreras commented Feb 1, 2024

astuyve commented Feb 1, 2024

viict commented Feb 1, 2024

luxion-lennart commented Feb 1, 2024

carmargut commented Feb 1, 2024

astuyve commented Feb 1, 2024

viict commented Feb 20, 2024

astuyve commented Feb 20, 2024

viict commented Feb 22, 2024

Harmonickey commented Feb 27, 2024 • edited Loading

atif-saddique-deel commented Apr 30, 2024

antamb commented Apr 30, 2024

atif-saddique-deel commented May 6, 2024

dvictory commented May 20, 2024

saibotsivad commented May 23, 2024

astuyve commented May 23, 2024

Max101 commented Oct 2, 2023 •

edited

Loading

Harmonickey commented Nov 20, 2023 •

edited

Loading

luxion-lennart commented Jan 26, 2024 •

edited

Loading

Harmonickey commented Feb 27, 2024 •

edited

Loading