Error: Client network socket disconnected before secure TLS connection was established

[ennioVisco]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Note: a fascinating related issue is this one: aws/aws-sdk-js#3591, although in our case the same error is gained both when the code is run locally and via a lambda.

Describe the bug

Hello,

It's been now a couple of weeks of debugging trying to address this issue, and we don't seem to be able to find a way around it.

We have a public s3 bucket, that we fail to access, getting the the title error, but this is a recent behaviour, it used to work perfectly, and in fact we have an old lambda executing a similar code that still works.

What could possibly be the problem?

SDK version number

@aws-sdk/client-s3: 3.550.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.9.0

Reproduction Steps

We want to simply access an object from it via the JS api, the code is more or less the following:

import {  GetObjectCommand, S3Client } from "@aws-sdk/client-s3";

async function endpoint() {
 // ...
  const client = new S3Client({
    region: "eu-west-1"
  });
  const getObjectCommand = new GetObjectCommand({
        Bucket: "BUCKET_NAME",
        Key: "FILE_KEY",
      });
  const result = await client.send(getObjectCommand);
// ...
}

SInce the bucket is public, I believe this should be reproducible by anyone.

Of course if you try to access it directly via browser it works LINK REMOVED.

Observed Behavior

Running the previous code, returns the following error (with extra logging enabled)

endpoints Initial EndpointParams: {
  "ForcePathStyle": false,
  "UseArnRegion": false,
  "DisableMultiRegionAccessPoints": false,
  "Accelerate": false,
  "DisableS3ExpressSessionAuth": false,
  "UseGlobalEndpoint": false,
  "UseFIPS": false,
  "Region": "eu-west-1",
  "UseDualStack": false,
  "Bucket": "BUCKET_NAME",
  "Key": "FILE_KEY"
}
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: substring($Bucket, 0, 6, true) = tadata
endpoints assign: bucketSuffix := tadata
endpoints evaluateCondition: stringEquals($bucketSuffix, --x-s3) = false
endpoints evaluateCondition: not(isSet($Bucket)) = false
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: substring($Bucket, 49, 50, true) = null
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($ForcePathStyle, false) = true
endpoints evaluateCondition: aws.isVirtualHostableS3Bucket($Bucket, false) = true
endpoints evaluateCondition: aws.partition($Region) = {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "Europe (Ireland)"
}
endpoints assign: partitionResult := {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "Europe (Ireland)"
}
endpoints evaluateCondition: isValidHostLabel($Region, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = true
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, false) = true
endpoints Resolving endpoint from template: {
  "url": "https://{Bucket}.s3.{Region}.{partitionResult#dnsSuffix}",
  "properties": {
    "authSchemes": [
      {
        "disableDoubleEncoding": true,
        "name": "sigv4",
        "signingName": "s3",
        "signingRegion": "{Region}"
      }
    ]
  },
  "headers": {}
}
endpoints Resolved endpoint: {
  "headers": {},
  "properties": {
    "authSchemes": [
      {
        "disableDoubleEncoding": true,
        "name": "sigv4",
        "signingName": "s3",
        "signingRegion": "eu-west-1"
      }
    ]
  },
  "url": "https://BUCKET_NAME.s3.eu-west-1.amazonaws.com/"
}
@aws-sdk/credential-provider-node defaultProvider::fromEnv
@aws-sdk/credential-provider-env fromEnv
@aws-sdk/credential-provider-node defaultProvider::fromSSO
@aws-sdk/credential-provider-node defaultProvider::fromIni
@aws-sdk/credential-provider-ini fromIni
@aws-sdk/credential-provider-ini resolveStaticCredentials
{
  clientName: 'S3Client',
  commandName: 'GetObjectCommand',
  input: { Bucket: BUCKET_NAME', Key: 'FILE_KEY' },
  error: Error: Client network socket disconnected before secure TLS connection was established
      at connResetException (node:internal/errors:721:14)
      at TLSSocket.onConnectEnd (node:_tls_wrap:1712:19)
      at TLSSocket.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1408:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    code: 'ECONNRESET',
    path: null,
    host: 'BUCKET_NAME.s3.eu-west-1.amazonaws.com',
    port: 443,
    localAddress: undefined,
    name: 'TimeoutError',
    '$metadata': { attempts: 3, totalRetryDelay: 272 }
  },
  metadata: { attempts: 3, totalRetryDelay: 272 }
}

Expected Behavior

Retrieving the object.

It used to work some time ago, I have a clone of the same lambda of a few months back and it works.

Possible Solution

No response

Additional Information/Context

This is the current bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        }
    ]
}

[ennioVisco]

It turns out the issue was caused by an env variable: AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1' that resulted in wrong re-use of the TLS connection.

We adopted that env variable a long time ago, probably by reading this: https://cloudash.dev/blog/http-keep-alive-lambda

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.