cdk deploy intermittently fails with ENOMEM

[bkchy]

Hi,

Recently, a number of people in our development team have been seeing cdk deploy fail with the following error:

[sdk error] CloudFormation.DescribeChangeSet({"StackName":"*******","ChangeSetName":"cdk-deploy-change-set"}) -> Error: connect ENOMEM *******:443 - Local (0.0.0.0:0)

If we enable debug logging for Node.js we see that the connection attempt to cloudformation.us-east-1.amazonaws.com is terminating almost immediately:

NET 31088: connect: attempting to connect to *******:443 (addressType: 4)
NET 31088: destroy
NET 31088: close
NET 31088: close handle

Interestingly, prior connections to CloudFormation to same address have succeeded in the same execution of the deploy command and the error does not always occur in the same spot. It seems every now and then polling the stack status during the deploy causes this error. We have tried a number things without any success:

• tried setting --concurrency 1
• changed between wired and wireless connections.
• changed between home and office network
• tried with or without VPN
• changed TCP keep-alive settings at Mac OS level
• disabled keep-alive entirely by editing the bundled index.js of CDK
• although we thought ENOMEM (no memory) error was a bit misleading we tried to change the Node.js heap settings like --max_old_space_size=... --max_semi_space_size=...
• tried different versions of Node.js 20,x and 22.x.
• upgraded CDK to latest (2.174.1).

Has anyone else run into an issue like this? We have been using AWS CDK on Node.js for almost 6 years now and this is the first time we are seeing this. All our developers are using Macbook with M1 or M2 chip.

Any suggestion for resolution or further diagnostics will be much appreciated.

[advdv]

This started happing to us as well, the error message looks like @bkchy metions, but also sometimes we get an "extended" error. This happens intermittently, we're on MacOS, Node v22.13, CDK is version: 2.175.1 (build afe6e87):

Error occurred while monitoring stack: Error: connect ENOMEM 3.78.206.123:443 - Local (0.0.0.0:0)
    at internalConnect (node:net:1115:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1520:9)
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:132:8) {
  errno: -12,
  code: 'ENOMEM',
  syscall: 'connect',
  address: '3.78.206.123',
  port: 443,
  '$metadata': [Object]
}

[bkchy]

We just discovered that disabling realtime internet scanning feature of our Sophos Endpoint software (endpoint security solution) seems to make this problem go away. We will monitor it for next couple of days to see if it's sticky but if you have any network filter/proxy enabled in your Mac (Settings -> Network -> VPN & Filters -> Filters & Proxies) then try disabling them to see if the situation improves.

[advdv]

Right, this could indeed be our AdGuard proxy messing things up. I'll see if things improve, thank you

[advdv]

Unfortunately, no. I spend about an hour going through the settings of AdGuard but i couldn't get it to work and are now simply living with it... If you have any more specific ideas please let me know.

[sudoplatform-engineering]

In our case, we had to add website exclusions in Sophos admin console for some of the AWS URLs that the cdk deploy was using.

[advdv]

@sudoplatform-engineering can you publish these urls here? It would help me (and others) out big time i think. Or link to the resource where they are described? thank you

[bkchy]

@advdv I think the list of URLs might be specific to the way we are deploying but you can try excluding the following URLs:

https://sts.<aws_region>.amazonaws.com
https:/<bucket_specific_to_our_deployment>.s3.<aws_region>.amazonaws.com
https://ssm.<aws_region>.amazonaws.com
https://cloudformation.<aws_region>.amazonaws.com

If you want to be sure you can run your deploy with export NODE_DEBUG=net,http,tls,http2 so you can see the addresses and ports that the deploy process is attempting to connect to.

[advdv]

Thank you @bkchy! that helped me a lot. For the others that are having specific issues with Adguard, i've added these domains to (on macos):

• click on the Adguard icon in the top menu
• advanced settings
• add the urls you need (with the regions you need ofc) to the option: "network.extension.exclude.domains"

this SEEMS to have helped, but they are intermittent issues so at this time i'm not sure yet.

[ronkoaveone]

Heads up, same here, and it was also our corp proxy software on our standard managed MacOS builds. It's producing ENOMEM while polling CloudFormation to check the status of stacks. That IP address is cloudformation.us-east-1.amazonaws.com.

Error occurred while monitoring stack: Error: connect ENOMEM 67.220.250.53:443 - Local (0.0.0.0:0)
    at __node_internal_captureLargerStackTrace (node:internal/errors:563:5)
    at __node_internal_exceptionWithHostPort (node:internal/errors:738:12)
    at internalConnect (node:net:1090:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1481:9)
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:130:8) {
  errno: -12,
  code: 'ENOMEM',
  syscall: 'connect',
  address: '67.220.250.53',
  port: 443,
  '$metadata': [Object]
}