Custodian job errors/failures potentially because of exhaustion of ephemeral ports on the instance #9031
Labels
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
We started investigating a "could not connect to metadata endpoint url error"
With a lot of back and forth with AWS, this is what we found:
Confirmation that urllib3, which is what boto3 uses to establish connections, does not have idle timeouts [1]. The expectation is that, if a connection is not going to be used further it will be closed. Apparently, the client here [2] is not created as an object. Close calls [3] would be what is needed to close connections once they are opened. Without that the connection will not be closed until it is idle for 2 hours [4]. Client call at line 712 of the aws.py section of Cloud Custodian must be called multiple times in order for the error to keep appearing in the logs after good calls which followed bad calls as the credentials that are used are stored in the client for at least 10 minutes between refreshes [5].
It was found that many sockets were in the CLOSE-WAIT state.
As per boto3 service team - Clients are generally thread safe as noted here and in my tests with Batch running with the client created in the threads with 100 threads resulted in around 400 to 500 sockets being used while, when I created the client outside of the threads and used it inside the threading it resulted in around 5 to 10 additional sockets at the same 100 threads. Additionally, it resulted in only one or two connections to the ECS Agent address rather than hundreds of them. If the client were created somewhere else outside of the threading then there would be one client which would only need to make one call to the ECS Agent and would also be able to manage the client connections more efficiently rather than having new clients for every thread which means that connections are essentially being managed by the code generating a new client for every thread.
I was suspecting the issue to be in S3Transfer library, but they confirmed that the S3Transfer package in boto3 [6] and it has not been updated in 9 months
[1] urllib3/urllib3#2498
[2] https://github.com/cloud-custodian/cloud-custodian/blob/main/c7n/resources/aws.py#L712
[3] https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/close.html
[4] https://man7.org/linux/man-pages/man7/tcp.7.html
[5] https://github.com/boto/botocore/blob/master/botocore/credentials.py#L61
[6] https://github.com/boto/boto3/blob/master/boto3/s3/transfer.py
As per AWS support team -
Additionally, I did replicate the error by reducing the net.ipv4.ip_local_port_range from the default “32768 61000” to "52768 61000", a reduction from around 29K to around 9K. The port range could be increased up to "1024 65535". In a run with 20 containers there were 1492 connections from process called “custodian” and of those 781 were connections to 169.254.170.2. This is expected behavior if there are hundreds of clients being created by hundreds of threads as each one will establish a connection to the ECS Agent in order to validate credentials. Moving the client creation out of the threads should reduce the number of sockets being used by close to half just from this if not more as it would allow the underlying http client to manage the connection pool and reuse connection rather than each client have to generate its own connection to the Service is it connecting to.
As part of investigating this we had to switch from AWS Batch managed ECS cluster to unmanaged AWS Batch ECS cluster. I suspect with multiple containers getting placed on the instance and sockets not getting garbage collected we run into a scenario where we are left with not enough ports on the instances. I will admit that we were running with the default port range configuration of “32768 61000”. We're planning on bumping up the port range to "1024 65535" as suggested in our lower environment this week to see if that helps reduce the errors
Extra information or context
The text was updated successfully, but these errors were encountered: