[ERROR] ConnectionTimeOut: Connect timeout on endpoint URL: while trying to upload .txt file from aws lambda to s3

[annu2612]

Please fill out the sections below to help us address your issue.

What issue did you see ?
I am working on aws lambda which creates a txt file and uploads the same in S3 folder. The timeout issue occurs at random times stating: [ERROR] ConnectionTimeOut: Connect timeout on endpoint URL: "https://sample-demo.s3.amazonaws.com/site1/output/status/trigger.txt" where trigger.txt is the file I am trying to upload to s3 and site1/output/status/ is the folder structure.

At times the file gets uploaded successfully but fails at random times making it difficult to test and debug.

P.S: I am trying to upload multiple files but random failures are happening.

Options Tried:

Upgraded version of boto3 == 1.14.57 and tried uploading as dependency

Used aws lambda's default boto3 version which is 1.14.48

s3 = boto3.client('s3', config=botocore.config.Config(connect_timeout=0.15, retries={'max_attempts': 5}))

s3 = boto3.client('s3', config=botocore.config.Config(connect_timeout=100, read_timeout=100, retries={'max_attempts': 4}, s3={'addressing_style':'path'}))

s3 = boto3.client('s3', region_name="us-east-1", config=botocore.config.Config(s3={'addressing_style':'path'}))

None of the above solutions worked.

Is there any option which I am missing, or is there any rate-limiting set where we cant upload more than n times.
Any suggestions would be really helpful.

[annu2612]

Kindly provide some suggestions

[swetashre]

@annu2612 - Thank you for your post. What is the connect_timeout value when you are getting the error ? Is it happening for each value ?
Can you provide me debug logs ? You can enable log by adding boto3.set_stream_logger('') to your code.

[annu2612]

What is the connect_timeout value when you are getting the error? Not Sure on how to check this, I am working on AWS lambda
Is it happening for each value? No not for every file.. out of 3 files sometimes for 1 or 2 rarely for all the 3 files
Can you provide me debug logs ?
[DEBUG] 2020-10-05T06:26:43.199Z c104d6a9-eeb2-4fd5-b109-401044b4c209 Exception raised.
Traceback (most recent call last):
File "/var/task/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/var/task/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/var/task/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/runtime/botocore/httpsession.py", line 254, in send
urllib_response = conn.urlopen(
File "/var/task/urllib3/connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "/var/task/urllib3/util/retry.py", line 379, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/var/task/urllib3/packages/six.py", line 735, in reraise
raise value
File "/var/task/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/var/task/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/var/task/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/var/task/urllib3/connection.py", line 309, in connect
conn = self._new_conn()
File "/var/task/urllib3/connection.py", line 164, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7f2fd3e74b20>, 'Connection to ect-qa-demo.s3.amazonaws.com timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/runtime/s3transfer/tasks.py", line 126, in call
return self._execute_main(kwargs)
File "/var/runtime/s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "/var/runtime/s3transfer/upload.py", line 692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/var/runtime/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 621, in _make_api_call
http, parsed_response = self._make_request(
File "/var/runtime/botocore/client.py", line 641, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/var/runtime/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/var/runtime/botocore/endpoint.py", line 156, in _send_request
raise exception
File "/var/runtime/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/var/runtime/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/var/runtime/botocore/httpsession.py", line 287, in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://ect-qa-demo.s3.amazonaws.com/site1/output/status/brand.json"

Connection details:
config = Config(connect_timeout=1, retries = {
'max_attempts': 10,
'mode': 'standard'
})
s3 = boto3.client('s3',region_name='us-east-1', config=config)

[swetashre]

connect_timeout is the time in seconds till a timeout exception is thrown when attempting to make a connection. The default is 60 seconds. Can you try by increasing connect_timeout to a larger value?
Can you provide me your sample code ?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[Hydrino]

Facing the same issue, trying to create a CloudWatch Log export task through Lambda.
Intermittently, the logs client connection times out.

Code:

import boto3
import os
from datetime import datetime, timezone, time, timedelta
import time as pytime
import logging
from botocore.session import Session
from botocore.config import Config
import botocore.exceptions

# Initialize Logger
logger = logging.getLogger()
logger.setLevel(logging.INFO)

s = Session()
client = s.create_client('logs', config=Config(connect_timeout=60, retries={'max_attempts': 2}))

LOG_GROUP_NAME = os.environ["logGroupName"]
DESTINATION = os.environ["s3BucketName"]
PREFIX = os.environ["s3Prefix"] + "/" + datetime.now(timezone.utc).strftime("%Y-%m-%d")

def lambda_handler(event, context):

    # 
    # ..some code ...
    # 
    
    try:
        create_task_response = client.create_export_task(
            logGroupName = LOG_GROUP_NAME,
            fromTime = start_time,
            to = end_time,
            destination = DESTINATION,
            destinationPrefix = PREFIX)
        
    except client.exceptions.LimitExceededException as e:
        logger.error(f"Can only have 1 running export task. Please check if your task is already triggered, or try again. Exception: {e}")
    except Exception as e:
        logger.error(f'Unknown exception occured\n{e}')
        raise 
    else: 
        taskId = create_task_response.get('taskId')
        logger.info(f'Successfully created task with id: {task}')
        return taskId
        

Boto retries the connection 2 more times as specified in the config and finally returns ConnectTimeOutError Exception

ConnectTimeoutError: Connect timeout on endpoint URL: "https://logs.eu-west-1.amazonaws.com/"
Traceback (most recent call last):
  File "/var/task/index.py", line 35, in lambda_handler
    create_task_response = client.create_export_task(
  File "/var/runtime/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/var/runtime/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/var/runtime/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/var/runtime/botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/var/runtime/botocore/endpoint.py", line 253, in _needs_retry
    responses = self._event_emitter.emit(
  File "/var/runtime/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/var/runtime/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/var/runtime/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/var/runtime/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/var/runtime/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/var/runtime/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/var/runtime/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/var/runtime/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/var/runtime/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/var/runtime/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/var/runtime/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/var/runtime/botocore/httpsession.py", line 287, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)

This issue is intermittent thus difficult to debug. Please help.

[annu2612]

In my case issue got resolved by configuring VPC endpoints, VPC was configured as part of lambda.

[AymanAzzam]

In my case issue got resolved by configuring VPC end

Can you provide more info ? steps ?

[jason-code78]

Jay has sent you an email via Gmail confidential mode: [image: Gmail logo]Re: [boto/boto3] [ERROR] ConnectionTimeOut: Connect timeout on endpoint URL: while trying to upload .txt file from aws lambda to s3 (#2611) <https://confidential-mail.google.com/msg/AMptQGClz_ThVwAHYpiE7DniboonbvqruoEXwd0T6Am3DWmQCpfyd2ukh4whvdOefBFaZwi-Tgqos8If39BS44ZtfqGrR69ZBHeY1im4UiGm-oKEMcWgqy-dGjgJ5hn2UXEdk7e13GfTY3S3n4fi-depxGlOMGYIRW4kM7WtssJ8Y-cAcJG1UZ8Cj0rD0ePH__NEUcIlRJAS_vglkRpYtrukXw==> This message was sent on Aug 16, 2021 at 9:07:51 AM PDT You can open it by clicking the link below. This link will only work for ***@***.*** View the email <https://confidential-mail.google.com/msg/AMptQGClz_ThVwAHYpiE7DniboonbvqruoEXwd0T6Am3DWmQCpfyd2ukh4whvdOefBFaZwi-Tgqos8If39BS44ZtfqGrR69ZBHeY1im4UiGm-oKEMcWgqy-dGjgJ5hn2UXEdk7e13GfTY3S3n4fi-depxGlOMGYIRW4kM7WtssJ8Y-cAcJG1UZ8Cj0rD0ePH__NEUcIlRJAS_vglkRpYtrukXw==> Gmail confidential mode gives you more control over the messages you send. The sender may have chosen to set an expiration time, disable printing or forwarding, or track access to this message. Learn more <https://support.google.com/mail/answer/7674059> Gmail: Email by Google Use is subject to the Google Privacy Policy <https://myaccount.google.com/privacypolicy?hl=en> Google LLC, 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA You have received this message because someone sent you an email via Gmail confidential mode. [image: Google logo]

[jason-code78]

Jay has sent you an email via Gmail confidential mode: [image: Gmail logo]Re: [boto/boto3] [ERROR] ConnectionTimeOut: Connect timeout on endpoint URL: while trying to upload .txt file from aws lambda to s3 (#2611) <https://confidential-mail.google.com/msg/AMptQGBrSXBGu8RJaA3OYGVxtetf0tyh8-53tGlzsZENnIca8U4Hm8DSburs9rrkBb71vOYyD5h4cqXEb_TE_scygZ72tZiZam2oNCZpBkRfJpfkPKQDWFc4yk9E37J426CXdhnzSFVz8VjCxAK5nW5dkllIwlttGOSfOMkELcUNDn7TlCvAYxTldoo6dYKWEchH7cTe9WLGI3TiMsCsv5m3aw==> This message was sent on Aug 16, 2021 at 9:10:47 AM PDT You can open it by clicking the link below. This link will only work for ***@***.*** View the email <https://confidential-mail.google.com/msg/AMptQGBrSXBGu8RJaA3OYGVxtetf0tyh8-53tGlzsZENnIca8U4Hm8DSburs9rrkBb71vOYyD5h4cqXEb_TE_scygZ72tZiZam2oNCZpBkRfJpfkPKQDWFc4yk9E37J426CXdhnzSFVz8VjCxAK5nW5dkllIwlttGOSfOMkELcUNDn7TlCvAYxTldoo6dYKWEchH7cTe9WLGI3TiMsCsv5m3aw==> Gmail confidential mode gives you more control over the messages you send. The sender may have chosen to set an expiration time, disable printing or forwarding, or track access to this message. Learn more <https://support.google.com/mail/answer/7674059> Gmail: Email by Google Use is subject to the Google Privacy Policy <https://myaccount.google.com/privacypolicy?hl=en> Google LLC, 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA You have received this message because someone sent you an email via Gmail confidential mode. [image: Google logo]

[AymanAzzam]

@jason-code78 !!

[jdgualtero]

Hello @jason-code78 can you share the info with me, I have a similar problem

[jinft-kr]

Hello @jason-code78 can you share the info with me, I have a similar problem, too.

[jcarlislel]

Hi @jason-code78 can you share the info with me as well? Thank you in advance!

[jaeseongpark91]

Hi @jason-code78 can you please share the info with me as well? Thank you :)