Client loses Lambda connection after 300 secs (per OP from #2325)

[cajund]

Describe the bug
This is in reference to #2325 - that issue was closed without a resolution.

Evidence here:

From the scripts log. Note the client is set to time out at 15 minutes, and it does so as instructed.

2020-07-11 19:25:33.237197	DEBUG: Start Lambda: 07/11/2020, 19:25:33
2020-07-11 19:40:33.431392	ERROR: Error invoking lambda
2020-07-11 19:40:33.431446	ERROR: Read timeout on endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/*********/invocations"
2020-07-11 19:40:33.431480	DEBUG: Finish Lambda: 07/11/2020, 19:40:33

From the Cloudwatch logs - note the Lambda finishes in just about 7 minutes:

2020-07-11T15:25:35.093-04:00  START RequestId: 6204a7f0-1673-432f-9098-d1b4b5c2de8e Version: $LATEST
2020-07-11T15:32:32.042-04:00  END RequestId: 6204a7f0-1673-432f-9098-d1b4b5c2de8e
2020-07-11T15:32:32.042-04:00  REPORT RequestId: 6204a7f0-1673-432f-9098-d1b4b5c2de8e	Duration: 416944.79 ms	Billed Duration: 417000 ms	Memory Size: 128 MB	Max Memory Used: 92 MB	Init Duration: 962.30 ms	

I cannot verify the original statement about a 300 second issue, but this worked fine for a long time until the data set on which this was operating grew large enough to extend the Lambda execution time to longer than 5 minutes. So there is definitely something happening with the client's connection to Lambda independent of the timeout set for the client.

BTW, this is happening with v.1.10.45 but I will be testing with the latest version. Expecting the same result given there are no other discussions regarding this issue.

Thanks for you help

[swetashre]

Thank for your post. This issue is likely due to nat configuration. This issue happens when your network has 5 min session timeout. You will need to implement keep alive. Details on doing that will vary depending on 'local environment'. If that is a laptop/desktop, then find out the OS and follow specific guidelines for implementing keep alive in that operating system.

These are the some resources that might help:
https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html#nat-gateway-troubleshooting-timeout
https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html
https://stackoverflow.com/questions/15860127/how-to-configure-tcp-keepalive-under-mac-os-x/23900051

[Slooz]

@swetashre, would implementing keep alive via boto3's configuration work?

tcp_keepalive
Toggles the TCP Keep-Alive socket option used when creating connections. By default this value is false; TCP Keepalive will not be used when creating connections. To enable TCP Keepalive with the system default configurations, set this value to true.

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-a-configuration-file

[firhatsungkar]

@swetashre, would implementing keep alive via boto3's configuration work?

tcp_keepalive
Toggles the TCP Keep-Alive socket option used when creating connections. By default this value is false; TCP Keepalive will not be used when creating connections. To enable TCP Keepalive with the system default configurations, set this value to true.

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-a-configuration-file

This work for me. Thanks a lot.