Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting Gateway Timeouts (504), Despite Setting timeout_seconds to 120 #2182

Open
JordanTreDaniel opened this issue Nov 3, 2020 · 2 comments

Comments

@JordanTreDaniel
Copy link

I have deployed a lambda that performs image manipulation, which can take a while sometimes. Longer than 30 seconds is a normal, expected time (for some photos)

At first, the timeout was showing in the zappa tail logs, but now, after setting the timeout_seconds option to 120, it seems as though the Amazon Cloudfront/Amazon Gateway is still enforcing a 30-second limit. I have dug around on AWS, and it seems I can only change this from a Cloudfront console. I don't believe there is an Cloudfront Distribution for this Lambda.

I am running on python 3.6

Expected Behavior

I would think that if I set the timeout_seconds to 120 in the Zappa settings, zappa would apply that setting to not only the lambda, but also whichever gates stand in front of it. (Gateway OR CloudFront Dist)

Actual Behavior

I get 504's when I don't choose to have the lambda downsample the image. (When it's normal for it to take long)

Possible Fix

Is there a way to apply the timeout_seconds to more than just the Lambda itself?

Steps to Reproduce

  1. Go to http://www.rapclouds.com
  2. Sign In (using oauth, super easy)
  3. Search a song
  4. Go to song
  5. Click blue settings button by word cloud
  6. Pick a "mask" (image to make the wordcloud from)
  7. Try diff values between 3 & 0 for the downSample option.
  8. 0 downsampling is almost a guaranteed timeout exception. (Network tab in devtools to verify)

Your Environment

  • Zappa version used: 0.51.0
  • Operating System and Python version: I use Mac, but this is only a problem on AWS. (python 3.6)
  • The output of pip freeze:
argcomplete==1.12.0
boto3==1.14.31
botocore==1.17.31
certifi==2020.6.20
cfn-flip==1.2.3
chardet==3.0.4
click==7.1.2
cycler==0.10.0
docutils==0.15.2
durationpy==0.5
Flask==1.1.2
future==0.18.2
hjson==3.0.1
idna==2.10
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.10.0
kappa==0.6.0
kiwisolver==1.2.0
MarkupSafe==1.1.1
matplotlib==3.3.0
numpy==1.19.1
Pillow==7.2.0
pip-tools==5.3.0
placebo==0.9.0
pyparsing==2.4.7
python-dateutil==2.6.1
python-slugify==4.0.1
PyYAML==5.3.1
requests==2.24.0
s3transfer==0.3.3
scipy==1.5.3
six==1.15.0
text-unidecode==1.3
toml==0.10.1
tqdm==4.48.0
troposphere==2.6.2
urllib3==1.25.10
Werkzeug==0.16.1
wordcloud==1.7.0
wsgi-request-logger==0.4.6
zappa==0.51.0
 {
	"dev": {
		"app_function": "app.app",
		"aws_region": "us-east-1",
		"profile_name": "default",
		"project_name": "wordcloudflask",
		"runtime": "python3.6",
		"s3_bucket": "word-cloud-bucket",
		"slim_handler": true,
		"cors": true,
		"binary_support": false,
		"timeout_seconds": 120
	}
}```

@ryancausey
Copy link

AWS API Gateway has a hard 30 second integration timeout that cannot be changed, and is most likely what you are running into. This is to say that API gateway will only wait for a maximum of 30 seconds for the lambda to respond, otherwise it will give up and respond with a 504 even if the lambda is still running due to you setting a higher timeout value in zappa.

https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

@JordanTreDaniel
Copy link
Author

Thanks for the response @ryancausey. I tried a lot of things like changing the limit, or having one lambda triggered by an API call trigger a SECOND lambda (which shouldn't have the 30 gateway constraint), and more stuff. But eventually, I just left AWS & went to Heroku since it is made simpler there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants