Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate syncstorage-rs stage timeouts #64

Closed
pjenvey opened this issue Jul 29, 2020 · 2 comments
Closed

Investigate syncstorage-rs stage timeouts #64

pjenvey opened this issue Jul 29, 2020 · 2 comments
Assignees

Comments

@pjenvey
Copy link
Member

pjenvey commented Jul 29, 2020

On stage we see intermittent upstream timeouts (nginx to syncstorage-rs), e.g.:

textPayload: "2020/07/27 14:20:33 [error] 9#9: *2372281 upstream timed out (110: Connection timed out) while reading response header from upstream, client: ..., server: , request: "GET /1.5/.../info/configuration HTTP/1.1", upstream: "http://127.0.0.1:8000/1.5/.../info/configuration", host: "stage.sync.nonprod.cloudops.mozgcp.net"

These sometimes happen even when there's no load testing, and even on the least resource intensive endpoints (e.g. /info/configuration which doesn't even check out a db connection).

@pjenvey pjenvey self-assigned this Jul 29, 2020
@erkolson
Copy link

We were seeing a max of 60s timeouts during stage loadtests, which alligns with a few timeout defaults. I changed one of them to a higher value:

proxy_read_timeout 625s

And a subsequent load test showed requests taking up to 300s:
https://earthangel-b40313e5.influxcloud.net/d/TebY7D1Zk/sync-rs-stage?viewPanel=34&orgId=1&from=1596725404492&to=1596729540325

@pjenvey
Copy link
Member Author

pjenvey commented Aug 17, 2020

Closing as dupe of #63

@pjenvey pjenvey closed this as completed Aug 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants