-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sync times out with 1000 sources #1007
Comments
Here's a tip on how to easily update your staging server with 1000 sources (for those who are working on reproducing this issue)
|
@kushaldas or @redshiftzero - here's an update to fill you both in... Notes
The fix
What else would help?
Test resultsI ran the client three times, each with a clean local database. n=100 sourcesThe sync did not time out on any of the three runs of the client with a clean local db. n=200 sourcesThe sync timed out one out of three times I ran the client with a clean local db. n=300 sourcesThe sync timed out all three times I ran the client. I left the client running for a few minutes after the third run and the sync recovered with the following results measured in
All subsequent syncs (after the local database had all 300 sources and the source widgets had all been added to the source list):
n=400+ sourcesAs n gets higher the syncs time out more and more and are less likely to recover the way one sync did in the |
We are actually not passing that timeout to |
In my tests with 500 sources + 2 submission each source, 60 seconds time worked well everytime, tried 5 times. |
It's important to remember (which I wasn't Friday 🙂) that the proxy's timeout works differently than the SDK's. As @kushaldas pointed out, the SDK timeout is a hard limit on the amount of time we'll wait for a response from the What we're seeing is that slow API endpoints can stay quiet for longer than ten seconds. Right now with up to 1000 sources a 60-second proxy timeout works. I think we can safely specify an even longer proxy timeout, say 120 or 300 seconds, and not implement the ability for the SDK to change it, because the SDK timeout will allow us to stay responsive to the workstation operator. The only downside I can think of to this is possibly running into problems spawning too many |
I'll create a separate issue for each of these things. For now, we have increased the proxy timeout to 120 seconds because it's a quick and easy change. |
The 120 seconds fix worked my test with 300 sources this morning, and for @redshiftzero's 505-source test this morning. Just want to mention that we still timeout on 1000 sources so I'm leaving this issue open. Also another reason we should prioritize freedomofpress/securedrop-proxy#145, freedomofpress/securedrop-sdk#114, and #1754 |
related server-side change in freedomofpress/securedrop#5184 |
Description
Sync times out when there are 1000 sources.
STR
Expected
You should be able to use the client and see new replies and messages succeed.
Actual
The sync times out on a thousand sources, even after a client restart, the sync times out on a 1000 sources.
The text was updated successfully, but these errors were encountered: