download: cURL "collected" files in series #485
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As per discussion in #484 this PR changes the "collected" OA downloads (ie.
openaddr-collected-global.zipandopenaddr-collected-global-sa.zip) to run in series rather than in parallel.The reason for this change is that the OA CDN has a "Maximum Connections Per IP" limit of 1.
Prior to this PR, cURL would intermittently receive an HTML file containing the text
503 Service Unavailable, whenunzipattempted to open this file it would error the cryptic messageEnd-of-central-directory signature not found.The positive effect of this PR is that the downloads will no longer only succeed intermittently, the negative effect is that downloads will be slower since the second file isn't started until the first has complete.
I noticed that the "filtered download" (ie. where the user selects only a subset of the OA database) code is already using async.series().
Hopefully in the future we can rework this a bit and return to parallel downloads, the financial costs of hosting these downloads at scale can be significant, and abuse is widespread, so I understand the need for the IP limits.
resolves #484