Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

optimize socrata paging #23

Closed
chelm opened this issue May 19, 2015 · 3 comments
Closed

optimize socrata paging #23

chelm opened this issue May 19, 2015 · 3 comments

Comments

@chelm
Copy link
Contributor

chelm commented May 19, 2015

We are seeing a massive slow down in socrata services that require paging of lots of data. One reason is that the current paging routing does not seem to be throttling its requests. It currently creates a list of pages to GET and immediately requests them. This is probably causing trouble on the server when we make 100 - 200 requests all at once for data.

The solution might be a scale it back a little bit and request pages a bit more slowly.
@dmfenton
Copy link
Contributor

Sold on that:
image

@chelm
Copy link
Contributor Author

chelm commented May 19, 2015

I have completed this work and will create a PR for it soon. The new version will be 0.2.0 and will support more throttled page requests via a queue and will return 202 while datasets are paging...

@dmfenton
Copy link
Contributor

Resolved by #24

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chelm @dmfenton