Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve speed of source uploads #88

Closed
Tracked by #54
bfhealy opened this issue Sep 2, 2022 · 1 comment
Closed
Tracked by #54

Improve speed of source uploads #88

bfhealy opened this issue Sep 2, 2022 · 1 comment
Labels
enhancement New feature or request

Comments

@bfhealy
Copy link
Collaborator

bfhealy commented Sep 2, 2022

Currently scope_upload_classification.py can upload ~1000 sources/classifications/light curves/annotations to Fritz per hour if two instances are run simultaneously on independent parts of source lists (using the -start -stop arguments). The server starts to complain if three or more instances are run together. See also #89.

Feature Summary
With the potential for millions of sources to be uploaded going forward, we should determine how much our code can be streamlined to make uploads complete faster.

Implementation details
Anything within our loop over each source is a good target for speed enhancements.

@bfhealy bfhealy added the enhancement New feature or request label Sep 2, 2022
@bfhealy
Copy link
Collaborator Author

bfhealy commented Oct 17, 2022

With the golden dataset uploaded to Fritz (see #57), I am closing this issue for now. It may be reopened once we are confronted with even larger sets of sources to upload to Fritz. Whitelisting the uploader's IP address may be a future enhancement of upload speed.

@bfhealy bfhealy closed this as completed Oct 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant