New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

break up collection sync into chunks #3

Merged

jctanner merged 1 commit into jctanner:main from drodowic:collection-sync

Apr 27, 2023

Contributor

drodowic commented Apr 11, 2023

disable InsecureRequestWarning to clean up the logs
change the sync_url to the repository ID
get chunk_size from envvar, or defaults to 100
add loop to sync the collections for every chunk
add loop to wait until the repo's last sync task state in not running or waiting
set the collections in the sync config's requirements_file based on the chunk_size
change the start sync status code check to 202


          break up collection sync into chunks

77af996

jctanner merged commit fa4e88c into jctanner:main

drodowic deleted the collection-sync branch

May 2, 2023 11:44

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment