Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Use streaming in repo-updater to make syncing process more responsive #5145
Currently the syncing process in
The drawback of this approach is that if the external service yields thousands and thousands of repositories it takes a long time for a repository to end up in the database and show up in the UI.
That leads to confusion with users thinking that the repo has not been added:
The idea is to change the architecture of repo-updater to stream new and modified repositories. That would make new repos show up immediately after adding a new external service.
Little update on the state of this ticket.
#5526 has been open for a few days now. It contains a feature-flagged streaming version of the repo-updater syncer.
It is feature-complete and all integration tests are passing,ut there are still some TODOs that should be tackled:
@tsenart and me decided that, even though my focus shifted from this ticket to A8N, I'll keep pursuing these TODOs in the coming weeks "on the side" so that we don't run into "PR rot" and get the PR merged as soon as possible.
The goal is to get #5526 merged ASAP with the feature disabled behind a flag. Then we can test the feature on k8s.sgdev.org and on local dev machines until we feel confident to enable it for customers, possibly in 3.9.