Optimization on sync_versions to use ls-remote on Git VCS
#6930
Conversation
When a webhook is received we synchronize the branches/tags to create new versions under Read the Docs. For this process we do a full clone of the repository and iterate over the branches/tags. As this process of cloning the repository just for updating versions could be expensive, this commit adds the ability to use `ls-remote` on Git VCS just to query the repository for its branches/tags without performing a full clone.
| if tag.endswith('^{}'): | ||
| # skip annotated tags since they are duplicated | ||
| continue |
There was a problem hiding this comment.
This formatting is weird, but it is git's default.
$ git ls-remote https://github.com/readthedocs/test-builds | grep annotated
886108064d63dde44c14f657c994c63233e9c3ab refs/tags/annotated-tag
d39922ff32b8bc5c0182e7dd976fd921dba0d873 refs/tags/annotated-tag^{}
^ is a reserved character and branches/tags can't use it. See https://git-scm.com/docs/git-check-ref-format
I'd be
When SyncRepositoryTaskStep calls `sync_repo` we decide if we need a full clone of just listing the remote branches/tags to sync versions between Read the Docs and the repository.
|
@ericholscher I added a feature flag for this and the code is working as I expected. I created a new method This new method, will decide to follow the normal |
Seems worth testing. I'm worried it will likely break, since we've added another way of parsing git tags/branches, but this is definitely a much better solution.
|
We should probably have a set of tests that hits a real repository and confirms we parse them properly, or something. |
|
@ericholscher this test uses a real git repository https://github.com/readthedocs/readthedocs.org/pull/6930/files#diff-da4bc8cdc4f46ca0050b3c974ebf2ca0R47 (created under a tmp directory by |
|
@humitos annotated tags should be one of them |
|
@stsewd there is an annotated tag already https://github.com/readthedocs/readthedocs.org/pull/6930/files#diff-da4bc8cdc4f46ca0050b3c974ebf2ca0R65 |
|
Merging this so we can test it in the next release with some of our projects. |
When a webhook is received we synchronize the branches/tags to create new
versions under Read the Docs. For this process we do a full clone of the
repository and iterate over the branches/tags.
As this process of cloning the repository just for updating versions could be
expensive, this commit adds the ability to use
ls-remoteon Git VCS just toquery the repository for its branches/tags without performing a full clone.
Closes #6508