GitDumper should also checkout the main branches

[erikyao]

When adding a new plugin to pending.biothings.io, the URL to the GitHub repo will be passed to http://localhost:19080/dataplugin/register_url,

which in the end calls the AssistantManager.register_url() method (assistant.py#L699).

The AssistantManager instance looks like to add a message (including the URL to register) to its corresponding MongoDB collection, and finally a GitDumper instance will receive the URL and then check the repo out. By default, GitDumper will only check master branches, but recently GitHub has changed its default branch name from master to main. Therefore our GitDumper cannot checkout the latest GitHub repo-based plugins.

The root cause in the code seems to be dumper.py#L1072:

DEFAULT_BRANCH = "master"

[zcqian]

I think the better solution is to figure out what the remote HEAD is and use that as the default branch.

But please do note that while HEAD usually points to master or main (in GitHub's case), having a default branch is not required (per git-scm documentation).

Use git remote show origin (origin or whatever the name of the remote is) to find the "HEAD branch"

[newgene]

Per this post, can use this to get the default git branch.

git ls-remote --symref "$REPO_URL" HEAD | sed -nE 's|^ref: refs/heads/(\S+)\s+HEAD|\1|p'

[zcqian]

The reason we don’t want to introduce sed is that there’s no guarantee that it exists, and piping probably doesn’t work as expected when running in a subprocess.

[newgene]

@zcqian certainly, within python code, you don't really need sed, should be easy to extract what you need with regex, same as what sed part intend to do in the above example.

[zcqian]

That’s what I’m thinking for now. We will invoke git and scrape its output for information, and in this case we will take the default branch in this order:

1. User defined default. The current class variable will default to None. Hoped this doesn’t break other people’s code.
2. Remote HEAD
3. master branch of the remote, if it exists
4. main if master does not exist but main exists

raise if none of the above applies

Edit: changed how things work, now only taking into account the configured repo URL, not what's actually configured in the local git repository.

In the long run I don’t like scraping the standard output of git, or using GitPython (we use this in some places, it is also probably scraping the output, given that it depends on the git command). We should look into solutions like libgit2.

[zcqian]

Fixed via #150