Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support mirror of account with > 100 repos #21

Closed
Yikun opened this issue Mar 20, 2020 · 7 comments · Fixed by #48
Closed

Support mirror of account with > 100 repos #21

Yikun opened this issue Mar 20, 2020 · 7 comments · Fixed by #48
Labels
bug Something isn't working

Comments

@Yikun
Copy link
Owner

Yikun commented Mar 20, 2020

That means we should support pagination query in [1].

[1] https://github.com/Yikun/hub-mirror-action/blob/v0.05/entrypoint.sh#L57

@Yikun Yikun added the bug Something isn't working label Mar 20, 2020
@ShixiangWang
Copy link

我的个人仓库没有 100 个,但也支持同步了 30 个左右。

@ShixiangWang
Copy link

目前只支持最多30个啊

https://stackoverflow.com/questions/27331849/github-api-v3-doesnt-show-all-user-repositories

image

建议修改下支持 100 个仓库。

@Yikun
Copy link
Owner Author

Yikun commented Aug 7, 2020

function get_all_repos {
    USER=yikun
    GITHUB_REQUEST_RATE=2
    last_page=`curl -sI "https://api.github.com/users/$USER/repos?page=1&per_page=10" | sed -nr 's/^[lL]ink:.*page=([0-9]+)&per_page=10.*/\1/p'`
    sleep $GITHUB_REQUEST_RATE

    # does this result use pagination?
    if [ -z "$last_page" ]; then
        # no - this result has only one page
        curl -s -i "https://api.github.com/users/$USER/repos" | jq '.[] | .name' |  sed 's/"//g'
    else
        p=1
        while [ "$p" -le "$last_page" ]; do
            x=`curl -s "https://api.github.com/users/$USER/repos?page=$p&per_page=10" | jq '.[] | .name' |  sed 's/"//g'`
            echo $x
            p=$(($p + 1))
            sleep $GITHUB_REQUEST_RATE
        done
    fi
}

github有流控,很容易出现查着查着就超时的情况。

@Yikun
Copy link
Owner Author

Yikun commented Aug 7, 2020

{"message":"API rate limit exceeded for xxx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://developer.github.com/v3/#rate-limiting"}

@ShixiangWang
Copy link

增加 GItHub token 的支持?这样我记得可以一小时几千次。

@Yikun
Copy link
Owner Author

Yikun commented Aug 13, 2020

@ShixiangWang 嗯,可以复用那个token配置项,有空我试试

@Yikun
Copy link
Owner Author

Yikun commented Oct 13, 2020

因为需要获取src端的列表,因此需要src端的token,目前暂时没加这个配置项。

测了下其实基本也还好,github的接口是大概1小时60次,基本满足诉求了,即使分页每次100个repo,超过5000个repo才有可能超时。

当然,精益求精,后面准备加一个src_token作为可选配置。其实代码已经写了,但是代码比较乱,会影响pagination的阅读,所以后面再单独合入[1]

[1] 6cb3a46

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants