Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

My Github account will not sync. Possibly too many repositories. #837

Closed
schwern opened this issue Jan 6, 2013 · 25 comments
Closed

My Github account will not sync. Possibly too many repositories. #837

schwern opened this issue Jan 6, 2013 · 25 comments

Comments

@schwern
Copy link

schwern commented Jan 6, 2013

I'm sorry if this is not the appropriate place to report a user problem, but I haven't had luck getting a response via the mailing list nor twitter. However, I suspect my case is special.

My profile hangs with "Please wait while we sync from GitHub". I've tried various things to get it unstuck including logging out of Github and back in again and reauthorizing TravisCI on Github.

I suspect the problem is I'm a member of the Gitpan organization which has 21,000 repositories and this is choking the syncing process. I know it gives parts of the Github API indigestion.

@svenfuchs
Copy link
Contributor

@schwern thanks for reporting!

@rkh I've tried to sync this manually in the console, but the process timed out after an hour or so (had it running in the background, didn't time it)

Looking at the sync code it seems that we first read all the orgs and then fetch all repos for both the user and all of his orgs. These seem to be 1 + n (orgs) requests, but since each of them is paginated in GH it's more like ~800 requests in this case.

We shouldn't hit our rate limit with a number like that (and nothing seems to indicate we do).

I'm not sure how to best approach this issue, any ideas? Wanna look into this?

@rkh
Copy link
Contributor

rkh commented Jan 6, 2013

Will investigate.

@roidrage
Copy link
Contributor

roidrage commented Jan 6, 2013

We've had several cases like this before, and it is indeed the case that the GitHub API never returns when fetching the repositories for an organization like that. The call will eventually time out on our end unfortunately.

@schwern
Copy link
Author

schwern commented Jan 6, 2013

On 1/6/13 10:59 AM, Mathias Meyer wrote:

We've had several cases like this before, and it is indeed the case that
the GitHub API never returns when fetching the repositories for an
organization like that. The call will eventually time out on our end
unfortunately.

If you let me know what API calls are timing out, I can handle
contacting Github support and see about getting them fixed.

@roidrage
Copy link
Contributor

roidrage commented Jan 6, 2013

Do you require building repositories from that organization? We might just start blacklisting large organizations like this one and skip them during sync.

On Sunday, 6. January 2013 at 22:00, Michael G. Schwern wrote:

On 1/6/13 10:59 AM, Mathias Meyer wrote:

We've had several cases like this before, and it is indeed the case that
the GitHub API never returns when fetching the repositories for an
organization like that. The call will eventually time out on our end
unfortunately.

If you let me know what API calls are timing out, I can handle
contacting Github support and see about getting them fixed.


Reply to this email directly or view it on GitHub (#837 (comment)).

@schwern
Copy link
Author

schwern commented Jan 6, 2013

On 1/6/13 1:13 PM, Mathias Meyer wrote:

Do you require building repositories from that organization? We might
just start blacklisting large organizations like this one and skip them
during sync.

Nope, Gitpan doesn't require travis at all. Gitpan is an automated
historical archive of Perl module releases taken from CPAN.

That sounds sensible. Maybe leave the user a message so they know their
org was skipped. Bonus points for a "try anyway" button. Sounds like a
good first order fix.

Maybe in the future the sync process can work like this.

  1. Sync the user's personal repositories.
  2. List all the user's organizations.
  3. Allow the user to tick off which ones they want synced.

I can imagine a few reasons people might not want their orgs to show up
on Travis.

  • The org is too large.
  • They're not very active with the org, so it just clutters travis.
  • Somebody else handles the build testing, so travis is redundant.
  • The org is private / business.

I have all of them. I have yet to use Travis, so I don't know how much
an organization affects the interface.

@chunyan
Copy link

chunyan commented Jan 30, 2013

I have the same problem with my personal repository. I have only one personal repository(https://github.com/chunyan/scrooge), and it's forked from my organization(https://github.com/twitter/scrooge).

I also noticed that in my Account Settings -> Applications, it shows that Travis has read-only access, i.e.:
Travis access public repositories and access user emails (readonly)
To my understanding Travis needs write access to set up service hooks and sync repositories, right? But I don't know where to authorize write access.

Any other possible causes?

@tylermenezes
Copy link

I'm having the same problem, unfortunately, however I'm only a member of 35 repositories.

@jmtuley
Copy link

jmtuley commented Jan 30, 2013

I also experience this. I do belong to two organizations, one of which has around 100 (mostly private) repositories.

@sarahhodne
Copy link
Contributor

Syncing is currently disabled due to RubyGems being compromised. Check the @travisci Twitter account for more info.

@zupo
Copy link

zupo commented Mar 22, 2013

I have the same problem, cannot sync via the web interface on the Pro account, OS X Lion, latest Chrome & FF. Syncing through travis CLI works fine.

@rkh
Copy link
Contributor

rkh commented Mar 22, 2013

I have the same problem, cannot sync via the web interface on the Pro account, OS X Lion, latest Chrome & FF. Syncing through travis CLI works fine.

That is strange, both the web UI and the CLI hit the same API endpoint.

@drogus?

@zupo
Copy link

zupo commented Mar 22, 2013

Hard reset of the browser, followed by purging all caches seems to help: I can sync now in FF.

@schwern
Copy link
Author

schwern commented Mar 22, 2013

Tried clearing all cookies and cache in Firefox, no luck. Also tried "Reset Safari" in Safari, no luck.

@viraptor
Copy link

Just a "me too" for user "viraptor". Although I've got 10 small own repos and 14 in another org, the "Please wait while we sync from GitHub" message is there for the last hour or so.

(and it went away on its own after logout/login)

@nschonni
Copy link
Contributor

Maybe a little progress indicator would help:

Syncing with GitHub:
Found 3 organizations.
Syncing 84 repos from Organization X.
...

@drogus
Copy link
Contributor

drogus commented Mar 24, 2013

I think that discussion here is a mix of a few issues, which is confusing, because workarounds for some of the issues (like clearing the cache etc.) will not help with the original problem: organisation with 21k repos.

@schwern I'm sorry that we made you wait so long for a solution. I created a PR for this: travis-ci/travis-core#199, I will probably deploy it tomorrow, which should allow to sync your repos

@schwern
Copy link
Author

schwern commented Mar 24, 2013

@drogus Yay! LMK when you deploy it and I'll try it out.

@drogus
Copy link
Contributor

drogus commented Mar 25, 2013

@schwern I've deployed and synced your account manually, please let me know if everything works correctly

@schwern
Copy link
Author

schwern commented Mar 25, 2013

Looks like it, thanks! I have my private repositories and my two
non-hugemungous organizations. The "Sync now" button also worked.

I haven't set up any service hooks yet, if you don't hear from me assume
everything's going fine.

@drogus
Copy link
Contributor

drogus commented Mar 25, 2013

@schwern great! And sorry again for making you wait so long.

I'm closing this issue, please open new one in case of other problems.

@drogus drogus closed this as completed Mar 25, 2013
@pocesar
Copy link

pocesar commented Mar 29, 2013

I'm having this problem as well, stuck in Please wait while we sync from GitHub. Using Firefox 19.0.2. have not tried in other browser though.

@drogus
Copy link
Contributor

drogus commented Mar 29, 2013

@pocesar could you try to log out and then log in again? Your account is sync, but UI is wrong, I will push a proper fix for this later today

@pocesar
Copy link

pocesar commented Mar 29, 2013

@drogus it worked, logged out and in again, thanks :)

@kud1ing
Copy link

kud1ing commented Oct 16, 2013

Hi,

i also get "You do not seem to have any repositories that we could sync".

There is a twist though: i want to enable the repository linuxfood/rustsqlite which i maintain and that shows up under "Your repositories". Is this supported?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests