Migrate to Github

[ghost]

Originally reported by: Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco)

---

In order to be more visible and friendly to more contributors, CherryPy should consider moving to Github.

Github is the canonical and preferred hosting for most open source projects.

I'm willing to help with the process, but I first want to get the nod from @cyraxjoe and @Lawouach. Also, we'll need to create a CherryPy org (if there's not a better one to host it).

I propose migrating the existing issues to Github using https://github.com/jeffwidman/bitbucket-issue-migration or one of its many forks. I'm happy to test and execute that.

---

• Bitbucket: https://bitbucket.org/cherrypy/cherrypy/issue/1410

[ghost]

Original comment by Sylvain Hellegouarch (Bitbucket: Lawouach, GitHub: Lawouach):

---

Hey Jason,

I'm all for it! I think there had been other people stating they were for it too.

As long as we manage to keep our history and we don't kill the bitbucket repo (keep it as a read-only to github for a long while so people can still refer to it).

Thanks for the proposal and offering to undertake the task.

[ghost]

Original comment by Joel Rivera (Bitbucket: cyraxjoe, GitHub: cyraxjoe):

---

Hi Jason,

I totally support this initiative and I'm willing to help on any task related to get it done 😃

Thanks!

[ghost]

Original comment by Joel Rivera (Bitbucket: cyraxjoe, GitHub: cyraxjoe):

---

@bhch we currently maintain a github mirror: https://github.com/cherrypy/cherrypy

Would you mind on expressing the reason behind preferring Mercurial over Git for a project that you don't actively contribute to? Just curios, do you maintain some sort of fork?

Personally I like Mercurial. But pragmatically GitHub is where the conversation is happening most of the time and if the move increase the engagement into the project (which I believe it will) then the change is justified.

Also, we don't really benefit for any peculiarity from mercurial. So the preferences is mostly a matter of personal taste or philosophy but from the perspective of the project I would always prefer the alternative that add less friction for contribution. Lot of people already knows git and it has become the defacto for main stream open source.

[ghost]

Original comment by Konstantin Molchanov (Bitbucket: moigagoo, GitHub: moigagoo):

---

Just a sidenote: after moving to GitHub, you could add a Gitter chat to the repo that would replace IRC. Gitter provides a convenient way to engage in a discussion with the developers and community members. It offers markdown, history, and email notifications, which really makes IRC pale in comparison.

Although I like hg more, I totally support the move to GitHub. I agree with Joel—personal tastes should not affect project-wide decisions.

[ghost]

Original comment by Sylvain Hellegouarch (Bitbucket: Lawouach, GitHub: Lawouach):

---

I think @nicklasb is interested too.

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

Yeah, this sounds great!

I agree on basically everything said here:
+1 moving to github. For example I just had to rewrite this response because the page refreshed or something.

+1 WRT mercurial, an open source project should aim to lower the bar for participation, and it is not like GIT is evil, closed source and proprietary

+1 using gitter, it is really usable. Actually very usable.

+1 keeping the bitbucket repo read-only.

+1 Creating an organization at github. However, isn't /cherrypy already an organisation?

WRT to moving issues, yeah if that works, great. It would be good if the issues on bitbucket could get a not that they have been moved at the end.

[ghost]

Original comment by Sylvain Hellegouarch (Bitbucket: Lawouach, GitHub: Lawouach):

---

I'm all for it too.

My only concern about gitter is that it requires a github account, not everyone may want that. Are there bots that could bridge to IRC?

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

Well, it is sort of that we now instead expect people to have google, IRC and bitbucket accounts instead.
I would think that it is quite uncommon to not have github accounts among open source developers.

I'd rather not involve IRC in any way, seems like that would just complicate stuff. Gitter is very de facto on github. And rightfully so IMO.

[ghost]

Original comment by Sylvain Hellegouarch (Bitbucket: Lawouach, GitHub: Lawouach):

---

We can certainly give gitter a spin anyway :D

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

@Lawouach or @cyraxjoe, do you know who administers the Github CherryPy org? I'm not a yet a member. Could you add me as an admin so I might create test repos and test migrating the issues?

Also, there's a problem with the existing mirror - it already has pull requests against it, and since Github uses the same number sequence for pull requests and issues, it won't be possible to migrate the existing set of issues from bitbucket to the existing Github mirror and retain numbering (which is highly useful). I recommend wiping (or renaming/archiving) the existing Github repo and creating a new one, resetting the issues for a clean migration.

Who is it that maintains the current Github mirror? How is that accomplished (with hg-git perhaps)? I'd like to see if I can make a clone that links to both Git and Mercurial repos as well.

[ghost]

Original comment by Sylvain Hellegouarch (Bitbucket: Lawouach, GitHub: Lawouach):

---

Hey Jason, what's your nickname on github?

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

I would agree on a clean sheet on the github repo. Renaming the current into cherrypy_obsolete or something would keep the issues for later migration if needed(which doesn't seem likely).

[ghost]

Original comment by Sylvain Hellegouarch (Bitbucket: Lawouach, GitHub: Lawouach):

---

Okay found it.

You're admin on it now.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

Issue migration is blocked due to issue 42 in the issue migration script.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

Working around issue 42, I now am blocked by issue 65:

Imported Issue: https://api.github.com/repos/jaraco/cherrypy-mig-test-6/issues/558
Completed 558 of 1411 issues
Traceback (most recent call last):
  File "migrate.py", line 542, in <module>
    sys.exit(main(options))
  File "migrate.py", line 194, in main
    status_url, options.gh_auth, headers).json()['issue_url']
  File "migrate.py", line 518, in verify_github_issue_import_finished
    .format(status_url, respo.status_code)
RuntimeError: Failed to check GitHub issue import status url: https://api.github.com/repos/jaraco/cherrypy-mig-test-6/import/issues/558353 due to unexpected HTTP status code: 404

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

558353? Is the migration script asking for an invalid issue?

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

Ok, I see. I think.

But if the url returned by github is wrong sometimes, isn't a workaround to just not use that and build it for the time being? You have the issue and the format is known, right?

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

@niklasb I hadn't noticed that before - namely that the URL appears to include the issue number but also three additional digits. Perhaps you're right and the URL is corrupted and can be corrected client side.

I also observe that the migration completed the migration of issue #559 (see github). How is that even possible? Perhaps the 558 in 558353 was simply a coincidence and that six digit number is otherwise arbitrary.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

Confirmed - the 558 prefix was a coincidence. If I print the status URL in the migration script, here's what I see.

https://api.github.com/repos/jaraco/cherrypy-mig-test-7/import/issues/558380
Imported Issue: https://api.github.com/repos/jaraco/cherrypy-mig-test-7/issues/11
Completed 11 of 1411 issues
https://api.github.com/repos/jaraco/cherrypy-mig-test-7/import/issues/558381
Imported Issue: https://api.github.com/repos/jaraco/cherrypy-mig-test-7/issues/12
Completed 12 of 1411 issues
https://api.github.com/repos/jaraco/cherrypy-mig-test-7/import/issues/558382
Imported Issue: https://api.github.com/repos/jaraco/cherrypy-mig-test-7/issues/13
Completed 13 of 1411 issues

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

Haha, those cases are soo annoying. :-)

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

After addressing issue 65, I'm now getting another error:

Imported Issue: https://api.github.com/repos/jaraco/cherrypy-mig-test-11/issues/852
Completed 852 of 1411 issues
Traceback (most recent call last):
  File "migrate.py", line 547, in <module>
  File "migrate.py", line 173, in main
    print("\nIssue: ", gh_issue)
  File "migrate.py", line 174, in <listcomp>
    print("\nComments: ", gh_comments)
  File "migrate.py", line 339, in convert_comment
    # 'reported_by' key, so just be sure to pass in None
  File "migrate.py", line 375, in format_comment_body
    raise RuntimeError("Could not parse date: {}".format(bb_date))
  File "migrate.py", line 408, in format_user
    before, _, _ = line.partition("`")
RuntimeError: GitHub returned HTTP Status Code 403 Forbidden when accessing: https://api.github.com/users/guest.
This may be due to rate limiting.
You can read more about GitHub's API rate limiting policies here: https://developer.github.com/v3/#rate-limiting

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

I'm going to break from this for a while. Others are welcome to pull those commits and work on this latest issue.

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

The important thing is that you have started the work. Picking it up is easier than going from scratch.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

I'm resuming work on this effort.

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

Cool. Don't hesitate to ask if there is any help needed.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

I ran again with this branch, and as you might expect, it failed with the same error from before.

Imported Issue: https://api.github.com/repos/jaraco/cherrypy-mig-test-12/issues/808
Completed 808 of 1415 issues
Traceback (most recent call last):
  File "migrate.py", line 547, in <module>
    sys.exit(main(options))
  File "migrate.py", line 194, in main
    status_url, options.gh_auth, headers)
  File "migrate.py", line 523, in verify_github_issue_import_finished
    .format(status_url, respo.status_code)
RuntimeError: Failed to check GitHub issue import status url: https://api.github.com/repos/jaraco/cherrypy-mig-test-12/import/issues/653342 due to unexpected HTTP status code: 403

No mention of rate limiting, though.

Perhaps that's because it failed on the check URL.

Next step - I'll try to devise something to watch the rate-limiting headers and slow down when approaching the limit.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

Darn. So close. After applying retry on rate limiting, the process got further (though without triggering the retry logic), but still failed with a 403:

Imported Issue: https://api.github.com/repos/jaraco/cherrypy-mig-test-15/issues/1260
Completed 1260 of 1415 issues
Traceback (most recent call last):
  File "migrate.py", line 566, in <module>
    sys.exit(main(options))
  File "migrate.py", line 195, in main
    status_url, options.gh_auth, headers)
  File "migrate.py", line 542, in verify_github_issue_import_finished
    .format(status_url, respo.status_code)
RuntimeError: Failed to check GitHub issue import status url: https://api.github.com/repos/jaraco/cherrypy-mig-test-15/import/issues/654955 due to unexpected HTTP status code: 403

So it seems that it's not just failing with 404 responses on the status check but also 403 responses. Still, it was so close. Maybe another run will succeed.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

Interestingly, seems I've hit the rate limit. Restarting the process failed at the github login step.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

Aha, so there was a bug in the ratelimit detection, now fixed. With any luck, that will see the migration through.

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

Not sure how to interpret that, but are you making the actual move now of are you just finding out if it is working?
Because I suppose that we need to time this with everything else and inform the users, tell them to raise github issues and chat via gitter and so on?

[ghost]

Original comment by Nicklas Börjesson (Bitbucket: nicklasb, GitHub: nicklasb):

---

Anyway, I added the normal repo gitter chat room to the CherryPy repo. So now that is done.

Gitter automatically creates a PR to add the typical gitter link at the top of the readme.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

The last migration (cherrypy-mig-test-17) migrated successfully, with only one stall in the middle (waiting for rate limit to clear worked). But then I realized that I was doing the migration with my Github account, which has the unfortunate effect of making every issue and comment assigned to me. I don't want that, so I'm running the migration again, this time with the https://github.com/bb-migration account (something I created to migrate setuptools).

[jaraco]

The migration is complete. CherryPy can now be found at https://github.com/cherrypy/cherrypy. I'll be doing some additional cleanup work like:

• hide issues in bitbucket
• update readme and other documentation to point at github
• cut a new release so the released version refers to github
• review and accept pull requests in bitbucket

[ngrilly]

@jaraco How did you manage to keep the same issue IDs in BitBucket and GitHub?

[jaraco]

@ngrilly - the bitbucket_issue_migration project is designed to do that. It does it by first requiring that the target repo have no existing issues. It then creates new issues in github, inserting dummy issues if there are gaps in the bitbucket numbering, and then lets github assign the numbers, which usually align with the numbers from bitbucket.

[ngrilly]

@jaraco Thanks. Nice hack :-)