Migrating issues from Redmine to GitHub

[jgrocha]

QGIS Enhancement: Moving issues from Redmine to GitHub

Date 2019/03/07

Author Jorge Gustavo (@jgorcha), Régis Haubourg
Contact jgr@geomaster.pt, regis.haubourg at gmail.com

Version QGIS (all)

Summary

QGIS issues are being tracked in Redmine. Our Redmine is becoming obsolete, difficult to upgrade and difficult to use by newcomers. Managing the issues in the same repository as the code (currently in GitHub), has technical advantages and "community" related advantages as well.

This proposal aims to be as specific as possible. The goal is that it can be tested and verified before deployment by any developer.

Proposed Solution

The proposed solution is to migrate all issues from Redmine to GitHub.

The Redmine instance can remain online for a while. But if we decide in the future to move to GitLab, we just need to migrate GitHub (because all the issue history is already in GitHub).

Workflow

Overall workflow

1. Proposal for new QGIS GitHub labels
   1.1 Define how to map Redmine attributes to GitHub labels.
   1.1 Create the labels in QGIS/qgis. Proposal: Bug, Feature request, Feedback, Priority: low, Priority: normal, Priority: high, Regression
2. Create a map between Redmine users and GitHub account
   1.1 Get a list of current user names (ie first name, last name) from Redmine (query done)
   1.1 Create a very simple form on Redmine site, that receives the user name as parameter and asks for the GitHub account (code done)
   1.1 Send an email (from redmine@qgis.org) to each user with the link for the form to provide his GitHub username (script to the written)
3. Create issue templates for bugs and feature requests (Done [WIP] Add future issue templates for github QGIS#9438)
4. Contact GitHub crew to allow more request per hour. Right now, even with 1 sec. delay between each request (to create issue or comment), the script fails. Our bot has been blocked!
5. Run the migration (Migration workflow detailed below)
   1.1 Check for errors
   1.1.1 Delete the last issue migrated and rerun the script from it
   1.1 Check if all issues and comments were created
6. Update all Redmine issues. Add a comment to each saying that the issue was migrated do GitHub and add a link to the new issue on GitHub.
7. Put Redmine read only
8. Start mirroring QGIS repository on GitLab.
9. Announce that issues were migrated and from now on the GitHub should be used to fill issues
   1.1 Update QGIS documentation
   1.2 Use all channels to spread the news

Migration workflow

1. Get issues from Redmine to GitHub. All issues will be transferred to QGIS code repository (some are related with the QGIS Plugin site and some other ones, but only a few).
2. Create a mapping between Redmine attributes and GitHub labels
3. Create a mapping between Redmine users and GitHub users
4. Run migration with: Issues x label map x account map. The result will be new issues on GitHub, plus a key pair json with: redmine id → GitHub id
   4.1 Reformat Textile to Markdown
   4.2 Assign issue to original assignee if we have the usermap
5. Run migration again with: issue map. This second step is necessary to replace all occurrences of Redmine with GitHub ids.

QGIS GitHub labels

Labels in GitHub are used for PR and for issues within the repository.

We already have 27 labels that have being used (and created) for PR.

New labels: Bug, Feature request, Feedback, Priority: low, Priority: normal, Priority: high, Regression

Tracker:
Bug report → Bug
Feature request → Feature request

Status:
Open → NIL
In Progress → NIL
Feedback → Feedback
Reopened → NIL
Resolved → NIL
Closed → NIL
Rejected → NIL
Fixed for bounty → NIL
Bounty paid → NIL

Priority:
Low → Priority: low
Normal → Priority: normal
High → Priority: high

Category:
3D → NIL
Actions → NIL
Analysis library → NIL
...
Web Services clients/WMS → NIL
Web Services clients/XYZ → NIL
Windows Package → NIL
mac_os_specific → NIL

Affected QGIS version: (asked on the template)

Operating System: (asked on the template)

Regression: → Regression

Affected Files

Performance Implications

We really need help from GitHub team. It will take about 26 hours migration all issues, if we need to respected their rate limits.

Further Considerations/Improvements

Backwards Compatibility

All issue history

Issue Tracking ID(s)

Votes

(required)

[haubourg]

@jgrocha I pushed a branch in my repo here with issue templates here : https://github.com/haubourg/QGIS/tree/add_issue_templates

[3nids]

point 9. > put Redmine as read-only ?

[rduivenvoorde]

Well, it is not " difficult to maintain", because we run the vanilla debian packages now. It just runs :-)

[jgrocha]

point 9. > put Redmine as read-only ?

Workflow updated.

[jgrocha]

Well, it is not " difficult to maintain", because we run the vanilla debian packages now. It just runs :-)

I changed to "difficult to upgrade". Probably we should focus more on what we can get by moving to GitHub, instead of keep hitting the poor Redmine, that supported us for so many years.

[NathanW2]

Start mirroring QGIS repository on GitLab.

I'm ok with this generally we just have to make sure GitHub is the point of truth and people don't link or use the one at Gitlab

[gessel]

:(

[3nids]

May I suggest to remove the low and medium priority levels? We would have only nothin or tagged as high priority.

[jidanni]

User sees:

Form
Fill your GitHub acount. Do not provide the email associated with your GitHub account (although you can log in using the account name or the email).

Name 
Email 
GitHub account 
Submit
If you don't have a GitHub account or if you prefer do not share it, you still be listed as the original author. The history of contributions will be preserved.

Which is nonsense. You need to say Please provide the email associated with your GitHub account

Only then does the form make sense.

[jef-n]

I thought gitlab was the final target - the mail doesn't mention this and in the QEP it also seems to be to github only - with a (remote?) possibility to move to gitlab later.

[jgrocha]

The first initial account we used to import all the issues was blocked by GitHub. We really need their help to do about 105k API calls without being blocked. The request was already sent.

We create another one to import a few issues so we can look at how they look like in GitHub. Please take a look at:
https://github.com/qgisissuebot/QGIS/issues

I think we aren't lost anything, but please confirm. Check open and closed issues. The link to the original issue is there, so you can put them side by side.

@jef-n Commit reference detection was improved. I think we were able to catch every reference.

@jef-n Links to specific comments/notes were preserved. There is one example qgisissuebot/QGIS#129 that has the pointer to a note (right now the link still points to the Redmine issue; on a second pass, it will point to the comment on the corresponding GitHub issue).

@jef-n We also import the qgisissuebot/QGIS#138 (the proj_lpz_dist crash) that has many references to others. The references to the other related issues were kept.

@elpaso You can see something like Affected QGIS version: 3.4.0 when it was provided in the original issue. This is just clear text on the GitHub issue. We can change it to label, or we can make it pointing to a version tag, but as it is, seems enough.

@3nids You proposal to remove and seems adequately. The only really helpful is the Priority: high. More comments on this are welcome.

[jgrocha]

Only then does the form make sense.

@jidanni Thank you for your comment. I'm not sure if I fully understood your point. Please comment if I fail to answer properly.

If you provide us your email associated with your GitHub account, we can not use it to find your account. The API does not allow us to do that (they do not want people searching for accounts based on the email).

This works:

curl https://api.github.com/users/jgrocha

This does not work:

curl https://api.github.com/users/jgr@di.uminho.pt

We ask for GitHub account to add a link to the user. If users do not provide the account, we simple don't link it.

Check the screen shot below from the previous import, were we mapped some users. The first user did provide a GitHub account. On mouse over event, the user is showed. For the second user we didn't have a GitHub account, so there is no link to him.

[jgrocha]

We were blocked using the current GitHub API to upload all issues and comments to GitHub. I contact GitHub support and they are not willing to change the rate limits, as it creates infrastructure problems with the number of notifications sent to repository watchers. Not good news.

As an alternative, they pointed to a different API specific for this kind of migrations, that does not trigger messages to watchers. The documentation is available at: https://gist.github.com/jonmagic/5282384165e0f86ef105

I thought they were going to help us, but we will have to rewrite the importer. I'll report back after testing the alternative API.

[nyalldawson]

:(

@gessel can you elaborate?