Migrating issues from Redmine to GitHub #141
Comments
jgrocha
changed the title
|
@jgrocha I pushed a branch in my repo here with issue templates here : https://github.com/haubourg/QGIS/tree/add_issue_templates |
|
point 9. > put Redmine as read-only ? |
|
Well, it is not " difficult to maintain", because we run the vanilla debian packages now. It just runs :-) |
Workflow updated. |
I changed to "difficult to upgrade". Probably we should focus more on what we can get by moving to GitHub, instead of keep hitting the poor Redmine, that supported us for so many years. |
I'm ok with this generally we just have to make sure GitHub is the point of truth and people don't link or use the one at Gitlab |
|
:( |
|
May I suggest to remove the low and medium priority levels? We would have only nothin or tagged as high priority. |
|
User sees: Form Name  Which is nonsense. You need to say Please provide the email associated with your GitHub account Only then does the form make sense. |
|
I thought gitlab was the final target - the mail doesn't mention this and in the QEP it also seems to be to github only - with a (remote?) possibility to move to gitlab later. |
|
The first initial account we used to import all the issues was blocked by GitHub. We really need their help to do about 105k API calls without being blocked. The request was already sent. We create another one to import a few issues so we can look at how they look like in GitHub. Please take a look at: I think we aren't lost anything, but please confirm. Check open and closed issues. The link to the original issue is there, so you can put them side by side. @jef-n Commit reference detection was improved. I think we were able to catch every reference. @jef-n Links to specific comments/notes were preserved. There is one example qgisissuebot/QGIS#129 that has the pointer to a note (right now the link still points to the Redmine issue; on a second pass, it will point to the comment on the corresponding GitHub issue). @jef-n We also import the qgisissuebot/QGIS#138 (the @elpaso You can see something like Affected QGIS version: 3.4.0 when it was provided in the original issue. This is just clear text on the GitHub issue. We can change it to label, or we can make it pointing to a version tag, but as it is, seems enough. @3nids You proposal to remove |
@jidanni Thank you for your comment. I'm not sure if I fully understood your point. Please comment if I fail to answer properly. If you provide us your email associated with your GitHub account, we can not use it to find your account. The API does not allow us to do that (they do not want people searching for accounts based on the email). This works: This does not work: We ask for GitHub account to add a link to the user. If users do not provide the account, we simple don't link it. Check the screen shot below from the previous import, were we mapped some users. The first user did provide a GitHub account. On mouse over event, the user is showed. For the second user we didn't have a GitHub account, so there is no link to him.
|
|
We were blocked using the current GitHub API to upload all issues and comments to GitHub. I contact GitHub support and they are not willing to change the rate limits, as it creates infrastructure problems with the number of notifications sent to repository watchers. Not good news. As an alternative, they pointed to a different API specific for this kind of migrations, that does not trigger messages to watchers. The documentation is available at: https://gist.github.com/jonmagic/5282384165e0f86ef105 I thought they were going to help us, but we will have to rewrite the importer. I'll report back after testing the alternative API. |
@gessel can you elaborate? |
|
@nyalldawson, it is entirely off topic and irrelevant, but I like redmine, find it fairly easy to maintain, upgrade, and extend. Philosophically it is for me in the same category as QGIS itself. This doesn't matter much practically, but it has some meaning ideologically. Migrating to a corporate, for profit, closed platform tends to contradict those values, for whatever that's worth. |
|
FYI: https://gitlab.com/qgis/QGIS/ is there to be used too (holding some old import/experiment I think; thanks @strk for 'claiming that one), maybe they are more friendly for imports like this one? |
|
Thanks -- your feedback is certainly valued here! The plan is actually to migrate from redmine -> github as a first step. Then, as a future piece of work it will be investigated whether a move from github to gitlab is both feasible and widely desired. But either way, getting the issues out of redmine and into the same place as the code repo is a |
|
@jgrocha I see, at "github account" I am supposed to enter jidanni, not jidanni@jidanni.org. |
|
The chronology of the |
Ouch, not a good move from Github's folks. |
Well spotted @uhliro26. The Priority change is a special comment with empty notes |
|
Great news! Please take a look at https://github.com/qgib/QGIS/issues
Feel free to search and navigate on the issues. Issues were created by the bot account qgib (for QGIS issue bot). If you want to search for your issues, better to search with something like: I also added @haubourg template proposal qgis/QGIS#9438 for issues and for feature requests. You can try them on the bot account. You can report issues there ;-) |
jgrocha
changed the title
Migration plan (update)The migration team needs the PSC engagement and its support to communicate with the large community of QGIS users and developers. We also need the help of the current QGIS infrastructure team to review the proposal and to provide the necessary authorizations (Redmine and Github) to run the process. Teams:
Preparation
Actual migration (on migration day)
Post migration
|
|
Looks fantastic to me, great work |
|
I miss the categories and searches feel rather unpredicable to me. Apparently I cannot search for exact phrases like |
|
@jef-n You are right. You found a bug (231 occurrences)! Thanks! If the issue number is followed by a carriage return, the script is not detecting it. On the input file, the carriage return is spelled literally, like |
Yes, I do too. What about having a simplified label list and use a lookup table to affect the redmine issues to this list? |
|
We have more than 100 categories in Redmine. For new issues (created on GitHub) users will not use the "old" categories, but only the labels defined in GitHub. For old issues, we can add a string for searching. We will use the string: 'category:label' (all lowercase, with '_' when there are spaces on the category). |
|
I'd suggest "print layouts" is a more modern term for the composer label. I wonder about "Actions". We could merge "C++ plugins" and "plugins" (most users wouldn't know the difference anyway!). Quality assessment could also just use the existing "ci/infrastructure" label? That drops another 3 at least 😀 |
|
One minor suggestion: can we have different colors (not red, maybe orange/yellow) for "regression" and "high priority"? Also I think we miss a label for "causes crash or data corruption" (?), if confirmed I think that this is an important flag of Redmine tickets we should keep here on github. |
|
the "crash" label is more important (is fundamental), if we have to choose, but if can have both is better. |
I totally agree, the color code we choose is almost as important as the text itself.
|
|
We have done another successful migration, adding much more Github labels, according to the suggestions of @haubourg and @gioman. We have changed a few colours (but this can be done later, at any time). The Redmine snapshot is from 2019-04-13. Please check the result at: https://github.com/qgib/QGIS/issues Bug and Feature request templates are enabled for testing on this qgib repository. If someone wants to have access to this https://github.com/qgib/ repository, feel free to ask for the credentials (for organizing issues with projects boards, to play with label colour, etc). Although not very important, we have fixed the majority of false positives references to issues from thread dumps, where each line starts with |
|
@jgrocha this all looks great to me. do you want me to trigger a psc vote today? |
|
Yes, yes, please! Everything is ready! Thank you!
|
|
congratulation to all worked in this and in this GEP :) |
|
Yeah! Kudos @jgrocha for tackling a such long running issue 👍 |
|
just for reference here the: |
|
The migration itself has ended and the interaction limitation was removed. Issue and feature request templates are in place and workig as expected. We still need to do some post processing. Work is not over yet. Tomorrow I'll report back here. |
|
@jef-n I've fixed the markdown translation of inline images for issues still open. |
|
@jgrocha Thanks. Example: qgis/QGIS#29945 BTW I just closed qgis/QGIS#29862 - I knew that there was a ticket, but I couldn't find it here by searching for "widget.value" - redmine did find it - turns out looking for "self.widget.value" also does find it here. Any idea why that is? |
Apparently you cannot search for substrings in github |
|
@strk already asked for the map from Redmine to GitHub. Can one of you review the following SQL code and run it against our redmine database? Please do the I can do it, if you provide me the postgresql connect string (with the credentials). The map file to load into the database is attached. -- create table with map
CREATE TABLE redmine2github_map (redmine_id integer, github_id integer);
COPY redmine2github_map FROM '/home/jgr/bonn/ownCloud/geomaster/QGisUserConf 2019/map.csv' csv header;
--test with one
insert into journals (journalized_id, journalized_type, user_id, notes, created_on) values (10245, 'Issue', 1122, 'Issue moved to https://github.com/qgis/QGIS/issues/18682', now() );
-- check the output at: https://issues.qgis.org/issues/10245
-- write a loop to update every issue
DO $do$
DECLARE
pair RECORD;
BEGIN
FOR pair in SELECT redmine_id, github_id FROM redmine2github_map limit 10
loop
raise notice 'Redmine issue %', format('Issue %s moved to %s', pair.redmine_id, pair.github_id);
EXECUTE format($$ insert into journals (journalized_id, journalized_type, user_id, notes, created_on) values (%s, 'Issue', 1122, 'Issue moved to https://github.com/qgis/QGIS/issues/%s', now() ) $$, pair.redmine_id, pair.github_id);
END LOOP;
END;
$do$; |
I added a custom field and filled it with the map instead ("Copied to github as #"). |
|
The technical part of the migration is over. All issues are on GitHub and the templates for reporting issues are in place. We have a bidirectional map between Github and Redmine (thanks to @jef-n), so it is easy to switch between them. It really nice to be able to move between the two platforms with just one click.
We need to update and improve the documentation, use QGIS blog to tell users about the changes, update our workshop materials, and so on. It will take some time. I've already sent the bot credentials (qgib account) to the PSC and I'll no longer use that account. We still have a problem… more than 3000 open issues to manage! QGIS success creates a whole set of problems, right? Let's start cleaning the issue queue, but first we need to express our gratitude for all the valuable work of those supporting Redmine until today. There were 19844 issues and that means a lot of triage effort and maintenance behind the scenes. Please join me in saying "Thank You" to @gioman, @jef-n, @rduivenvoorde and others running Redmine (since 2010). |
|
@jgrocha and all others involved, thanks for the effort
They are not more than they were on Redmine :) Actually the real bug are "only" 1283, and of those ~250 are awaiting feedback for more than 15 days, so likely to be closed as not reproducible or because already fixed. Usually those tickets (bugs, in feedback since long) are the ones I focus when I start some cleanup. Possibly a large chunk of feature requests could be closed (as already implemented) but I feel they don't get much attention (personally I don't look to feature requests as frequently as I do for bugs).
Thank you, after 1084 tickets open and 24053 comments (both high scores) on Redmine let's see how it plays out doing the same here. |
|
For reference, I've filed a ticket for Gitea to request allowing commit-metadata based redirection of issue references: go-gitea/gitea#7084 -- not sure it'll ever be implemented |
|
@jgrocha issue qgis/QGIS#29494 references ticket #21367, which actually is the redmine ticket number. It should reference qgis/QGIS#29184 instead. |
I can't find an explanation. The issue number was detected. In the the migration log we can see that two issues were identified, as expected. The mapping pair is present in the mapping. There is no error reported from the GitHub side. I've identified similar issues (with the same structure) and all were rewritten properly. If we can find another one where the ID was not replaced, maybe we can understand where the script failed. If I find any clue, I'll return to you. |
|
@jgrocha the updated description from github_issue_maker.py:246 is overwritten when there are related tickets. Does https://gist.github.com/jef-n/f6f0f52d577e615bfcc6dc5ae2b66468 help? |
@jef-n Thank god you exist! The problem are the ones with issues references on the description and related issues, as you spotted. Doing a query (on Redmine!), it reports 222 issues, but some of them are false positives. I've done a query searching for bugs or feature requests, having related issues and My proposal is check each one and do the proper replacement. It is dangerous to run the migration script again, because we might change issues already mapped. I've wrote a small script to do this changes, and it only changes the part of the description. (The issue description is divided in tree parts and we can trust the My proposal is to call it for each issue (from the and so on. The script and the two supporting files are at: https://gist.github.com/jgrocha/be7853392bc921f4be23d4e704f933e5 @jef-n Can you take a look at the script? Right now, for testing, the script creates a copy of the issue in one of my repositories. By changing the authentication and uncomment the |
|
I've fixed the ids that weren't mapped at the migration time. 153 issues were edited and 181 references were fixed. Thanks @jef-n for spotting the missing mappings. |
QGIS Enhancement: Moving issues from Redmine to GitHub
Date 2019/03/07
Author Jorge Gustavo (@jgorcha), Régis Haubourg
Contact jgr@geomaster.pt, regis.haubourg at gmail.com
Version QGIS (all)
Summary
QGIS issues are being tracked in Redmine. Our Redmine is becoming obsolete, difficult to upgrade and difficult to use by newcomers. Managing the issues in the same repository as the code (currently in GitHub), has technical advantages and "community" related advantages as well.
This proposal aims to be as specific as possible. The goal is that it can be tested and verified before deployment by any developer.
Proposed Solution
The proposed solution is to migrate all issues from Redmine to GitHub.
The Redmine instance can remain online for a while. But if we decide in the future to move to GitLab, we just need to migrate GitHub (because all the issue history is already in GitHub).
Workflow
Overall workflow
1.1 Define how to map Redmine attributes to GitHub labels.
1.1 Create the labels in QGIS/qgis. Proposal: Bug, Feature request, Feedback, Priority: low, Priority: normal, Priority: high, Regression
1.1 Get a list of current user names (ie first name, last name) from Redmine (query done)
1.1 Create a very simple form on Redmine site, that receives the user name as parameter and asks for the GitHub account (code done)
1.1 Send an email (from redmine@qgis.org) to each user with the link for the form to provide his GitHub username (script to the written)
1.1 Check for errors
1.1.1 Delete the last issue migrated and rerun the script from it
1.1 Check if all issues and comments were created
1.1 Update QGIS documentation
1.2 Use all channels to spread the news
Migration workflow
4.1 Reformat Textile to Markdown
4.2 Assign issue to original assignee if we have the usermap
QGIS GitHub labels
Labels in GitHub are used for PR and for issues within the repository.
We already have 27 labels that have being used (and created) for PR.
New labels: Bug, Feature request, Feedback, Priority: low, Priority: normal, Priority: high, Regression
Tracker:
Bug report → Bug
Feature request → Feature request
Status:
Open → NIL
In Progress → NIL
Feedback → Feedback
Reopened → NIL
Resolved → NIL
Closed → NIL
Rejected → NIL
Fixed for bounty → NIL
Bounty paid → NIL
Priority:
Low → Priority: low
Normal → Priority: normal
High → Priority: high
Category:
3D → NIL
Actions → NIL
Analysis library → NIL
...
Web Services clients/WMS → NIL
Web Services clients/XYZ → NIL
Windows Package → NIL
mac_os_specific → NIL
Affected QGIS version: (asked on the template)
Operating System: (asked on the template)
Regression: → Regression
Affected Files
Performance Implications
We really need help from GitHub team. It will take about 26 hours migration all issues, if we need to respected their rate limits.
Further Considerations/Improvements
Backwards Compatibility
All issue history
Issue Tracking ID(s)
Votes
(required)
The text was updated successfully, but these errors were encountered: