Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrating issues from Redmine to GitHub #141

Open
jgrocha opened this issue Mar 7, 2019 · 78 comments

Comments

Projects
None yet
@jgrocha
Copy link

commented Mar 7, 2019

QGIS Enhancement: Moving issues from Redmine to GitHub

Date 2019/03/07

Author Jorge Gustavo (@jgorcha), Régis Haubourg
Contact jgr@geomaster.pt, regis.haubourg at gmail.com

Version QGIS (all)

Summary

QGIS issues are being tracked in Redmine. Our Redmine is becoming obsolete, difficult to upgrade and difficult to use by newcomers. Managing the issues in the same repository as the code (currently in GitHub), has technical advantages and "community" related advantages as well.

This proposal aims to be as specific as possible. The goal is that it can be tested and verified before deployment by any developer.

Proposed Solution

The proposed solution is to migrate all issues from Redmine to GitHub.

The Redmine instance can remain online for a while. But if we decide in the future to move to GitLab, we just need to migrate GitHub (because all the issue history is already in GitHub).

Workflow

Overall workflow

  1. Proposal for new QGIS GitHub labels
    1.1 Define how to map Redmine attributes to GitHub labels.
    1.1 Create the labels in QGIS/qgis. Proposal: Bug, Feature request, Feedback, Priority: low, Priority: normal, Priority: high, Regression
  2. Create a map between Redmine users and GitHub account
    1.1 Get a list of current user names (ie first name, last name) from Redmine (query done)
    1.1 Create a very simple form on Redmine site, that receives the user name as parameter and asks for the GitHub account (code done)
    1.1 Send an email (from redmine@qgis.org) to each user with the link for the form to provide his GitHub username (script to the written)
  3. Create issue templates for bugs and feature requests (Done qgis/QGIS#9438)
  4. Contact GitHub crew to allow more request per hour. Right now, even with 1 sec. delay between each request (to create issue or comment), the script fails. Our bot has been blocked!
  5. Run the migration (Migration workflow detailed below)
    1.1 Check for errors
    1.1.1 Delete the last issue migrated and rerun the script from it
    1.1 Check if all issues and comments were created
  6. Update all Redmine issues. Add a comment to each saying that the issue was migrated do GitHub and add a link to the new issue on GitHub.
  7. Put Redmine read only
  8. Start mirroring QGIS repository on GitLab.
  9. Announce that issues were migrated and from now on the GitHub should be used to fill issues
    1.1 Update QGIS documentation
    1.2 Use all channels to spread the news

Migration workflow

image

  1. Get issues from Redmine to GitHub. All issues will be transferred to QGIS code repository (some are related with the QGIS Plugin site and some other ones, but only a few).
  2. Create a mapping between Redmine attributes and GitHub labels
  3. Create a mapping between Redmine users and GitHub users
  4. Run migration with: Issues x label map x account map. The result will be new issues on GitHub, plus a key pair json with: redmine id → GitHub id
    4.1 Reformat Textile to Markdown
    4.2 Assign issue to original assignee if we have the usermap
  5. Run migration again with: issue map. This second step is necessary to replace all occurrences of Redmine with GitHub ids.

QGIS GitHub labels

Labels in GitHub are used for PR and for issues within the repository.

We already have 27 labels that have being used (and created) for PR.

New labels: Bug, Feature request, Feedback, Priority: low, Priority: normal, Priority: high, Regression

Tracker:
Bug report → Bug
Feature request → Feature request

Status:
Open → NIL
In Progress → NIL
Feedback → Feedback
Reopened → NIL
Resolved → NIL
Closed → NIL
Rejected → NIL
Fixed for bounty → NIL
Bounty paid → NIL

Priority:
Low → Priority: low
Normal → Priority: normal
High → Priority: high

Category:
3D → NIL
Actions → NIL
Analysis library → NIL
...
Web Services clients/WMS → NIL
Web Services clients/XYZ → NIL
Windows Package → NIL
mac_os_specific → NIL

Affected QGIS version: (asked on the template)

Operating System: (asked on the template)

Regression: → Regression

Affected Files

Performance Implications

We really need help from GitHub team. It will take about 26 hours migration all issues, if we need to respected their rate limits.

Further Considerations/Improvements

Backwards Compatibility

All issue history

Issue Tracking ID(s)

Votes

(required)

@jgrocha jgrocha changed the title [draft][WIP] Migrating issues from Redmine to GitHub [WIP] Migrating issues from Redmine to GitHub Mar 8, 2019

@haubourg

This comment has been minimized.

Copy link

commented Mar 8, 2019

@jgrocha I pushed a branch in my repo here with issue templates here : https://github.com/haubourg/QGIS/tree/add_issue_templates

@3nids

This comment has been minimized.

Copy link
Member

commented Mar 8, 2019

point 9. > put Redmine as read-only ?

@rduivenvoorde

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2019

Well, it is not " difficult to maintain", because we run the vanilla debian packages now. It just runs :-)

@jgrocha

This comment has been minimized.

Copy link
Author

commented Mar 8, 2019

point 9. > put Redmine as read-only ?

Workflow updated.

@jgrocha

This comment has been minimized.

Copy link
Author

commented Mar 8, 2019

Well, it is not " difficult to maintain", because we run the vanilla debian packages now. It just runs :-)

I changed to "difficult to upgrade". Probably we should focus more on what we can get by moving to GitHub, instead of keep hitting the poor Redmine, that supported us for so many years.

@NathanW2

This comment has been minimized.

Copy link
Member

commented Mar 8, 2019

Start mirroring QGIS repository on GitLab.

I'm ok with this generally we just have to make sure GitHub is the point of truth and people don't link or use the one at Gitlab

@gessel

This comment has been minimized.

Copy link

commented Mar 9, 2019

:(

@3nids

This comment has been minimized.

Copy link
Member

commented Mar 9, 2019

May I suggest to remove the low and medium priority levels? We would have only nothin or tagged as high priority.

@jidanni

This comment has been minimized.

Copy link

commented Mar 9, 2019

User sees:

Form
Fill your GitHub acount. Do not provide the email associated with your GitHub account (although you can log in using the account name or the email).

Name 
Email 
GitHub account 
Submit
If you don't have a GitHub account or if you prefer do not share it, you still be listed as the original author. The history of contributions will be preserved.

Which is nonsense. You need to say Please provide the email associated with your GitHub account

Only then does the form make sense.

@jef-n

This comment has been minimized.

Copy link
Member

commented Mar 10, 2019

I thought gitlab was the final target - the mail doesn't mention this and in the QEP it also seems to be to github only - with a (remote?) possibility to move to gitlab later.

@jgrocha

This comment has been minimized.

Copy link
Author

commented Mar 10, 2019

The first initial account we used to import all the issues was blocked by GitHub. We really need their help to do about 105k API calls without being blocked. The request was already sent.

We create another one to import a few issues so we can look at how they look like in GitHub. Please take a look at:
https://github.com/qgisissuebot/QGIS/issues

I think we aren't lost anything, but please confirm. Check open and closed issues. The link to the original issue is there, so you can put them side by side.

@jef-n Commit reference detection was improved. I think we were able to catch every reference.

@jef-n Links to specific comments/notes were preserved. There is one example qgisissuebot/QGIS#129 that has the pointer to a note (right now the link still points to the Redmine issue; on a second pass, it will point to the comment on the corresponding GitHub issue).

@jef-n We also import the qgisissuebot/QGIS#138 (the proj_lpz_dist crash) that has many references to others. The references to the other related issues were kept.

@elpaso You can see something like Affected QGIS version: 3.4.0 when it was provided in the original issue. This is just clear text on the GitHub issue. We can change it to label, or we can make it pointing to a version tag, but as it is, seems enough.

@3nids You proposal to remove and seems adequately. The only really helpful is the Priority: high. More comments on this are welcome.

@jgrocha

This comment has been minimized.

Copy link
Author

commented Mar 10, 2019

Only then does the form make sense.

@jidanni Thank you for your comment. I'm not sure if I fully understood your point. Please comment if I fail to answer properly.

If you provide us your email associated with your GitHub account, we can not use it to find your account. The API does not allow us to do that (they do not want people searching for accounts based on the email).

This works:

curl https://api.github.com/users/jgrocha

This does not work:

curl https://api.github.com/users/jgr@di.uminho.pt

We ask for GitHub account to add a link to the user. If users do not provide the account, we simple don't link it.

Check the screen shot below from the previous import, were we mapped some users. The first user did provide a GitHub account. On mouse over event, the user is showed. For the second user we didn't have a GitHub account, so there is no link to him.

image

@jgrocha

This comment has been minimized.

Copy link
Author

commented Mar 10, 2019

We were blocked using the current GitHub API to upload all issues and comments to GitHub. I contact GitHub support and they are not willing to change the rate limits, as it creates infrastructure problems with the number of notifications sent to repository watchers. Not good news.

As an alternative, they pointed to a different API specific for this kind of migrations, that does not trigger messages to watchers. The documentation is available at: https://gist.github.com/jonmagic/5282384165e0f86ef105

I thought they were going to help us, but we will have to rewrite the importer. I'll report back after testing the alternative API.

@nyalldawson

This comment has been minimized.

Copy link

commented Mar 10, 2019

:(

@gessel can you elaborate?

@gessel

This comment has been minimized.

Copy link

commented Mar 10, 2019

@nyalldawson, it is entirely off topic and irrelevant, but I like redmine, find it fairly easy to maintain, upgrade, and extend. Philosophically it is for me in the same category as QGIS itself. This doesn't matter much practically, but it has some meaning ideologically. Migrating to a corporate, for profit, closed platform tends to contradict those values, for whatever that's worth.
Practically, I would not trust that Microsoft invested $7.5B for the good of the open source community and humanity at large; there was some projected return on investment equal to or greater than other options for that capital. I don't yet see how that plays out.
Gitlab seems more consistent, perhaps @jef-n has some thoughts.
Overall, the success of the project is more important than symbolic gestures. If Github is best for the project, I'm fully supportive.

@rduivenvoorde

This comment has been minimized.

Copy link
Contributor

commented Mar 10, 2019

FYI: https://gitlab.com/qgis/QGIS/ is there to be used too (holding some old import/experiment I think; thanks @strk for 'claiming that one), maybe they are more friendly for imports like this one?

@nyalldawson

This comment has been minimized.

Copy link

commented Mar 10, 2019

@gessel

Thanks -- your feedback is certainly valued here!

The plan is actually to migrate from redmine -> github as a first step. Then, as a future piece of work it will be investigated whether a move from github to gitlab is both feasible and widely desired. But either way, getting the issues out of redmine and into the same place as the code repo is a necessary highly desirable step toward this.

@jidanni

This comment has been minimized.

Copy link

commented Mar 11, 2019

@jgrocha I see, at "github account" I am supposed to enter jidanni, not jidanni@jidanni.org.
Well OK then.

@uhliro26

This comment has been minimized.

Copy link

commented Mar 11, 2019

The chronology of the
priority changes from redmine were not migrated only comments in redmine. Is this per design?
See
https://issues.qgis.org/issues/21428#note-12 (2019-03-05-15-36)
and
qgisissuebot/QGIS#79 (comment)

@haubourg

This comment has been minimized.

Copy link

commented Mar 11, 2019

As an alternative, they pointed to a different API specific for this kind of migrations, that does not trigger messages to watchers. The documentation is available at: https://gist.github.com/jonmagic/5282384165e0f86ef105

I thought they were going to help us, but we will have to rewrite the importer. I'll report back after testing the alternative API.

Ouch, not a good move from Github's folks.

@jgrocha

This comment has been minimized.

Copy link
Author

commented Mar 11, 2019

The chronology of the
priority changes from redmine were not migrated only comments in redmine. Is this per design?
See
https://issues.qgis.org/issues/21428#note-12 (2019-03-05-15-36)
and
qgisissuebot/QGIS#79 (comment)

Well spotted @uhliro26. The Priority change is a special comment with empty notes { "notes": "" }. Maybe we can create a comment with some verbose message. I'll try and I'll report back.

@jgrocha

This comment has been minimized.

Copy link
Author

commented Mar 28, 2019

Great news! Please take a look at https://github.com/qgib/QGIS/issues
All Redmine issues were imported (the Redmine export was on March, 7).

Yabadabadoo

Feel free to search and navigate on the issues.

Issues were created by the bot account qgib (for QGIS issue bot). If you want to search for your issues, better to search with something like: is:open mentions:jgrocha

I also added @haubourg template proposal qgis/QGIS#9438 for issues and for feature requests. You can try them on the bot account. You can report issues there ;-)

@jgrocha jgrocha changed the title [WIP] Migrating issues from Redmine to GitHub Migrating issues from Redmine to GitHub Mar 28, 2019

@jgrocha

This comment has been minimized.

Copy link
Author

commented Mar 30, 2019

Migration plan (update)

The migration team needs the PSC engagement and its support to communicate with the large community of QGIS users and developers. We also need the help of the current QGIS infrastructure team to review the proposal and to provide the necessary authorizations (Redmine and Github) to run the process.

Teams:

Preparation

  • Plan and develop the scripts
  • Get and prepare all maps (for versions, labels, statuses, categories, priorities, milestones and users)
  • Test the scripts using a full migration. All issues are available at qgib/QGIS/issues
  • Vote the proposal
  • Decide the date. Date selected: 2019-05-24. We would like to have a window of 24 hours to do the migration, even if it take less than that
  • Use the media (Twitter, QGIS-User, QGIS-Developer) to announce the upcoming changes at the date defined

Actual migration (on migration day)

  • Change Redmine to read only
  • Export all issues from Redmine to individual json files
  • Grant permission to bot account qgib on QGIS/qgis repository. Permission must be admin. This can be done with: curl -X PUT -H "Content-Type: application/json" -d '{"permission":"admin"}' -H "Authorization: token <token>" https://api.github.com/repos/qgis/QGIS/collaborators/qgib using a token from one admin account.
  • Accept GitHub invitation by qgib
  • On Github, enable the “Temporary interaction limits” for a 24-hour period
  • On Github, enable issues on QGIS/qgis repository
  • Run the issue migration to GitHub (first pass of two)
  • Keep an eye on the logs (the script can break on the sever side)
  • Run the issue migration to GitHub (second pass of two)
  • Keep an eye on the logs (the script can break on the sever side)
  • Check the logs and the results

Post migration

  • On GitHub, merge pull request qgis/QGIS#9438
  • On GitHub, merge pull request (upcoming PR to change the QGIS app dialogue regarding bug reporting)
  • On GitHub, disable the "Temporary interaction limits”
  • On GitHub, merge pull requests regarding documentation and web site (start by updating https://github.com/qgis/QGIS-Website/blob/master/source/site/getinvolved/development/bugreporting.rst)
  • Simple SQL script to update all issues on Redmine. Each line will be like: insert into journals (journalized_id, journalized_type, user_id, notes, created_on) values (12345, 'Issue', 1122, 'Issue moved to https://github.com/qgis/QGIS/issues/98765', now() );
  • On Redmine, update all issues on to point to their new location using the previous script
  • Use the media (Twitter, QGIS-User, QGIS-Developer) to announce that the migration is over
  • Post a new blog entry how to submit a issue
  • Remove account qgib from QGIS repository, with curl -I -X DELETE -H "Authorization: token <token>" https://api.github.com/repos/qgis/QGIS/collaborators/qgib (or using the GitHub interface).
@nyalldawson

This comment has been minimized.

Copy link

commented Mar 31, 2019

Looks fantastic to me, great work

@jef-n

This comment has been minimized.

Copy link
Member

commented Mar 31, 2019

I miss the categories and searches feel rather unpredicable to me. Apparently I cannot search for exact phrases like Redmine original issue: 2000 let along something like category_id was changed from .* to Processing/QGIS. Looks like github just splits them into pieces and searches for the individual words.
I cannot for instance find DXF only related bugs - it'll also dumps unrelated stuff, just because DXF appears in them in an otherwise unrelated note. category_dxf however seems to work - and brings only the one issue I added it to.
BTW in ticket qgib/QGIS#17965 (#20000) there is a forward reference to issue 20009 that is left as is.

@jgrocha

This comment has been minimized.

Copy link
Author

commented Mar 31, 2019

@jef-n You are right. You found a bug (231 occurrences)! Thanks! If the issue number is followed by a carriage return, the script is not detecting it. On the input file, the carriage return is spelled literally, like "notes": "Issue 1 is now #20009\r\n\r\nIssue 2 is related to #13997, .... I'll fix it and run it again against this repository.

@haubourg

This comment has been minimized.

Copy link

commented Apr 1, 2019

I miss the categories

Yes, I do too. What about having a simplified label list and use a lookup table to affect the redmine issues to this list?
I suggest we use prefixed naming for category labels. Something like category:Labeling for instance ?

@jgrocha

This comment has been minimized.

Copy link
Author

commented Apr 1, 2019

We have more than 100 categories in Redmine. For new issues (created on GitHub) users will not use the "old" categories, but only the labels defined in GitHub.

For old issues, we can add a string for searching. We will use the string: 'category:label' (all lowercase, with '_' when there are spaces on the category).

@jgrocha

This comment has been minimized.

Copy link
Author

commented Apr 9, 2019

@jgrocha really nice work. Back to category topic, I fail to perform exact search over categories. I really think we still need a simplified category list in github labels. Am I the only one?

@haubourg Thanks for the feedback. We just made a minimalist proposal for Github labels. It is easier to get the agreement for such a minimal set. But we can create more labels.

We provide an overview of what we have on Github and Redmine.

Labels already in used (on Github)

QGIS repo already uses 27 labels, mostly used to tag PR.

Labels added by the migration scripts

We are adding the following 4 new labels ( already exist in QGIS repo)

Note: on the last migration, the label Feedback was not added (it was missing in the map).

More labels?

We can take advantage of existing categories on Redmine and create labels for them. We just need a proposal for which categories to use and the corresponding label.

To have more labels created during the migration, we just need to add more entries to the label map file to feed the script. The current map is:

redmine_type, redmine_name, github_label_name, github_label_color
status, Feedback, Feedback, 008672
tracker, Bug report, Bug, ee0701
tracker, Feature request, Feature request, 84b6eb
priority, High, Priority: high, ee0701
Component, Regression?, Regression, ee0701

From the Redmine repository:

select count(issues.*),  name 
from issue_categories, issues
where issues.category_id = issue_categories.id and issues.project_id = 17
group by issue_categories.name order by 1 desc
count name
1512 GUI
1140 Symbology
963 Unknown
935 Vectors
918 Map Composer/Printing
777 Build/Install
756 Digitising
557 Processing/QGIS
519 Processing/Core
518 Python plugins
499 Data Provider
480 Rasters
466 GRASS
433 Map Canvas
402 DB Manager
371 Projection Support
351 Web Services clients/WMS
340 Project Loading/Saving
329 Labelling
321 C++ Plugins
311 Map Legend
282 QGIS Server
282 Attribute table
227 Data Provider/PostGIS
199 Processing/Modeller
195 Browser
175 Web Services clients/WFS
173 Processing/GDAL
163 Processing/GRASS
163 Forms
151 Documentation and Help
150 Expressions
148 Data Provider/OGR
140 GDAL Tools
126 Processing/SAGA
120 Edit widget
104 Data Provider/MSSQL
101 Processing/GUI
96 Editing
87 Map Tools
82 Data Provider/Oracle
82 Translations and international
80 Plugin Manager
79 Windows Package
78 C++ plugins/Georeferencer
77 3D
77 PyQGIS Console
67 Actions
66 Data Provider/SpatiaLite
57 Field calculator
54 Python bindings / sipify
51 Geometry
44 mac_os_specific
40 MetaSearch Catalogue Client
39 DXF export
37 Analysis library
37 Data Provider/Delimited Text
36 Diagrams
34 Relations
34 OsX UI
33 Processing/OGR
30 C++ plugins/Geometry Checker
29 Data Source Manager
29 Customisation Framework
27 Raster Calculator
27 C++ plugins/Offline editing
26 Authentication system
25 Tests Suite
25 Virtual Layers
23 C++ plugins/Topology checker
21 Data Provider/MDAL
21 C++ plugins/Globe
20 Web Services clients/WCS
20 C++ plugins/GPS plugin
18 Decorations
17 Network
17 DWG/DXF import
16 Web Services clients/XYZ
15 Web Services clients/ArcGIS
13 Virtual Fields
11 Metadata
9 C++ plugins/Evis
8 GPS Live Tracking
8 C++ plugins/Spatial Query
8 Simplification
4 Data Provider/GeoNode
3 Processing/OTB
2 Overview
2 C++ plugins/Zonal statistics
1 OpenCL
1 Processing/WPS

Mapping all these categories to Github labels is too much. But if we can create a subset of these, it would be perfect. Suggestions are welcome.

@haubourg

This comment has been minimized.

Copy link

commented Apr 11, 2019

@jgrocha I tried a first attempt to reclassify categories:

I easily go down from 91 to 29 categories, but I think it is still a lot. Something like ~15/20 would be nicer. I discarded the category "documentation", assuming that this will move to the dedicated repository.
Any idea to squeeze that down ?

count name new label
1512 GUI GUI
1140 Symbology Symbology
963 Unknown
935 Vectors Vectors
918 Map Composer/Printing Map Layout
777 Build/Install Build/Install
756 Digitising Digitising
557 Processing/QGIS Processing
519 Processing/Core Processing
518 Python plugins Plugins
499 Data Provider Data Provider
480 Rasters Raster
466 GRASS Processing
433 Map Canvas Map and legend
402 DB Manager Database
371 Projection Support Projection Support
351 Web Services clients/WMS Data Provider
340 Project Loading/Saving  Project
329 Labelling Labeling
321 C++ Plugins Plugins
311 Map Legend Map and legend
282 QGIS Server QGIS Server
282 Attribute table Attribute table
227 Data Provider/PostGIS Data Provider
199 Processing/Modeller Processing
195 Browser  
175 Web Services clients/WFS Data Provider
173 Processing/GDAL Processing
163 Processing/GRASS Processing
163 Forms Forms
151 Documentation and Help
150 Expressions Expressions
148 Data Provider/OGR Data Provider
140 GDAL Tools Raster
126 Processing/SAGA Processing
120 Edit widget Forms
104 Data Provider/MSSQL Data Provider
101 Processing/GUI Processing
96 Editing Editing
87 Map Tools Map tools
82 Data Provider/Oracle Data Provider
82 Translations and international Translations and international
80 Plugin Manager Plugins
79 Windows Package Build/Install
78 C++ plugins/Georeferencer Plugins
77 3D 3D
77 PyQGIS Console PyQGIS
67 Actions Actions
66 Data Provider/SpatiaLite Data Provider
57 Field calculator Expressions
54 Python bindings / sipify PyQGIS
51 Geometry Expressions
44 mac_os_specific Build/Install
40 MetaSearch Catalogue Client Plugins
39 DXF export CAD
37 Analysis library API
37 Data Provider/Delimited Text Data Provider
36 Diagrams Symbology
34 Relations Forms
34 OsX UI Build/Install
33 Processing/OGR Processing
30 C++ plugins/Geometry Checker Plugins
29 Data Source Manager Data Provider
29 Customisation Framework GUI
27 Raster Calculator Raster
27 C++ plugins/Offline editing Plugins
26 Authentication system Authentication
25 Tests Suite Quality Assessment
25 Virtual Layers Database
23 C++ plugins/Topology checker Plugins
21 Data Provider/MDAL Data Provider
21 C++ plugins/Globe Plugins
20 Web Services clients/WCS Data Provider
20 C++ plugins/GPS plugin Plugins
18 Decorations Map and legend
17 Network API
17 DWG/DXF import CAD
16 Web Services clients/XYZ Data Provider
15 Web Services clients/ArcGIS Data Provider
13 Virtual Fields Database
11 Metadata Metadata
9 C++ plugins/Evis Plugins
8 GPS Live Tracking GPS
8 C++ plugins/Spatial Query Plugins
8 Simplification Expressions
4 Data Provider/GeoNode Data Provider
3 Processing/OTB Processing
2 Overview Map and legend
2 C++ plugins/Zonal statistics Plugins
1 OpenCL  
1 Processing/WPS Processing
@jgrocha

This comment has been minimized.

Copy link
Author

commented Apr 11, 2019

@haubourg Looks great to me. 29 labels are not too much. If we look around (other successful Github projects) we can see many with 100 and more labels.

Two comments:

  1. I would keep the Map Composer/Printing as it is right now. Map Layout sound too general (and has more interception with Map and legend).
  2. GRASS can be under Processing, but we have so many issues related with GRASS, that I would keep GRASS as a label, so our GRASS experts (and people from GRASS team) can pick them quickly.
@nyalldawson

This comment has been minimized.

Copy link

commented Apr 11, 2019

I'd suggest "print layouts" is a more modern term for the composer label. I wonder about "Actions". We could merge "C++ plugins" and "plugins" (most users wouldn't know the difference anyway!). Quality assessment could also just use the existing "ci/infrastructure" label? That drops another 3 at least 😀

@gioman

This comment has been minimized.

Copy link

commented Apr 14, 2019

One minor suggestion: can we have different colors (not red, maybe orange/yellow) for "regression" and "high priority"? Also I think we miss a label for "causes crash or data corruption" (?), if confirmed I think that this is an important flag of Redmine tickets we should keep here on github.

@qgib

This comment has been minimized.

Copy link

commented Apr 14, 2019

One minor suggestion: can we have different colors (not red, maybe orange/yellow) for "regression" and "high priority"? Also I think we miss a label for "causes crash or data corruption" (?), if confirmed I think that this is an important flag of Redmine tickets we should keep here on github.

  1. Yes, using a different colour might be better.

  2. Good point! Can we call this label just "Data corruption"? I'll improve the migration script to catch this on old Redmine issues.

@gioman

This comment has been minimized.

Copy link

commented Apr 14, 2019

2\. Good point! Can we call this label just "Data corruption"? I'll improve the migration script to catch this on old Redmine issues.

the "crash" label is more important (is fundamental), if we have to choose, but if can have both is better.

@haubourg

This comment has been minimized.

Copy link

commented Apr 15, 2019

One minor suggestion: can we have different colors (not red, maybe orange/yellow) for "regression" and "high priority"? Also I think we miss a label for "causes crash or data corruption" (?), if confirmed I think that this is an important flag of Redmine tickets we should keep here on githu

I totally agree, the color code we choose is almost as important as the text itself.
Let's try to put some rules so that we keep some readability when searching for issues with many label tags, with the help of http://colorbrewer2.org :

  • Primary red only for regressions #f00

  • crash label could be black

image

  • Priorities should be in a coherent color ramp. I'd suggest this one

image

  • All other categories should be using soft colors, grouped by family if we have some kind of groups. I'll try to see if this is possible.
@jgrocha

This comment has been minimized.

Copy link
Author

commented Apr 21, 2019

migration-issues-to-github

We have done another successful migration, adding much more Github labels, according to the suggestions of @haubourg and @gioman. We have changed a few colours (but this can be done later, at any time). The Redmine snapshot is from 2019-04-13.

Please check the result at: https://github.com/qgib/QGIS/issues

Bug and Feature request templates are enabled for testing on this qgib repository.

If someone wants to have access to this https://github.com/qgib/ repository, feel free to ask for the credentials (for organizing issues with projects boards, to play with label colour, etc).

Although not very important, we have fixed the majority of false positives references to issues from thread dumps, where each line starts with #1, #2, etc. Example: https://github.com/qgib/QGIS/issues/17309 (only references to issues were made).

@mbernasocchi

This comment has been minimized.

Copy link
Member

commented May 6, 2019

@jgrocha this all looks great to me. do you want me to trigger a psc vote today?

@qgib

This comment has been minimized.

Copy link

commented May 6, 2019

@luipir

This comment has been minimized.

Copy link

commented May 23, 2019

congratulation to all worked in this and in this GEP :)

@haubourg

This comment has been minimized.

Copy link

commented May 23, 2019

Yeah! Kudos @jgrocha for tackling a such long running issue 👍

@mbernasocchi

This comment has been minimized.

Copy link
Member

commented May 23, 2019

just for reference here the:

@qgib

This comment has been minimized.

Copy link

commented May 25, 2019

The migration itself has ended and the interaction limitation was removed. Issue and feature request templates are in place and workig as expected.

We still need to do some post processing. Work is not over yet.

Tomorrow I'll report back here.

@qgib

This comment has been minimized.

Copy link

commented May 26, 2019

@jef-n I've fixed the markdown translation of inline images for issues still open.

@jef-n

This comment has been minimized.

Copy link
Member

commented May 26, 2019

@jgrocha Thanks. Example: qgis/QGIS#29945

BTW I just closed qgis/QGIS#29862 - I knew that there was a ticket, but I couldn't find it here by searching for "widget.value" - redmine did find it - turns out looking for "self.widget.value" also does find it here. Any idea why that is?

@jef-n

This comment has been minimized.

Copy link
Member

commented May 26, 2019

BTW I just closed qgis/QGIS#29862 - I knew that there was a ticket, but I couldn't find it here by searching for "widget.value" - redmine did find it - turns out looking for "self.widget.value" also does find it here. Any idea why that is?

Apparently you cannot search for substrings in github

@qgib

This comment has been minimized.

Copy link

commented May 26, 2019

@jef-n @rduivenvoorde

@strk already asked for the map from Redmine to GitHub. Can one of you review the following SQL code and run it against our redmine database?

Please do the insert for testing and chech if it displays properly. If it does, you can run the loop. Feel free to change and improved it.

I can do it, if you provide me the postgresql connect string (with the credentials).

The map file to load into the database is attached.

-- create table with map
CREATE TABLE redmine2github_map (redmine_id integer, github_id integer);
COPY redmine2github_map FROM '/home/jgr/bonn/ownCloud/geomaster/QGisUserConf 2019/map.csv' csv header;

--test with one
insert into journals (journalized_id, journalized_type, user_id, notes, created_on) values (10245, 'Issue', 1122, 'Issue moved to https://github.com/qgis/QGIS/issues/18682', now() );
-- check the output at: https://issues.qgis.org/issues/10245

-- write a loop to update every issue
DO $do$
DECLARE
    pair RECORD;
BEGIN
    FOR pair in SELECT redmine_id, github_id FROM redmine2github_map limit 10
    loop
    	raise notice 'Redmine issue %', format('Issue %s moved to %s', pair.redmine_id, pair.github_id);	
    	EXECUTE format($$ insert into journals (journalized_id, journalized_type, user_id, notes, created_on) values (%s, 'Issue', 1122, 'Issue moved to https://github.com/qgis/QGIS/issues/%s', now() ) $$, pair.redmine_id, pair.github_id);
    END LOOP;
END;
$do$;

map.zip

@jef-n

This comment has been minimized.

Copy link
Member

commented May 26, 2019

The map file to load into the database is attached.

I added a custom field and filled it with the map instead ("Copied to github as #").

@jgrocha

This comment has been minimized.

Copy link
Author

commented May 26, 2019

The technical part of the migration is over. All issues are on GitHub and the templates for reporting issues are in place.

We have a bidirectional map between Github and Redmine (thanks to @jef-n), so it is easy to switch between them. It really nice to be able to move between the two platforms with just one click.

image

We need to update and improve the documentation, use QGIS blog to tell users about the changes, update our workshop materials, and so on. It will take some time.

I've already sent the bot credentials (qgib account) to the PSC and I'll no longer use that account.

We still have a problem… more than 3000 open issues to manage! QGIS success creates a whole set of problems, right?

Let's start cleaning the issue queue, but first we need to express our gratitude for all the valuable work of those supporting Redmine until today. There were 19844 issues and that means a lot of triage effort and maintenance behind the scenes. Please join me in saying "Thank You" to @gioman, @jef-n, @rduivenvoorde and others running Redmine (since 2010).

@gioman

This comment has been minimized.

Copy link

commented May 26, 2019

@jgrocha and all others involved, thanks for the effort

We still have a problem… more than 3000 open issues to manage! QGIS success creates a whole set of problems, right?

They are not more than they were on Redmine :) Actually the real bug are "only" 1283, and of those ~250 are awaiting feedback for more than 15 days, so likely to be closed as not reproducible or because already fixed. Usually those tickets (bugs, in feedback since long) are the ones I focus when I start some cleanup.

Possibly a large chunk of feature requests could be closed (as already implemented) but I feel they don't get much attention (personally I don't look to feature requests as frequently as I do for bugs).

There were 19844 issues and that means a lot of triage effort and maintenance behind the scenes. Please join me in saying "Thank You" to @gioman, @jef-n, @rduivenvoorde and others running Redmine (since 2010).

Thank you, after 1084 tickets open and 24053 comments (both high scores) on Redmine let's see how it plays out doing the same here.

@strk

This comment has been minimized.

Copy link

commented May 30, 2019

For reference, I've filed a ticket for Gitea to request allowing commit-metadata based redirection of issue references: go-gitea/gitea#7084 -- not sure it'll ever be implemented

@jef-n

This comment has been minimized.

Copy link
Member

commented Jun 2, 2019

@jgrocha issue qgis/QGIS#29494 references ticket #21367, which actually is the redmine ticket number. It should reference qgis/QGIS#29184 instead.

@jgrocha

This comment has been minimized.

Copy link
Author

commented Jun 2, 2019

@jgrocha issue qgis/QGIS#29494 references ticket #21367, which actually is the redmine ticket number. It should reference qgis/QGIS#29184 instead.

I can't find an explanation. The issue number was detected. In the the migration log we can see that two issues were identified, as expected. The mapping pair is present in the mapping.

May 25 21:30:07 (19393) Loading redmine issue: [21678] from file [21678.json]
May 25 21:30:07 ----------------------------------------
May 25 21:30:07 29488
May 25 21:30:07 ----------------------------------------
May 25 21:30:07 Match (2nd) found: 21367
May 25 21:30:07 ----------------------------------------
May 25 21:30:07 ----------------------------------------
May 25 21:30:07 Match (2nd) found: 21672
May 25 21:30:07 ----------------------------------------
May 25 21:30:07 Comment updated!
May 25 21:30:07 Redmine related issues: [21672](https://issues.qgis.org/issues/21672)
May 25 21:30:07 Github related issues: #29488 (duplicates)
May 25 21:30:07 Redmine sub-issues:
May 25 21:30:07 Github sub-issues:
May 25 21:30:07 Issue updated!

There is no error reported from the GitHub side.

I've identified similar issues (with the same structure) and all were rewritten properly. If we can find another one where the ID was not replaced, maybe we can understand where the script failed.

If I find any clue, I'll return to you.

@jef-n

This comment has been minimized.

Copy link
Member

commented Jun 3, 2019

@jgrocha the updated description from github_issue_maker.py:246 is overwritten when there are related tickets. Does https://gist.github.com/jef-n/f6f0f52d577e615bfcc6dc5ae2b66468 help?

@jgrocha

This comment has been minimized.

Copy link
Author

commented Jun 4, 2019

@jgrocha the updated description from github_issue_maker.py:246 is overwritten when there are related tickets. Does https://gist.github.com/jef-n/f6f0f52d577e615bfcc6dc5ae2b66468 help?

@jef-n Thank god you exist! The problem are the ones with issues references on the description and related issues, as you spotted.

Doing a query (on Redmine!), it reports 222 issues, but some of them are false positives. I've done a query searching for bugs or feature requests, having related issues and # on the description. The ones with references to issues are less than that.

My proposal is check each one and do the proper replacement. It is dangerous to run the migration script again, because we might change issues already mapped. I've wrote a small script to do this changes, and it only changes the part of the description. (The issue description is divided in tree parts and we can trust the \n---\n separator).

My proposal is to call it for each issue (from the github_candidates.csv file), like:

$ python3 issues.py 29637
Redmine issue not in map
Redmine issue not in map
Redmine issue not in map
Number of replacements: 0
Skip issue 29637
$ python3 issues.py 29598
Redmine 20921 mapped to 28740
Redmine 20980 mapped to 28799
Redmine 21648 mapped to 29464
Number of replacements: 3

and so on.

The script and the two supporting files are at: https://gist.github.com/jgrocha/be7853392bc921f4be23d4e704f933e5

@jef-n Can you take a look at the script?
https://gist.github.com/jgrocha/be7853392bc921f4be23d4e704f933e5#file-issues-py

Right now, for testing, the script creates a copy of the issue in one of my repositories. By changing the authentication and uncomment the issue.edit(body=newbody) we get it running on our qgis/QGIS repo. Doing one by one is safe, I think, and I can revert any if something goes wrong.

@jgrocha

This comment has been minimized.

Copy link
Author

commented Jun 7, 2019

I've fixed the ids that weren't mapped at the migration time. 153 issues were edited and 181 references were fixed. Thanks @jef-n for spotting the missing mappings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.