Look into upgrading Redmine (or moving to GH?)

[bitprophet]

Description

Decision made: we're moving to Github

Rationale

The improved Issues support in v2, plus the increasingly useful add-ons such as Pull Requests/commit comments/etc, means that on a feature level Github is in a much better place vs Redmine than it was a year or so ago. Redmine still has superior project management options, but none that are so essential as to outweigh the benefits of switching.

Another major reason is simply that we will not need to run our own Redmine instance any longer; this also has many pluses and minuses (see below + the link to the wiki page) but at this point in time it feels best to let somebody else deal with the hosting hassle. As long as we routinely back up the repository & extra data (issues, etc) the benefits of using a hosted service outweigh the downsides.

Means

I haven't found any existing Redmine => Github Issues v2 export scripts, but will search a bit more.

Otherwise, it seems we'll make our own using their reasonably fleshed out API. Probably a Ruby script which interfaces with Redmine's internals and makes API calls to Github.

Prior art:

• An example of API v3 use (albeit on a GH-to-GH use case) can be found in this Gist.
• gcit2ghi is a Ruby script to migrate from Google Code to Github

Either way I will probably test against my personal fork now that the main fork is at fabric/fabric, and then delete everything from there and import again to fabric/fabric once I am satisfied.

Most of the Redmine metadata will likely be preserved as tags, at least the tracker and component. Not sure what to do with the comment/submitter usernames; probably just have it all be created by my user with the original username bolded at top of body text or something. Alternately, don't bother with any comments and just include links back to the Redmine instance, as it will be read-only for some time after.

Other notes

There is no way to turn off or delete pull requests on Github and they effectively take up Issue slots, at least re: commit messages and so on.

I have copied almost all non-duplicate pull requests from Github over here to Redmine and closed them on Github's end; I will need to email Github support asking them to nuke all PRs/issues just before I import.

The only exception is PR 22 which has actual commentary and should probably be preserved. Will need to copy Redmine #22 to be a new ticket, make sure PR 22 is not deleted, and ensure that the import script skips over it.

---

Everything below is historical info.

---

Re: potentially moving to Github Issues, see this wiki page on tswicegood's GH wiki. The below is background info, and/or useful reading for a future "GH issues won't cut it, let's upgrade Redmine after all" discussion.

---

Basics:

• Current Redmine is a heavily modified version of Redmine 0.8's release branch.
• Upstream is now up to at least 1.0 representing a year, year and a half of their own dev.
• Upstream has fixed at least some of the major problems our fork addresses, such as Git support improvements.
• Go over the newest Redmine and the forked version and see what they did for us, what ours still did or did better, and see about either upgrading the fork or (probably easier/better) starting over with a copy of 1.0 and reapply whatever we need from the fork.
• There's also the issue of upgrading our 0.8 database; even outside of the custom Git stuff (which can hopefully just be axed and re-read again under the new version) there may be pain points.
• Also look into moving more/all of the collab to Github, since a big shakeup is a good time to consider that:
  • Their new pull requests are excellent code-review tools that take over all the "code related" aspects of classical Redmine style tickets (though still not fully integrated into GH Issues)
  • Speaking of which, GH issues are still a bit lighter than I'd like, but realistically the tags should cover all our use cases save for the next bullet point...
  • No nice release management like Redmine's Roadmap?
  • And the usual eggs-in-one-basket issue.

Features with branches in our fork (pretty sure all of these are represented in the version deployed at fabfile.org, some may only be internal at Jeff's job):

• Git branch support
  • Hopefully obsoleted by upstream work
• Adding in Github-style, Flash based clipboard
  • I don't use this on fabfile any, possibly in upstream
• Upgraded Capistrano deploy architecture
  • Probably negligible given that we don't really deploy this too often
• Merged-in grouping2 plugin that allows ticket grouping
  • Double check whatever the upstream plugin is at now; also not used much on fabfile
• Updated robots.txt file
  • Check on what this did and why, I don't recall offhand.
• Fix to the Markdown (or builtin?) syntax, which causes removal of 'r' characters when some WYSIWYG buttons are used
  • This may have been sent upstream to the Markdown plugin, if it's what I think it is. Check my clone on GH.
• Merge of the Markdown plugin
  • See above, this may just be the vanilla "0.8 + Markdown" merge branch
• A branch with misc small changes -- check to see what's in here
• Removed PDF functionality
  • I recall this caused problems when the site is crawled by Web crawlers; that may also be addressed to a degree in the robots.txt branch, though I think both are needed.
  • This is also something that might be addressed upstream -- check.
• Pygments support
  • Think this applied a recipe found elsewhere, not a merge of a plugin branch, but check to see if there's anything out there now
• Unique file download URLs not using OIDs
  • This was for Fabric specifically, so if upstream hasn't fixed this, we definitely need to reapply it

Other stuff to look out for:

• See ticket openid login #167, a request for OpenID support. This ought to be in 1.0 but just double check and make sure to enable it post-upgrade.
• And see Add a bot or two to the IRC channel #169 -- keep an eye on web API related stuff in 1.0 that we could use in Ghostwheel

---

Originally submitted by Jeff Forcier (bitprophet) on 2010-09-01 at 10:05pm EDT

Relations

• Related to Hit tracking #232: Hit tracking
• Related to openid login #167: openid login
• Related to Add a bot or two to the IRC channel #169: Add a bot or two to the IRC channel

[bitprophet]

Travis Swicegood (tswicegood) posted:

---

I've created a wiki page on my fork outlining pros and cons of each system:

---

on 2010-09-16 at 06:06pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Just a brief note in support of "my side" of the argument -- GitHub has had a number of outages of various types in the last week and change, including this morning :( so they're apparently not entirely out of the woods on that front.

---

on 2010-09-28 at 10:59am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

A) Yet more multi-hour GitHub outages over the last two months :(

B) Just updated your wiki page with a few things I forgot to mention till now (mostly the tight integration between commit msgs and tickets) and other general cleanup (eg removed the "github wiki is a plus" line -- Redmine has a wiki too, I just didn't enable it because I wanted to try out the MoinMoin install at wiki.fabfile.org)

---

on 2010-11-18 at 11:45pm EST

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Github Issues has just been upgraded. Have not played around with it a ton yet or updated Travis' wiki page, but that should be the next step: see how many of my "Redmine > Github" points are solved on GHI now.

---

on 2011-04-09 at 09:58pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Noted this on the wiki page, but I only now just noticed the new Issues has zero priority support or ability to sort tickets. This sucks and a lot of other people seem to be complaining too, but who knows if they will add it back in again.

It's ironic because I could see Issues' labels being used for just about every Redmine-like field except Priority, which is primarily used for its sortability instead of searchability/human readable metadata.

---

on 2011-04-12 at 10:33am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Add notes to desc re: pull requests.

---

on 2011-07-07 at 03:42pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Started developing this over at bitprophet/redmine2github, including a NOTES file with actual schema mappings and such.

Will still keep some higher level meta discussion in here...such as what Ruby REST client to use (using Ruby for this will be easiest since Redmine is Rails; no point writing my own DB wrapper for their schema in eg Python...):

• RestClient is most popular, is used by abovelinked gcit2ghi script, seems OK though the only doc for how to use basic auth requires use of the more ActiveRecord style API (meh)
• HTTParty has a silly name, is second most popular, is of the "make a class to represent your API endpoint" style (seems to work well).
• Faraday is in third place but seems way too heavy/convoluted for a one time script.

Thinking HTTParty is the way to go, will jump to RestClient if the former gives me problems.

Have a second GH user which I created to test some angles on how GHI works; may fork Fabric with it and use that as the testbed. Or just make a new unrelated repo, for that matter.

---

on 2011-07-14 at 02:51am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

HTTParty seems to work fine.

Need to set up a dev env on my old production server (no modern zsh, no rvm...cry) and get the Redmine Rails app loaded into the script, and then it's Iteration Time.

---

on 2011-07-15 at 01:42am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Started (briefly) working on the actual logic/loop; realized RestClient probably is better documented/more featureful than HTTParty (which was actually sort of a pain to use outside the initial naive test) and switched to that.

Next up is to apply JSON.load or similar to the to_str value of these response objects from the API -- neither of these libs seems to automatically do so.

---

on 2011-07-18 at 10:44am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Currently waiting to hear back from Github about whether it is even possible to delete existing issues/pull requests and to reset the ID counter. Submitted a support request a week or more ago, no response; Github-mailed Chris a few days after, no response; Twitter DM'd him yesterday, still waiting on that :(

Morgan had a good idea, which is to see about "transforming" the existing issues into their Redmine data by editing the title/description/comments. I'm not sure the comments can be edited post-facto via the API, especially those left by other people. And it would still leave bizarre links to any referencing pull-request-related forks (i.e. the opposite problem of what will happen if we don't backfill issue numbers). But it's worth a look.

Otherwise, I think a final fallback would be to recreate the lower-number Redmine issues (everything in Redmine >current Github issue number) as higher numbered ones and just write off those first few ~20ish issues as losses re: link rot, commit links and so on.

---

on 2011-08-01 at 05:26pm EDT

[bitprophet]

Import: successful! Spinoff ticket time.