Skip to content

Bug Triage

Watson edited this page May 24, 2012 · 6 revisions

Not all bug reports are created equal. Following is a collection of steps that can be taken to improve the quality of a bug report, gauge the impact of the bug, and make finding and fixing the bug as simple as possible.

Reproduce the Bug

Reproducibility is the single most important requirement for a good bug report. This is the first thing you should do when investigating a new bug. It should go without saying, but a bug that cannot be reproduced cannot be fixed!

When attempting to reproduce a bug, make sure you are working with the most recent nightly build of MacRuby. Use only the information provided in the bug report for reproduction. If you cannot reproduce a bug do not close the bug until you have confirmed with the original reporter that they also cannot reproduce the issue. Send a friendly request to the reporter to see if they are still experiencing the bug. If they are, then more information in needed for the bug report. Some common things to check for are:

  • What version/build of MacRuby are they using?
  • What version of OS X are they running?
  • What version of Xcode is installed?
  • What other gems/libraries/frameworks are they using when the bug occurs?
  • Does the bug occur in IRB? When running via macruby? When running as compiled code?
  • What environment variables do they have set?
  • Are they using a Ruby version manager (RVM or rbenv)? Are they using MacRuby via their version manager?
  • Have they installed alternate versions of system libraries? (e.g. OpenSSL installed via Homebrew or MacPorts)
  • Is their filesystem case-sensitive or case-insensitive? (You'd be surprised how many bugs come down to this...)

Create a Reduced Test Case

The less code needed to reproduce an issue, the better. Creating a reduced test case is the most important thing anyone can do to assist in the process of finding and fixing a bug. Unless the bug involves Rubygems or the require statement, try to avoid requireing anything in the test case. If the bug was discovered in a gem, try to extract the buggy code into a separate file. Eliminate as many lines of code and method calls as you can, while still reproducing the issue. If the bug is not related to scoping, try to un-nest modules and classes. Better yet, if a class is not required, try to reduce the bug to a few method definitions or method calls. Usually, this process involves some amount of experimentation, but you can learn a lot about a bug this way. Finally, make sure to record your findings in the bug report!

Adjust the Title and Tags

Once a reduced test case has been found, it may be necessary to adjust the title of the bug report to reflect the root cause of the issue. Since reading titles is the most common way to browse open bugs, it is important that the title be informative. A title like "Loading the Foo gem causes MacRuby to crash" is much less informative than "Calling the bar method on a Baz object created inside of a module causes a segfault". Try to be explicit, but also try not to make it too long. If you do change the title, be sure to make a note of the original title in the report. Similarly, make sure the tags applied to the bug are informative and appropriate. If necessary create a new tag, but try to keep the number of tags to a minimum.

Look for Duplicates

After a bug has a reduced test case and it's title has been adjusted appropriately, take a moment to look through the list of open bugs for similar looking titles. If two bugs can be reduced to the same test case, close the newer one as a duplicate (unless the newer report contains significantly more information than the older report, in which case favor closing the report with less information). This step is not super critical, but it can save the time and effort of trying to reproduce a bug that has already been fixed because it was a duplicate. Also, counting duplicate reports of a bug is a good way to gauge how many people are being affected by the bug.

Try to Craft a Reasonable Workaround

If a reasonable workaround for a bug can be found, the we can focus on other, more critical bugs. What qualifies as "reasonable" is largely subjective. Some examples of reasonable workarounds include: calling different methods to achieve identical results (e.g. using gsub and an assignment in place of gsub!), creating temporary placeholder variables, or changing the order of statements (so long as the end result is the same). Some examples of unreasonable workarounds include: changing the global structure of a program (e.g. un-nesting modules or classes), changing API exposed data types, or excessive monkey-patching of core classes. Ultimately, what qualifies as "reasonable" is up to the original reporter, so if you have found a workaround, be sure to ask them if it is reasonable.

Look for the Bug "In the Wild"

Prioritizing bugs comes down to a simple bit of mental math: take the number of people affected and divide by the difficulty of implementing a fix. So, determining where this bug occurs "in the wild" can aid in setting the correct priority. Does the bug occur in a gem? Is it a popular gem? Or, is it just in someone's project? If the report comes from a project, is it a project under development? Is it a product that has already been shipped to customers or put up on the app store? Be sure to make a note of this information in the bug report.

Regressions

Some bugs will fall into the category of "regressions". These are things that used to work, but now they don't. If this is the case, make sure that the bug is properly labeled as a regression. Ideally, we should not release a new version of MacRuby if there are unresolved regressions. Try to use git bisect to find the change that introduced the regression. If you manage to find a single commit that introduced the regression, make a note of its SHA in the bug report.

Something went wrong with that request. Please try again.