KnownFailure tests appearing as errors

[takluyver]

Running iptest IPython.core, I get two tests failing because they raise KnownFailureTest (they should show up as K, and not be counted as failures). This is on Ubuntu Oneiric, with nose 1.0.0 installed systemwide. git bisect tracks it down to commit 8274f38, which changed how we load nose plugins.

[minrk]

Weird - I get the correct behavior with nose 0.11.4, and current (1.1.2) on Ubuntu 10.04, 11.04, and OSX.

Do you have numpy? The imports for the KnownFailures are different depending on whether numpy is present.

[takluyver]

Numpy 1.5.1, also installed systemwide.

[minrk]

hm, same for me on my 11.04 machine.

Is IPython.external.decorators.KnownFailure the numpy one, or the bundled one?

It makes the most sense to me that this would be a mismatch in how nose registers plugins, but I don't see how that would be true for nose both older and newer than yours.

[takluyver]

IPython.external.decorators.KnownFailure is the one defined in numpy.testing.noseclasses. Installing nose 1.1.2 inside the virtualenv where I'm testing doesn't resolve the issue.

[fperez]

This is seriously bizarre. I've tracked it down somewhat, but I don't have an answer. This is a way to run just one test that fails, but with lots of debug info:

iptest -vvvvsx IPython.core.tests.test_magic:test_numpy_clear_array_undec

Here's the key: at the top, I see:

nose.plugins.manager: DEBUG: Plugins enabled: [
IPython.testing.plugin.ipdoctest.IPythonDoctest object at ..., 
nose.plugins.logcapture.LogCapture object at 0x476ff10,
nose.plugins.deprecated.Deprecated object at 0x7f3204026090, 
nose.plugins.failuredetail.FailureDetail object at 0x7f3204026210,
nose.plugins.skip.Skip object at 0x7f3204026310,
matplotlib.testing.noseclasses.KnownFailure object at 0x48dc350
]

The mystery is: what the hell is that matplotlib plugin doing there? We're explictly building our own KnownFailure plugin, I have no clue how that one ends up in there.

Furthermore, if I install a new matplotlib to my home directory from source, this problem goes away. That, despite the fact that the plugin code in matplotlib hasn't changed at all!

I'm pretty stumped.

[fperez]

Some more info: I printed the stack from inside the mpl plugin listed above, and it has this:

  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/testing/iptest.py", line 400, in run_iptest
    TestProgram(argv=argv, addplugins=plugins)
  File "/home/fperez/usr/opt/lib/python2.7/site-packages/nose-1.0.0-py2.7.egg/nose/core.py", line 118, in __init__
    **extra_args)
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/home/fperez/usr/opt/lib/python2.7/site-packages/nose-1.0.0-py2.7.egg/nose/core.py", line 135, in parseArgs
    self.config.configure(argv, doc=self.usage())
  File "/home/fperez/usr/opt/lib/python2.7/site-packages/nose-1.0.0-py2.7.egg/nose/config.py", line 275, in configure
    options, args = self._parseArgs(argv, cfg_files)
  File "/home/fperez/usr/opt/lib/python2.7/site-packages/nose-1.0.0-py2.7.egg/nose/config.py", line 262, in _parseArgs
    self.getParser(), self.configSection, file_error=warn_sometimes)
  File "/home/fperez/usr/opt/lib/python2.7/site-packages/nose-1.0.0-py2.7.egg/nose/config.py", line 546, in getParser
    self.plugins.loadPlugins()
  File "/home/fperez/usr/opt/lib/python2.7/site-packages/nose-1.0.0-py2.7.egg/nose/plugins/manager.py", line 404, in loadPlugins
    super(BuiltinPluginManager, self).loadPlugins()
  File "/home/fperez/usr/opt/lib/python2.7/site-packages/nose-1.0.0-py2.7.egg/nose/plugins/manager.py", line 377, in loadPlugins
    plugcls = ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1955, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/pymodules/python2.7/matplotlib/testing/noseclasses.py", line 15, in 
    class KnownFailure(ErrorClassPlugin):

What this means is that pkg_resources is picking up the MPL KnownFailure instead of ours, when 8274f38 has been applied. That commit defers loading of the plugins to much later, and by that point we're at the mercy of pkg_resources finding things by its own mechanisms.

It means there's no way to have our own plugin be the one that goes in, because pkg_resources looks for things on its own, and somehow finds mpl's.

Just tracing this has been a nightmare, and I'm out of time. I'm going to have to revert that commit; I'll try to at least protect against the NOSE_WITH_DOCTEST env variable, but users simply can't have a noserc file that uses with-doctest=1 if they want to run our test suite, at least for now. It's a bummer, but the nose flow is so complicated that I got lost.

And I'd rather keep this limitation (we've had it for ages) than have a broken test suite in the next ubuntu out of the box.

[fperez]

Here's more info... I'm mostly recording what I find out as I go, it might later help us figure this mess out.

nose is loading two plugins unconditionally. If I simply run nosetests -h at the command line, having instrumented nose to open ipython at plugin load time, I see it's loading these two:

nose loadPlugins::
In [1]: ep
Out[1]: EntryPoint.parse('EARLPlugin = rdflib_tools.EARLPlugin:EARLPlugin')

In [2]: 

nose loadPlugins::
In [2]: ep
Out[2]: EntryPoint.parse('KnownFailure = matplotlib.testing.noseclasses:KnownFailure')

So somehow, even when simply asking to print the cmd line help, nose will unconditionally load a matplotlib plugin. This is really, really odd. I have no idea why it's happening, but that's what's tricking us. That KnownFailure plugin, for some reason, then confuses our own test suite.

I'm really quite puzzled here as to what's going on.

[fperez]

Note, the lines above were obtained by embedding IPython into the manager.py file in nose, line 376, right before the ep.load() call. The mpl stack print above shows this.

[fperez]

OK, this is actually an ubuntu bug in how they are packaging matplotlib. I've filed the report here:

https://bugs.launchpad.net/ubuntu/+source/matplotlib/+bug/871176

and I'm discussing with mpl how to go about it in mpl itself. But in the meantime Thomas, a solution is to manually edit the file /usr/share/pyshared/matplotlib-1.0.1.egg-info/entry_points.txt and comment out the whole thing (it's just two lines). That takes care of the problem.

I won't revert Matthew's commit then, as this is best handled by fixing the ubuntu package.

We can close this when they fix it in ubuntu, but I'm leaving it open as a reminder for us...

[takluyver]

Wow, good work tracking that down. I confirm that the workaround of commenting out that file makes the KnownFailures appear as expected again.

[fperez]

Major PITA, I basically burned the whole day on this one, I'm afraid :)

[takluyver]

It got fixed in Ubuntu, but only for the next release (12.04). Should we close this now, or wait for that Ubuntu release?

[fperez]

I'm OK closing it, in the sense that we're done with it. We know what the problem is, this report serves as a reference of what to do, and the solution is on its way. We need very aggressive triage anyway, so I'm closing this now.