nbconvert test failure

[ivanov]

this was originally reported by @gabraganca as ipython/nbconvert#200, but we're not using that repo anymore, so I decided to move the conversation here:

@gabraganca wrote:

Hi,

I have run iptest as @ivanov asked on twitter and the test have thrown me an error:

Test suite completed for system with the following information:
{'codename': 'An Afternoon Hack',
 'commit_hash': 'c5abb22',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/usr/local/lib/python2.7/dist-packages/IPython',
 'ipython_version': '1.0.0-dev',
 'os_name': 'posix',
 'platform': 'Linux-3.7.0-7-generic-x86_64-with-Ubuntu-12.10-quantal',
 'sys_executable': '/usr/bin/python',
 'sys_platform': 'linux2',
 'sys_version': '2.7.3 (default, Apr 10 2013, 05:13:16) \n[GCC 4.7.2]'}

Tools and libraries available at test time:
   curses cython jinja2 matplotlib numpy pexpect pygments qt sphinx sqlite3 tornado wx wx.aui zmq

Tools and libraries NOT available at test time:
   azure oct2py pymongo rpy2

Ran 14 test groups in 159.194s

Status:
ERROR - 1 out of 14 test groups failed.
----------------------------------------
Runner failed: IPython.nbconvert
You may wish to rerun this one individually, with:
/usr/bin/python /usr/local/lib/python2.7/dist-packages/IPython/testing/iptest.py IPython.nbconvert

I then rerun this as asked and got

======================================================================
FAIL: Do post processors work?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/IPython/nbconvert/tests/test_nbconvertapp.py", line 87, in test_post_processor
    assert os.path.isfile('notebook1.pdf')
AssertionError: 
    assert <module 'os' from '/usr/lib/python2.7/os.pyc'>.path.isfile('notebook1.tex')
>>  assert <module 'os' from '/usr/lib/python2.7/os.pyc'>.path.isfile('notebook1.pdf')


----------------------------------------------------------------------
Ran 150 tests in 10.007s

FAILED (failures=1)

I hope that this helps.

[ivanov]

Thanks again for this report. There are two issues here:

1. pdf was not generated and we need to figure out why
2. nbconvert did not return an error code saying that pdf generation did not succeed.

Can you apply this patch, and report on what you get back? I threw in an os.abort() so that the temporary directory which our test creates doesn't get cleaned up and deleted, so you should be able to cd into it and try running pdflatex manually to see what happens.

diff --git a/IPython/nbconvert/tests/test_nbconvertapp.py b/IPython/nbconvert/tests/test_nbconvertapp.py
index 8649fc7..8927234 100644
--- a/IPython/nbconvert/tests/test_nbconvertapp.py
+++ b/IPython/nbconvert/tests/test_nbconvertapp.py
@@ -81,8 +81,11 @@ def test_post_processor(self):
         Do post processors work?
         """
         with self.create_temp_cwd(['notebook1.ipynb']):
-            self.call('nbconvert --log-level=0 --to="latex" notebook1'
+            o,e = self.call('nbconvert --log-level=10 --to="latex" notebook1'
                       ' --post="PDF" --PDFPostProcessor.verbose=True')
+            print e
+            print os.path.abspath('.')
+            os.abort()
             assert os.path.isfile('notebook1.tex')
             assert os.path.isfile('notebook1.pdf')

my output with that patch looks like this:

iptest IPython.nbconvert.tests.test_nbconvertapp:TestNbConvertApp.test_post_processor -vs
Do post processors work? ... [NbConvertApp] Config changed:
[NbConvertApp] {'NbConvertApp': {'export_format': u'latex', 'post_processor_class': u'PDF', 'log_level': 10}, 'PDFPostProcessor': {'verbose': True}}
[NbConvertApp] Using existing profile dir: u'/home/pi/.ipython/profile_default'
[NbConvertApp] Searching path [u'/tmp/tmpr5gWTq', u'/home/pi/.ipython/profile_default'] for config files
[NbConvertApp] Attempting to load config file: ipython_config.py
[NbConvertApp] Loaded config file: /home/pi/.ipython/profile_default/ipython_config.py
[NbConvertApp] Config changed:
[NbConvertApp] {'TerminalIPythonApp': {'display_banner': False}, 'TerminalInteractiveShell': {'banner1': ''}, 'NbConvertApp': {'export_format': u'latex', 'post_processor_class': u'PDF', 'log_level': 10}, 'InteractiveShellApp': {'pylab_import_all': False, 'extensions': ['storemagic', 'memory_profiler', 'django_notebook']}, 'ProfileDir': {}, 'PDFPostProcessor': {'verbose': True}, 'InteractiveShell': {'colors': 'LightBG'}}
[NbConvertApp] Attempting to load config file: ipython_nbconvert_config.py
[NbConvertApp] Config file not found, skipping: ipython_nbconvert_config.py
[NbConvertApp] Converting notebook notebook1.ipynb to latex
[NbConvertApp] Support files will be in notebook1_files/
[NbConvertApp] Applying transform: SVG2PDFTransformer
[NbConvertApp] Applying transform: ExtractOutputTransformer
[NbConvertApp] Applying transform: LatexTransformer
[NbConvertApp] Attempting to load template article.tplx
[NbConvertApp] Attempting to load template article
[NbConvertApp] Attempting to load template latex_article.tplx
[NbConvertApp] Loaded template latex_article.tplx
[NbConvertApp] Making directory ./notebook1_files
[NbConvertApp] Writing 41 bytes to support file ./notebook1_files/notebook1_6_0.text
[NbConvertApp] Writing 47 bytes to support file ./notebook1_files/notebook1_8_0.latex
[NbConvertApp] Writing 43 bytes to support file ./notebook1_files/notebook1_6_0.latex
[NbConvertApp] Writing 272 bytes to support file ./notebook1_files/notebook1_7_0.png
[NbConvertApp] Writing 5 bytes to support file ./notebook1_files/notebook1_7_0.text
[NbConvertApp] Writing 863 bytes to support file ./notebook1_files/notebook1_6_0.png
[NbConvertApp] Writing 53 bytes to support file ./notebook1_files/notebook1_8_0.text
[NbConvertApp] Writing 914 bytes to support file ./notebook1_files/notebook1_8_0.png
[NbConvertApp] Writing 7 bytes to support file ./notebook1_files/notebook1_7_0.latex
[NbConvertApp] Writing 16661 bytes to ./notebook1.tex
[NbConvertApp] Building PDF: `pdflatex ./notebook1.tex`

/tmp/tmpr5gWTq
Aborted

(so in this case, I could have gone on to cd /tmp/tmpr5gWTq and start looking around in there)

[jdfreder]

👂

[jdfreder]

It sounds like it may be a problem with the pdflatex installation. If you have time, and if the output @ivanov requested doesn't seem to provide much more information, try these steps:

1. Copy notebook1.ipynb from /usr/local/lib/python2.7/dist-packages/IPython/nbconvert/tests/files/ into a blank directory that you have write access to.
2. Navigate to that directory and run ipython nbconvert --to latex notebook1.ipynb
3. Run pdflatex notebook1.tex.
4. Check to see if notebook1.pdf exists, if it doesn't copy&paste the output of pdflatex here.

[fperez]

Just ran the test, the error is:

! LaTeX Error: File `/usr/lib/python2.7/dist-packages/sphinx/texinputs/sphinxho
wto.cls' not found.

[jdfreder]

AH! Do you have sphinx on your machine?

[minrk]

sphinx version?

[fperez]

BTW, you should always run pdflatex with --interaction=batchmode to avoid any interactive queries.

[jdfreder]

This brings up a good point, that test should check for sphinx and report something meaningful.

[jdfreder]

you should always run pdflatex with --interaction=batchmode to avoid any interactive queries.

I have it set as a flag in the templates, is that a bad idea?

[minrk]

it might be the howto.cls has moved, or in debian-style installs it may live somewhere weird.

[fperez]

I have Version: 1.1.3+dfsg-7ubuntu2

[minrk]

I have it set as a flag in the templates, is that a bad idea?

what does this mean? the --interaction=batchmode should just go in the PDF post processor command

[fperez]

There's no point in running with interactive queries at least in the test suite, since it's meant to be a completely automated process.

[minrk]

1.1.3 is latest. Does Ubuntu split sphinx into a few different packages, so you might have a partial install?

[jdfreder]

Ok, I'll add the flag to the post-processor. I meant it's here - https://github.com/ipython/ipython/blob/master/IPython/nbconvert/templates/latex/latex_basic.tplx#L4

[fperez]

I have the file: /usr/share/sphinx/texinputs/sphinxhowto.cls. It means we're somehow looking for it in the wrong place.

[fperez]

You may want to play with the $TEXINPUTS env. variable to get a handle on this one...

[minrk]

arg, so debian puts files in a different place from a normal sphinx install. I love it when they do that.

[jdfreder]

It's in the sphinx transformer
https://github.com/ipython/ipython/blob/master/IPython/nbconvert/transformers/sphinx.py#L171

[fperez]

But we should try to fix this before release, I don't want a broken test suite on debian/ubuntu out of the gate...

[fperez]

That path isn't guaranteed to be valid, unfortunately.

[minrk]

I think I found it:

from sphinx import package_dir
tex_inputs = os.path.join(package_dir, "texinputs")

[minrk]

resolves to /usr/share/sphinx on my debian machine, and site-packages/sphinx on my manual install.

[fperez]

Great @minrk! Confirming it works here:

In [1]: from sphinx import package_dir

In [2]: os.path.join(package_dir, "texinputs")
Out[2]: '/usr/share/sphinx/texinputs'

@jdfreder, you have your fix now :)

[fperez]

Go forth and spawn a PR...

[jdfreder]

Nice! Thanks @minrk , do you want to open the PR or should I?

[minrk]

PR #3948

[gabraganca]

You guys are great! As soon the PR is merged, I will run iptest. Thanks.