nbconvert: fixed latex characters not escaped properly in nbconvert #3951

jdfreder · 2013-08-08T02:07:23Z

No description provided.

jdfreder · 2013-08-08T02:08:30Z

@ellisonbg this fixes the output for the IPython.ipynb notebook you pointed me to.

ivanov · 2013-08-08T02:34:03Z

Travis still failing on 2.6 and 2.7

FAIL: escape_latex test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/IPython/testing/_paramtestpy2.py", line 54, in run_parametric
    next(testgen)
  File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/IPython/nbconvert/filters/tests/test_latex.py", line 37, in test_escape_latex
    yield self._try_escape_latex(test[0], test[1])
  File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/IPython/nbconvert/filters/tests/test_latex.py", line 42, in _try_escape_latex
    self.assertEqual(escape_latex(test), result)
AssertionError: 'How are \\{\\textbackslash\\}you doing today?' != 'How are \\textbackslashyou doing today?'
    "'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'" = '%s != %s' % (safe_repr('How are \\{\\textbackslash\\}you doing today?'), safe_repr('How are \\textbackslashyou doing today?'))
    "'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'" = self._formatMessage("'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'", "'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'")
>>  raise self.failureException("'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'")

fperez · 2013-08-08T04:40:42Z

This gives me three failures here (linux, python 2.7):

======================================================================
FAIL: escape_latex test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/testing/_paramtestpy2.py", line 54, in run_parametric
    next(testgen)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/nbconvert/filters/tests/test_latex.py", line 37, in test_escape_latex
    yield self._try_escape_latex(test[0], test[1])
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/nbconvert/filters/tests/test_latex.py", line 42, in _try_escape_latex
    self.assertEqual(escape_latex(test), result)
AssertionError: 'How are \\{\\textbackslash\\}you doing today?' != 'How are \\textbackslashyou doing today?'
    "'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'" = '%s != %s' % (safe_repr('How are \\{\\textbackslash\\}you doing today?'), safe_repr('How are \\textbackslashyou doing today?'))
    "'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'" = self._formatMessage("'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'", "'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'")
>>  raise self.failureException("'How are \\\\{\\\\textbackslash\\\\}you doing today?' != 'How are \\\\textbackslashyou doing today?'")


======================================================================
FAIL: Generate PDFs with graphics if notebooks have spaces in the name?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/nbconvert/tests/test_nbconvertapp.py", line 91, in test_filename_spaces
    assert os.path.isfile('notebook with spaces.pdf')
AssertionError: 
    assert <module 'os' from '/usr/lib/python2.7/os.pyc'>.path.isfile('notebook with spaces.tex')
    assert <module 'os' from '/usr/lib/python2.7/os.pyc'>.path.isdir('notebook with spaces_files')
>>  assert <module 'os' from '/usr/lib/python2.7/os.pyc'>.path.isfile('notebook with spaces.pdf')

======================================================================
FAIL: Do post processors work?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/nbconvert/tests/test_nbconvertapp.py", line 103, in test_post_processor
    assert os.path.isfile('notebook1.pdf')
AssertionError: 
    assert <module 'os' from '/usr/lib/python2.7/os.pyc'>.path.isfile('notebook1.tex')
>>  assert <module 'os' from '/usr/lib/python2.7/os.pyc'>.path.isfile('notebook1.pdf')

----------------------------------------------------------------------

minrk · 2013-08-08T04:43:23Z

We've been looking at it on HipChat, and have a new, simpler regex-free approach that should work better.

ivanov · 2013-08-08T04:48:30Z

IPython/nbconvert/filters/latex.py

@@ -22,7 +22,7 @@
 #Latex substitutions for escaping latex.
 LATEX_SUBS = (
    (re.compile('\033\[[0-9;]+m'),''),  # handle console escapes
-    (re.compile(r'\\'), r'\\textbackslash'),
+    (re.compile(r'\\'), r'{\\textbackslash}'),


removing this change makes all tests pass for me. (only one test was failing with this PR: FAIL: escape_latex test

fperez · 2013-08-08T04:49:21Z

Mmh, now I'm seeing the last two of those three above also on master... Not good.

It seems the issue is the latex tikz package, which I don't have on this box. Do you know if it's necessary?

! LaTeX Error: Filetikz.sty' not found.`

I installed the ubuntu pgf and those two spurious failures are now gone, the other one is still there.

fperez · 2013-08-08T04:49:35Z

Ah @minrk, ok. So this one will get closed eventually, I take?

ivanov · 2013-08-08T04:50:41Z

@jdfreder maybe include the notebook Brian pointed you toward (or the relevant portion of it) into the test suite - not sure what portion of it is causing things to fail

minrk · 2013-08-08T04:57:18Z

Yup, @jdfreder had to run, but I think he plans to get to it tonight or tomorrow morning. Or I can do it in the morning if he doesn't get to it. The new approach is very simple.

minrk · 2013-08-08T05:28:51Z

IPython/nbconvert/filters/latex.py

-    (re.compile(r'"'), r"''"),
-    (re.compile(r'\.\.\.+'), r'\\ldots'),
-)
+# Latex substitutions for escaping latex.


still need to apply the first and last of these, right?

I don't think we can apply the first without a regular expression. We already make that substitution in ansi.strip_ansi , I'll just call that here

The last also needed logic to be able to apply multicharacter replace

Added ... to \ldots escape Escape ansi like before

jdfreder · 2013-08-08T16:30:46Z

@minrk updated

minrk · 2013-08-08T16:35:22Z

I made a PR against your branch last night - should have both fixes.

jdfreder · 2013-08-08T16:36:06Z

Sorry I didn't catch that, I rushed over to here first thing in the morning

jdfreder · 2013-08-08T16:36:15Z

I'll go take a peek

This reverts commit 69adeb1. This allows me to merge min's code to give him credit.

Update PR ipython#3951

jdfreder · 2013-08-08T16:52:00Z

@minrk I reverted my fix and merged yours so you could get credit 😁

minrk · 2013-08-08T16:53:13Z

You didn't have to do that - it was just putting the regex replacements back in that mattered. But thanks :)

jdfreder · 2013-08-08T16:53:44Z

Any reason to do the ansi sub using a regex instead of the ansi.strip_ansi filter/function?

minrk · 2013-08-08T17:21:43Z

IPython/nbconvert/filters/latex.py

-    return_text = text
-    for pattern, replacement in LATEX_SUBS:
-        return_text = pattern.sub(replacement, return_text)
-    return return_text


remove square brackets here

minrk · 2013-08-08T17:21:56Z

IPython/nbconvert/filters/latex.py

-    (re.compile(r'\^'), r'\^{}'),
-    (re.compile(r'"'), r"''"),
+LATEX_RE_SUBS = (
+    (re.compile('\033\[[0-9;]+m'), ''),  # handle console escapes


remove this one - it's redundant

fperez · 2013-08-08T17:22:09Z

IPython/nbconvert/filters/latex.py

-    for pattern, replacement in LATEX_SUBS:
-        return_text = pattern.sub(replacement, return_text)
-    return return_text
+    text = ''.join([LATEX_SUBS.get(c, c) for c in text])


Square brackets not needed, generator comprehension will work by itself.

minrk · 2013-08-08T17:24:42Z

IPython/nbconvert/filters/latex.py


 def escape_latex(text):
    """
-    Escape characters that may conflict with latex.
-
+    Remove ansi codes and escape characters that may conflict with latex.


update this to reflect that it doesn't remove ansi codes

minrk · 2013-08-08T18:07:05Z

IPython/nbconvert/templates/latex/sphinx.tplx

@@ -443,7 +443,7 @@ Note: For best display, use latex syntax highlighting. =))
 ((* macro custom_verbatim(text) -*))
    \begin{alltt}
    ((*- if resources.sphinx.centeroutput *))\begin{center} ((* endif -*))
-((( text | wrap_text(wrap_size) )))
+((( text | wrap_text(wrap_size) | escape_latex )))


this should not be wrapped - it's already in a verbatim environment, wrapping messes that up.

This was added here as a safeguard. If it's not here, the output of a long sequence of character blows outside of the table and messes things up. Are you sure you want me to remove it? This won't allow users to output a long base64 string, a long byte string, etc... Latex isn't smart enough to break long words.

hm, I guess not. It's definitely doing the wrong thing, even with relatively short lines. Since it fixes a real issue, we can look at it more carefully at a later point. Shall I go ahead and merge this now, then?

This reverts commit 0b94505.

jdfreder · 2013-08-08T18:34:47Z

👍

nbconvert: fixed latex characters not escaped properly in nbconvert use simple dict lookup process instead of sequential regular expressions that confuse each other.

Update PR ipython#3951

nbconvert: fixed latex characters not escaped properly in nbconvert use simple dict lookup process instead of sequential regular expressions that confuse each other.

FIXED, latex characters not escaped properly in nbconvert

8b5ade9

ivanov reviewed Aug 8, 2013
View reviewed changes

FIX, don't use regex

8ba1c9a

minrk reviewed Aug 8, 2013
View reviewed changes

minrk and others added 4 commits August 7, 2013 22:41

update LATEX_SUBS table

3fd7401

put back a couple of regexp subs in escape_latex

8bfac62

update text_escape_latex

0b8f154

Escape latex fixes

69adeb1

Added ... to \ldots escape Escape ansi like before

jdfreder added 2 commits August 8, 2013 09:48

Revert "Escape latex fixes"

d644eec

This reverts commit 69adeb1. This allows me to merge min's code to give him credit.

Merge pull request #3 from minrk/pr/3951

548d2ec

Update PR ipython#3951

Better comment

0b94505

minrk reviewed Aug 8, 2013
View reviewed changes

remove sq brackets

1057965

minrk reviewed Aug 8, 2013
View reviewed changes

fperez reviewed Aug 8, 2013
View reviewed changes

Remove escape ansi

577df22

minrk reviewed Aug 8, 2013
View reviewed changes

Fixed latex test to reflect removal of ansi strip

26785e2

minrk reviewed Aug 8, 2013
View reviewed changes

Revert "Better comment"

aae5320

This reverts commit 0b94505.

minrk merged commit 87855e3 into ipython:master Aug 8, 2013

jakobgager mentioned this pull request Aug 8, 2013

Upcoming issues with nbconvert #3603

Closed

7 tasks

jdfreder deleted the custom_verbate_esc_tex branch March 10, 2014 18:42

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

Merge pull request ipython#3 from minrk/pr/3951

11f0810

Update PR ipython#3951

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nbconvert: fixed latex characters not escaped properly in nbconvert #3951

nbconvert: fixed latex characters not escaped properly in nbconvert #3951

jdfreder commented Aug 8, 2013

jdfreder commented Aug 8, 2013

ivanov commented Aug 8, 2013

fperez commented Aug 8, 2013

minrk commented Aug 8, 2013

ivanov Aug 8, 2013

fperez commented Aug 8, 2013

fperez commented Aug 8, 2013

ivanov commented Aug 8, 2013

minrk commented Aug 8, 2013

minrk Aug 8, 2013

jdfreder Aug 8, 2013

jdfreder Aug 8, 2013

jdfreder commented Aug 8, 2013

minrk commented Aug 8, 2013

jdfreder commented Aug 8, 2013

jdfreder commented Aug 8, 2013

jdfreder commented Aug 8, 2013

minrk commented Aug 8, 2013

jdfreder commented Aug 8, 2013

minrk Aug 8, 2013

minrk Aug 8, 2013

fperez Aug 8, 2013

minrk Aug 8, 2013

minrk Aug 8, 2013

jdfreder Aug 8, 2013

minrk Aug 8, 2013

jdfreder Aug 8, 2013

jdfreder commented Aug 8, 2013

nbconvert: fixed latex characters not escaped properly in nbconvert #3951

nbconvert: fixed latex characters not escaped properly in nbconvert #3951

Conversation

jdfreder commented Aug 8, 2013

jdfreder commented Aug 8, 2013

ivanov commented Aug 8, 2013

fperez commented Aug 8, 2013

minrk commented Aug 8, 2013

Choose a reason for hiding this comment

fperez commented Aug 8, 2013

fperez commented Aug 8, 2013

ivanov commented Aug 8, 2013

minrk commented Aug 8, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdfreder commented Aug 8, 2013

minrk commented Aug 8, 2013

jdfreder commented Aug 8, 2013

jdfreder commented Aug 8, 2013

jdfreder commented Aug 8, 2013

minrk commented Aug 8, 2013

jdfreder commented Aug 8, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdfreder commented Aug 8, 2013