Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Jinja Templates #77

Merged
merged 313 commits into from

8 participants

@Carreau
Owner

Re-write Nbconvert using Jinja Template.

It Is much more easier to tweek and to make inheriting template,
without having to code a class that inherit from Converter.py

look at converter/converter_template.py
and
templates/*.tpl

I considere templates/python.tpl almost done, and which fixes some
strangeness of previous implementation that had inconsistencies in blank lines.

usage

./nbconvert2.py profile ipynb_file

generating a diff with current ipynb references:

diff -U3  tests/ipynbref/IntroNumPy.orig.html <(./nbconvert2.py profile ipynb_file)

I would really like the though of @ellisonbg, @fperez.
Pinging @damianavila which might be interested.

Todo :

  • write a preprocessor that extract files
  • make the 'extrator' key configurable
  • cleanup the makefile that generate jinja-tex template from jinja template
  • create a kind of 'profile' which would contain the configuration for each 'format'
  • make the filter/preprocessors 'Configurables'
  • write preprocessor worksheet/cell/output decorator to simplify the task
  • write RST converter
  • write blogger converter
  • document
  • stabilize API
  • check header delimiters : {1: '=', 2: '-', 3: '~', 4: '.', 5: '\'', 6: '`'}
  • write (more) test
  • write separated smaller test
  • replicate remove_math replace_math step in md2latex / other(?) converter

Maybe/Later PR

  • integrate Damián Reveal converter
    • people interested in the same converter but to beamer.
  • add coalesce stream preprocessor
  • write a preprocessor that inline files
  • write Markdown converter
  • add a preprocessor that remove non prioritary display data for conversion speed lower memory footprint

To Fix :

  • output not monospace in output html
  • tex: <obj> rendered as ¡obj¿
@ellisonbg
Owner

You read our minds! A few weeks ago when I visited Fernando, we talked briefly about the growing complexity of the converters. The number of options for each converter was growing quickly and the problem with the old approach is that each time someone wanted to do something slightly different, they had to hack the converter code. We specifically talked about using templates as a way to solve this problem and I am thrilled with this direction. We need to review this particular implementation still, but overall this is a great direction.

@Carreau
Owner

I now have the HTML converter working with only a few new lines differences with the old one.
I will clean and organise templates, I have some ideas about how to add more reusability/configurability to templates.

@ellisonbg
Owner
@ellisonbg
Owner

How does this PR related to your other one where you convert to using Configurable? Can that one be closed? Are both needed?

@ellisonbg
Owner

Sometimes we need the ability to convert a block of markdown to another format, such as reST or LaTeX. What is your thought on how to handle that? Expose simple function to the templates, such as md_to_rest and md_to_latex?

@Carreau
Owner

How does this PR related to your other one where you convert to using Configurable? Can that one be closed? Are both needed?

This one is mostly orthogonal, but I will need configurability at some point, hence I builded on top of the other.

The other one can be merge independently I will rebase after.

Sometimes we need the ability to convert a block of markdown to another format, such as reST or LaTeX. What is your thought on how to handle that? Expose simple function to the templates, such as md_to_rest and md_to_latex?

Yes, and you can chain those 'filter' (in jinja language)

Some transformation will still need to be done before template like extracting/re-inlining files.

Latex will need a special treatment because it rely on curly bracket. There is doc on jinja on how to do it. It will just require to generate a null_tex.tpl from null.tpl by replacing all { and } by ( and ) and the inheritance tree of tex template will be separate from the html one.

@ellisonbg
Owner
@Carreau
Owner

Well, what would be great would be to agree on the name on different block, see if they make sens (null.tpl) and see if the way to extend it is understandable enough.

Having configurable PR merge would be already a step forward.
You can try to have a look at how to reconciliate configurable and docstring inheritor.
I wasn't able to figure out.

Also what would be great would be more ipynb test case to be sure if the new version works ok, I was thinking on bringing back the most popular files of nbviewer as references.

Lastly one big PR that will be a lot of work to convert and might conflict would be Damian's RevealJS one.

So my plan is (more or less in order) :

  • Finish Metaui/CellTollbar PR.
  • Land the slideshow extension for metaUI/celltoolbar
  • Merge Damian's PR.
  • Finish this one by testing wether the framework is ok by re-implementing Damian's one
  • Progressively replace the current converters by the new ones
  • Use the new converter in nbviewer.
  • Rest.

How does that sound ?

@damianavila
@damianavila
@Carreau
Owner

Here, the same... how can I help you with this?

You can try to see if it is usable.
I can send PR agains my PR with new implemented converter, or more test, or more Jinja 'filter'
let's try to build a list of what need to be done.

@damianavila

OK, I will look into it soon...

@Carreau
Owner

Ok, more blocks added.
I thinking os splitting null.tpl into multiple files.

full html seem to be really close in behavior to current version.

Only a handfull of newline differences and a few other small stuff.

To quickly compare you can run

clear ;  diff -U3  tests/ipynbref/00_notebook_tour.orig.html <(./runme.py fullhtml tests/ipynbref/00_notebook_tour.orig.ipynb)  | less -S

On files.

@damianavila

I do not understand why you have differences in the css at the header level... I hope some minor differences in the body because of the templating...but in the css (it is not just a copy from static and added them to the header)? Maybe, I am missing some things (this is the my first encounter with jinja).
Btw, I am trying to understand all this stuff to begin to code some of the converters.

@Carreau
Owner

I do not understand why you have differences in the css at the header level...

The css is extracted from the IPython repo. if there are change in it, it probably come from here.
What are the changes you see ?

I hope some minor differences in the body because of the templating...

Newline if not that easy to get right but that shouldn't matter for HTML

but in the css (it is not just a copy from static and added them to the header)?

Depending on the commit, it use to be a copy and past, now it tries to extract it from different places.

Maybe, I am missing some things (this is the my first encounter with jinja).

Me to, it is very powerful, the doc is well done, it only takes time to fully comprehend to see what is possible.

If you see big differences with the above command line, feel free to past your diff, i'll try to see where it cames from.

@damianavila
@Carreau
Owner

This is what I get different in the css block (I saw another differences
in the body but I understand that those are because of the templating...):

http://pastebin.com/YZq0c5Pj

There are some missing colons, margin, changes from . to #
BTW, the final outcome is the same... so there are subtle changes...

Those are because nbconvert is picking up a old non dev version
of IPython.

Try checking out laster master in your IPython repository.

Coming back... if I wanna to implement a converter the steps would be:

1- Generated the appropiate tpl extended from previous one...

Yes.
You can overwright any {% block %} you want, eventually calling {% super %} if needed.
You should have acces to metadata from template using cell.metadata.[...].

2- Inherit from ConverterTemplate class to create my converter...

Not even sure you need to do that.
To start you can "just" use './runme reveal myfile.ipynb'
If you have a reveal.tpl file it will pick it up.

Something else?

Feel free to submit PR against my branch, and ask questions.

@damianavila
@Carreau
Owner

Things like rm_fake should totally be bart of the base converter.

Depending on what need to be done in process, maybe it also should be optional.
almost nothing from the ConverterTemplate is final now, so why you not try to do it,
ping me and we discuss what should inherited and what should not ?

@MichaelMartinez

@Carreau, how should we pull/clone your changes to test? I would like to make this work for getpelican/pelican

@Carreau
Owner

@MichaelMartinez
Right now, pull the branch as is.
Create a template in template folder.
Then

./runme.py [templateNameWithoutDotTPL] [path to ipynb file convert]

Nothing is really final now don't heavily rely on name etc...

Feel free to send me modification or ask if you need.

Short view of where I want to go, not sure if it would work.

in template folder

  • null.tpl
  • python
    • master.tpl
    • input.tpl
    • output.tpl
    • header_cell.tpl
    • ...
  • rst
    • master.tpl
    • input.tpl
    • output.tpl
    • header_cell.tpl
    • ...
  • html
    • master.tpl
    • input.tpl
    • output.tpl
    • header_cell.tpl
    • ...

each master.tpl inherit from null.tpl and can 'include' all the other tpl it wishes.
So that you can 'mix and match' by just creating a template file that include what you need.
People could then just write block (like input_no_highlight.tpl, or fancy_in_prompt)

right now I made a small test and Jinja throw undefined variable, I'm still wondering why and didn't had time to look at the reason.

@damianavila
@Carreau
Owner
@damianavila
@Carreau
Owner

I would have loop through the cell, potentially happending helper metadata ( that exist only for the time of the conversion) like

  • slide start
  • slide end
  • fragment_start
  • fragment_end
# if cell n+1 is a slide cell, then we close the section tag here.
for i,cell in enumerate(cells):
     if cells[i+1].metadata.slideshow.type in ['slide']:
           cell.metadata.slideshow.helper = 'slide_end'

Then in the template you can overwite any_cell

{% block any_cell %}
if slide_start
    < section >
{{super()}}

if slide_end
  </section>
{% endblock %)

second way

You potentially can look at loop helper of jinja

as actually you can append </section><section> before every 'slide' cell except the first one and the last one (note the closing tag first) and treat separately each the first and last iteration :

if is_slide :
   is loop.first?
        <section>
   else
        </section><section>
{{ super() }}

if loop.last ?
    </section>

Still you seem to be the one that know reveal, so I might be missing something.

@damianavila
@Carreau
Owner

I will play with this recommendations when I get back home and let you
know...

any time :-)

@Carreau
Owner

@damianavila

I've slightly change the structure of the converter.
This should help you to preprocess the ipynb file for your converter.

@damianavila

After fighting with templates and a lot of thinking (and finally a simple solution)... all the implementation is templated, I mean, all the html is written by templates :-)
I have preprocessed the cells to get some new metadata to help me delimit the slides - as you as pointed out before... I will make some changes for the slightly changes you have commited yesterday... but tomorrow, now, time to go to bed...

@Carreau
Owner

Ok, this should now works also for Tex.
And the ./runme.py now also accepts --Class.trait=value flags.

@Carreau
Owner

This does merge cleanly (fast forward even) no idea why github says otherwise.

@jakobgager

Great to hear, you have implemented tex support. I will have a look at the new system and maybe I get an idea how this jinja stuff works :)

@Carreau
Owner

@jakobgager tex support is mainly changing jinja delimiters so the compiler is not confused by {and }.
This is still really early stage in dev, but I'll be happy to get feedback.
What is missing now (especially for latex, but for html also) is a way to extract graphic files from notebook as separate files. But except that finishing latex template is just a question of time.

@jakobgager

I had a quick look at the new code yesterday and may give some comments (basically for ipynb2latex conversion):

  • I guess the extract graphics stuff can be take from the "old" latex converter.
  • Is there a way to change the display_priority for the individual converters? -> I strongly suggest to put the latex output before the image type outputs, otherwise you would get images in nearly all of the cases!
  • Do you intend to keep the latex_base.tplx converter as it is and create an additional file like with html to handle the treatment of the individual output types?(to be consistent you probably should rename the html files to html_base and html_full)
  • As I posted somewhere else (#76), is it really necessary to distinguish between pyout and display style output? (both can contain the same output types!)
@Carreau
Owner
This was referenced
@Carreau
Owner

We start to have flags from the command line :

see --help-all flag.

ConverterTemplate options
-------------------------
--ConverterTemplate.display_data_priority=<List>
    Default: ['html', 'pdf', 'svg', 'latex', 'png', 'jpg', 'jpeg', 'text']
    [wrong help due to copy/past]
--ConverterTemplate.extract_figures=<Bool>
    Default: False
    wether to remove figure data from ipynb and store them in auxiliary
    dictionary
@Carreau
Owner

Pinging readers here, few lines, lots of changes.

Instead of 'format' you can now use a 'profile' name on the command line.

look at profile/*.nbcv

## file latex.nbcv
c = get_config()

c.ConverterTemplate.extract_figures=False
c.ConverterTemplate.template_file='latex_base'
c.ConverterTemplate.tex_environement=True

So basically, you put all your option in this, and then issue a
./runme.py [profile name] [file to treat] it does what you expect. And like IPython if you pass extra flag on the command line they will overwrite the one of the profile.

Does of course works if you inherit from ConverterTemplate, with the name of your custom Class as prefix, as long as the properties are Traits.

@jakobgager

Sounds great! Hope i find some time this week to test the new profiles.

@Carreau
Owner

One can now write ipynb preprocessors that inherit from configurable and hence get their configuration from the selected profile.

@Carreau
Owner

Add an example on how to write external files like for Latex.

@Carreau
Owner

some fixes for Latex the files should now be extracted in a folder instead as in the same dir as the ipynb, and the Extractig preferences can also be set, so that for example you extract svg preferably and don't try to extract text as it is useless.

BTW, to write the converted file, use the following :

./runme.py latex_base tests/ipynbref/Gun_Data.orig.ipynb --write=True --no-stdout
@Carreau
Owner

Converter should now be mostly stateless (unless you play some weird stuff with jinja filters)
but you can now instanciate a converter and feed it many file as you like in a for loop.

@Carreau
Owner

Rst converter done.

I'm looking at the markdown one, I know that markdown is a superset of html, but the current converter look weird.

I'm wondering if

Let's make sure we have pylab again, in case we have restarted the kernel due to the crash demo above

<div class="highlight"><pre><span class="o">%</span><span class="k">pylab</span> <span class="n">inline</span>
</pre></div>



    Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
    For more information, type 'help(pylab)'.

Is really what should be spit by the current converter.

I would have expected

Let's make sure we have pylab again, in case we have restarted the kernel due to the crash demo above

    %pylab inline

    Welcome to pylab, a matplotlib-based Python environment [backend:module://IPython.zmq.pylab.backend_inline].
    For more information, type 'help(pylab)'.

With maybe some delimiters...

Otherwise I didn't had any comments from core dev in ~3weeks on this PR..

@Carreau
Owner

We now have reveal converter !!! (ok, a few css bug, but the harder part is done), I was hoping for a less than 100 lines templates files, but actually 150... not too bad.

Update :
This WILL BREAK on current master of IPython since the introduction of bootstrap.
You have to use a pre- ipython/ipython#2708 for full_html and jinja converter to work.

@damianavila

Hey, Mathias.. you are fast... I am finishing my implementation (different from your logic, less code, more logic in the template), but with your jinja template version from some weeks ago... I have to do an update for the last changes, but i do not know if it makes sense to do it now. What do you think?
BTW, I will try to test this one during the weekend...

@jdfreder
Collaborator

Hi @Carreau,
So I've been playing around with the new converter and the old converter. I love that the new converter uses Jinja as an output template system. Great work! I want to start working on a Latex converter that is similar to to Sphinx's latex exporter (like we talked about in the 'Introductions' email.) Should I write a Jinja template that inherits from the "latex_base.tplx" template and go from there? Or should I work on writing my own template from scratch and worry about inheriting from the base template later?

@Carreau
Owner

Hey, Mathias.. you are fast... I am finishing my implementation (different from your logic, less code, more logic in the template), but with your jinja template version from some weeks ago... I have to do an update for the last changes, but i do not know if it makes sense to do it now. What do you think?

You can stil point me to the changes if even they are Ugly I'm curious to see how you did it. My version is still a hack also so we may merge the 2 versions together.

@Carreau
Owner

Great work! I want to start working on a Latex converter that is similar to to Sphinx's latex exporter (like we talked about in the 'Introductions' email.) Should I write a Jinja template that inherits from the "latex_base.tplx" template and go from there? Or should I work on writing my own template from scratch and worry about inheriting from the base template later?

Feel free to modifie latex_base , If you can't do something by inheriting from it that's probably because it has been badly written.

The way I see it, latex_base should provide markup/environment in the right place that are defined as no-op in the header block so you just have to overwrite the environment definition if you inherit. Feel free to open a PR as soon as you start working on it, and don't be afraid to ask question.

templates/reveal.tpl
((14 lines not shown))
+{% endif %}
+
+{% if cell.metadata.slideshow.open_section %}
+<section>
+{%- endif %}
+{% if cell.metadata.slideshow.open_subsection %}
+<section>
+{%- endif %}
+{% if cell.metadata.slideshow.open_fragment %}
+<div class='fragment'>
+{% endif %}
+{% if cell.metadata.slideshow.slide_type in ['notes'] %}
+<aside class="notes">
+{{ super() }}
+</aside>
+{% elif cell.metadata.slideshow.slide_tyle not in ['skip'] %}

I had not time to test it yet, but reading it... you have a typo here... cell.metadata.slideshow.slide_type instead of slide_tyle...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
converters/transformers.py
((194 lines not shown))
+
+ def maybe_close_fragment(self):
+ """return True is already open, false otherwise
+ and change state to close
+ """
+ if self.fragment_open :
+ self.fragment_open = False
+ return True
+ else :
+ return False
+
+ def cell_transform(self, cell, other,count):
+ ctype = cell.metadata.get('slideshow',{}).get('slide_type',None)
+ if ctype is None :
+ cell.metadata.slideshow = {}
+ cell.metadata.slideshow['slide_type'] = None

Again, just reading... but you have to include the slide_type = '-' ... it appears when you have marked a cell with other label, saved the notebook, and later change again this cells to a "none" ('-') cell. I forgot to include this kind of cell in reveal.orig.ipynb, but you can get it easily just make one cell of this reference to label '-' and you will see this string in the metadata slide_type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@Carreau
Owner

thanks @damianavila, fixed.

@fperez fperez was assigned
@Carreau
Owner

Assigning @fperez :-)

@Carreau
Owner

Ho ! don't even need to rebase !
I'll see if I can mer damian work on jinja reveal !

@damianavila

OK, this is great! Now, we only need to finish with the other PR to have the reveal converter fully functional...
But, coming back to this PR, I saw from your commits that you change the css to a new one. Is this css derived from the work with less? I need to use the current master to see it?

@Carreau
Owner

But, coming back to this PR, I saw from your commits that you change the css to a new one. Is this css derived from the work with less? I need to use the current master to see it?

Yes it is still work in progress, should look the same in ipython notebook but we will slowly moving almost everything to less.

@damianavila

OK, I am bother you with some PRs over this branch ;-)

Damián.

Carreau and others added some commits
@Carreau Carreau Copy IPython-configurable with the same hierarchy 43d09e2
@Carreau Carreau copy IPython application and loader a32b1b7
@Carreau Carreau nbconvert is now configurable app 23aa4a2
@Carreau Carreau working config 958845e
@Carreau Carreau add extract_figures configurable.
remove old useless configurable
2fcecca
@Carreau Carreau Allow to build a converter without input file 48f7d4d
@Carreau Carreau fix tests 1f16844
@Carreau Carreau missing comma 753203f
@Carreau Carreau fix after exclude 983c059
@Carreau Carreau downgrade docutils in requiremetns.txt 8f42d65
@Carreau Carreau regenerate test case with newest version of ipython 8dc6f15
@Carreau Carreau skip html on travis fbcb4a3
@Carreau Carreau starting templates f5f3603
@Carreau Carreau less newline f9a642a
@Carreau Carreau null template 6ba5bac
@Carreau Carreau handle unicode output 42ef8ed
@Carreau Carreau onlynewline eeedcb2
@Carreau Carreau inheritance example and doc f2a9360
@Carreau Carreau add any-cell block 5c346ed
@Carreau Carreau deal with worksheets 61e2016
@Carreau Carreau basic test for converter template 9618792
@Carreau Carreau ensure base templates returns 0 length file as they should fb05caf
@Carreau Carreau require jinja, merge base templates 16189ca
@Carreau Carreau fix utf8 0ccbc94
@Carreau Carreau working on html ac8105b
@Carreau Carreau some other fixes 1e22853
@Carreau Carreau svgoutput 25b9878
@Carreau Carreau read css from ipython dir 4e94de8
@Carreau Carreau add other text example 8a3a60d
@Carreau Carreau add other text example c56ac07
@Carreau Carreau inline css from external files f7c114b
@Carreau Carreau remove blank lines 7544367
@Carreau Carreau modularize inlining 722aa10
@Carreau Carreau try to play with data display priority c342e6b
@Carreau Carreau fix some stream/pyout/pyerr f4df1de
@Carreau Carreau Start to think on api
converter now return a tuple:
the converted file and potential other resources in a dict
378c10b
@Carreau Carreau remove and clean code c19c6ee
@Carreau Carreau add possibility to preprocess ipynb files 086c520
@Carreau Carreau preprocessing and namespace fix
use preprocessing with current transform,
and ability to access notebook namespace from template
14b695b
@Carreau Carreau bring back python.tpl efa2543
@Carreau Carreau should fix tests 6e68b73
@Carreau Carreau add docstring 8d59a31
@Carreau Carreau prepare ability to convert template for Tex 7ed2329
@Carreau Carreau multiple env a449ac2
@Carreau Carreau start tex template 44388d7
@Carreau Carreau lots of modification for latex 8e698a1
@Carreau Carreau rename runme2 to runme c336f65
@Carreau Carreau test_file extractor ff87077
@Carreau Carreau some improvement 5acd1ee
@Carreau Carreau flag for extracting figure 5cc9c4c
@Carreau Carreau Start to organize template folder be33d1e
@Carreau Carreau add readme in skeleton be73d44
@Carreau Carreau use configuration file to do nice stuff a4b9513
@Carreau Carreau pretransformer configurables 656282c
@Carreau Carreau add python profile 1e75df8
@Carreau Carreau move transformer in separate file 5c9515f
@Carreau Carreau mofe filter in separate file f456e76
@Carreau Carreau cleaning 6213275
@Carreau Carreau create configurable preprocessors df9dfe4
@Carreau Carreau add stdout flag and description in help 7086bcb
@Carreau Carreau regenerate templates 2592ed9
@Carreau Carreau fix soem figure extraction logic 036e7d2
@Carreau Carreau actually don't need to extract display text... 4dd0880
@Carreau Carreau unified figure folder 6a58eae
@Carreau Carreau add blogger html profile 820f9c1
@Carreau Carreau move tex skeleton d587537
@Carreau Carreau inherit blogger 53fe59b
@Carreau Carreau run some more test
should yield them for more grained result
d0ee335
@Carreau Carreau yield tests f35fd12
@Carreau Carreau add display_priority_tex in tests 3ddfea5
@Carreau Carreau partial tex fixes e4965af
@Carreau Carreau fix fileext and html multidisplay 1829793
@Carreau Carreau html title 011469c
@Carreau Carreau put figures in subkey e2f7fa1
@Carreau Carreau stateless converter 933bc1d
@Carreau Carreau start rst converter 5bb3aa8
@Carreau Carreau create rst converter and fix some bugs e92c5c8
@Carreau Carreau fix data priority rst 59a1632
@Carreau Carreau reveal converter 11e4857
@Carreau Carreau extract css from ipython dir bfc9ce4
@Carreau Carreau remove debug statement 2364deb
@Carreau Carreau fix typo db847f0
@Carreau Carreau create an activatable transformer classes
those are no-op by default
5122f41
@Carreau Carreau make ExtracfigureTransformer Activable 01f5693
@Carreau Carreau allow configurable filters 0841eab
@Carreau Carreau rename runme into nbconvert2 c0e95e2
@Carreau Carreau remove commented code 05b6de6
@Carreau Carreau start writting more references test 446f347
@Carreau Carreau move html header out d93b423
@Carreau Carreau pylint c54e064
@Carreau Carreau add a coalesce stream transformer 6cd2ae4
@Carreau Carreau start to modify fig key configurable 1ae07e7
@Carreau Carreau fix test for coalesce stream, and fig key config d95cbaf
@Carreau Carreau cleanup entry point d1755c2
@damianavila damianavila A new implementation of reveal converter with jinja templates. da06655
@damianavila damianavila Fixed mathjax inline and some not implemented notice. a14dc4a
@damianavila damianavila Fixed path to notes.js because reveal now detect the reveal path to n…
…otes html.
686fa09
@Carreau Carreau document 4edfdbe
@Carreau Carreau more doc and future API dc9567a
@Carreau Carreau docs 822ebc8
@Carreau Carreau share some global config state c8311da
@Carreau Carreau pylint 5304c73
@Carreau Carreau all header leven in rst e8d325c
@Carreau Carreau deprecation message in old nbconvert.py 365068a
@Carreau Carreau cleanup profile fixup rst 94361f7
@Carreau Carreau use new css with old nbconvert html 1ce1776
@Carreau Carreau regenerate those damned exampel to work with HEAD ff99a3b
@Carreau
Owner

Rebased. Seem to work.

@damianavila

@Carreau, I see some discrepancies (the last commits regarding new css support are not here)... the PR #69 was merged without problem into nbconvert... but when I see this branch, it seems rebased to a previous state of that PR (4 days ago instead yesterday)... can it be possible?
I think you will have to rebase against the master again to get the last changes incorporated...

Cheers

Damián.

@Carreau
Owner

Thanks, I'll investigate, probably not today though.

Carreau added some commits
@Carreau Carreau Merge 'damianavila/reveal_css_fix' into templates
again

Conflicts:
	converters/transformers.py
	nbconvert.py
	templates/fullhtml.tpl
	templates/reveal.tpl
	tests/ipynbref/reveal.orig_slides.html
d76bca5
@Carreau Carreau forgott conflict 7026c28
@Carreau Carreau regen slide references as merging by hand was too hard 6bc2bf4
@Carreau Carreau regen references templates 63ef506
@Carreau
Owner

And markdown converter done.
Pinging @fperez

@Carreau Carreau referenced this pull request in CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
Open

ipynb -> tex converter #5

jdfreder and others added some commits
@jdfreder jdfreder Fixed an out of date comment. 3c31d30
@jdfreder jdfreder Fixed comment typo. b1be609
@jdfreder jdfreder Added basic Sphinx template.
This template does not really inherit anything from the
base yet.  I'm no where near done with this.  It's just
a starting point.
4ad153a
@jdfreder jdfreder Added document level description and cleaned template up.
Note: Still need to fill in doc variables to load the
Sphinx format.
4d2a445
@jdfreder jdfreder Made significant improvements to the Sphinx-Latex export
template.

The inheritance was cleaned up.  Currently no content is
rendered (that's a task for later.)  The important thing
is that the template can be built through the Jinja 2
structure and converted to a PDF using pdflatex.
Unfortunately it seems that Sphinx requires texlive-full
which is a huge dependency.
ee813da
@jdfreder jdfreder Updated README to reflect package requirements. 1f04b02
@jdfreder jdfreder Fixed formatting error on README. 3647086
@jdfreder jdfreder Removed unneccessary parameters that are now in
converters/base.py.
5f76140
@jdfreder jdfreder Got pdflatex to work on this empty sphinx styled template. 18db3d0
@jdfreder jdfreder Renamed latex_sphinx to latex_sphinx_base 364adc3
@jdfreder jdfreder Split Sphinx template into sub-style templates. e1a1bc0
@jdfreder jdfreder Fixed profiles so they inherit from eachother. 03d7795
Jonathan Frederic Reverted changes to original sphinx template 761eaa3
Jonathan Frederic Sphinx templates draw underlaying data from Sphinx classes. 9000e51
@jdfreder jdfreder Got both of the main template formats from sphinx rendering.
Todo:
 [ ] Fill with NB data
 [ ] Move latex around and make use of latex base
b09ea97
@jdfreder jdfreder Reverted change accidently commited. b4c6885
@jdfreder jdfreder Removed full path names for latex dependencies, relative
used now.
62af968
@jdfreder jdfreder Process header cells.
Removed unnecessary tabs.
61b3b53
@jdfreder jdfreder Added rawcell formatting
Fixed missing new lines between blocks (prevent Jinja block trim)
Removed LaTeX comments
f463fa5
@jdfreder jdfreder Moved latex comments in to Jinja blocks so they show in the
    compile file.
Added Jinja macro to make sure input text does not screw up
    latex.
Added simple text output for unkown cells.
a73deec
@jdfreder jdfreder Updated installation instructions to install
working markdown2latex.
1a1f776
@jdfreder jdfreder Fixed markdown 6a40668
@jdfreder jdfreder README Markdown fixes...README.rst 763597d
@jdfreder jdfreder Update README.rst
Finally fixed readme formatting...
e40fa12
@jdfreder jdfreder Changed readme to correct dependency. b535ce1
@jdfreder jdfreder Enabled markdown blocks. 4911964
@jdfreder jdfreder Added instructions on how to test the new template. 30acf25
@jdfreder jdfreder Added markdown support to headers (temporary) f4b421b
@jdfreder jdfreder Update README.rst
Fixed restructured text.
51a8aae
@jdfreder jdfreder Moved header definitions into individual sphinx doc types.
Fixed raw text cell so it doesn't apply any formatting
  to it's output.
1bef040
@damianavila damianavila Changed name option from slider to reveal 387781e
@damianavila damianavila Added test_reveal in test_references and fixed test_simple. df644ee
@damianavila damianavila Skipped cells are now hided instead of deleted. 3dcc585
@Carreau Carreau working config 1143ce9
@damianavila damianavila A new implementation of reveal converter with jinja templates. 931104e
@Carreau Carreau working config d45446d
@jdfreder jdfreder Added datetime access to Jinja b0bae6c
@jdfreder jdfreder Fixed some weird things that happened after my first
git merge conflict experience.
75283d7
@jdfreder jdfreder Fixed typo 0ac2fba
@jdfreder jdfreder Fixed typo 0d63fba
@jdfreder jdfreder Added Sphinx transformer.
TODO: Investigate alternative to hardcoding...
de2ef64
@jdfreder jdfreder Fixed: getattr does not work on dictionaries.
Instead use dict.get(key,default)
Method declaration uses standard dict, not traitlet dict.
c959c7c
@jdfreder jdfreder Enforce strict Sphinx style on headers and doc. attrib.
Temporarly copy-and-pasted standard latex declares to
	make output work.
fdf5273
@jdfreder jdfreder Changed SphinxTransformer to inherit
from ActivatableTransformer
f1c1cf6
@jdfreder jdfreder Added code to fix the math $ in the notebook so it
works with markdown2latex.
328a2ce
@jdfreder jdfreder Updated readme to reflect changes in procedure. 26da26e
@jdfreder jdfreder PEP8 866dcf3
@jdfreder jdfreder Unecessary re-coding, inherits from base now. 54dd348
@jdfreder jdfreder Added pygments syntax highlighting.
Added an HR, need get_lines to bind the HR to the first n
	lines of text
9677a57
@jdfreder jdfreder Added HR, pygments, made HR attach to first n lines. 819733b
@Carreau Carreau fix issues on rebase 467779f
@jdfreder jdfreder Merge pull request #1 from Carreau/fix-9
fix issues on rebase
18d9672
@Carreau Carreau fix to readme 7e14f62
@jdfreder jdfreder Fixed missing latex_transformer (rebase issue)
Added wrapping to template to prevent overflow.
cae0fe9
@jdfreder jdfreder Merge pull request #2 from Carreau/fix-9
Backport to readme
2015c19
@Carreau Carreau Merge pull request #9 from jdfreder/master
Sphinx Latex Jinja Template
a8af33a
@Carreau Carreau Merge branch master templates
Conflicts:
	README.rst
3622208
@Carreau Carreau fix enable ebc98ee
@Carreau Carreau fix forgot display priority de5191b
@Carreau Carreau Do now wrap before converting md to latex
It can append in the middle of code which breaks this.
b500826
@Carreau
Owner

\lt does not work in math mode in LaTeX, but does with mathjax.

Same with \cases{} which should be replaced with begin/end{cases}

@jdfreder
Collaborator

Ah, thanks for point this out. Where does \lt and \cases{} show up in the Sphinx stuff?

@jdfreder
Collaborator

Whoops, nvm, I see we are on the main nbconvert repo. (I thought this was my repo)

@Carreau
Owner

No problem. this is still a PR of mine so not really the main repo.

This are the issue I encounter while programming that I just throw here not to forget.

@Carreau
Owner

Diverse fix/things to improve (mostly detected on sphinx manual)

  • extract author/info from notebook metadata (define a structure for that)
  • issue with bold font and math (**$Z$ is discrete**) space removed after second $
  • wrong coloration after text wrap see here
  • output section even if no output
  • some antislash stripped $$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$(frac)
  • Four underscore are hline in markdown, try not to escape....
@jdfreder
Collaborator

The coloration problem (3rd on your list) also raises another question. Should we make note of when lines are wrapped? Marbe a character like this - http://www.fileformat.info/info/unicode/char/21b5/index.htm . I asked @ellisonbg on HipChat, but I think he is still away from keyboard.

@Carreau
Owner

I'm not even sure we can do smth for coloration like that. it is probably in issue upstream.
Personally I would leave the wrapping without anything.

@jdfreder
Collaborator

Well, I was looking into this problem earlier, and I found that Latex really is of no help here. There isn't anything that we can do to force the text to wrap nicely when formatted. Unfortunately, this means you are right, it's upstream of the Latex; however, we aren't totally helpless. We could add some logic to wrap the Latex after it's been applied. Some sort of latex - intelligent wrap... Just a thought.

But we definitely cannot wrap before passing into Pygments, it won't highlight properly. A perfect example is a long comment line that gets wrapped, the preceding # won't get noticed.

@jdfreder

Good point, this may screw up the md, right?

Owner

Yes, wrapping was screwing up the '4 space indented' code as the new line was not indented and appearing as normal text.

@jdfreder

Why did you remove this block? It won't be prefixed with "Output" anymore, nor will it be formatted, intentional?

Owner

Hum, I probably missed the output part.
I replaced, because pyout do not alway have only text. it can have png, jpg and the rest.
I'll see if I can fix it.

Collaborator

Ok, in the PR I just started, I merged this change in anyways- so it should be easier for you to merge.

Collaborator

I'm working on a new branch where I'm cleaning and commenting some stuff (you'll see a PR soon). This comment may help explain why there is this line splitting logic -

In order to make sure that the input/output header follows the code it
preceeds, we have to use a minipage environment. This causes problems
for large blocks of input and output. If there is a large input/output
block, we don't want to minipage the whole thing since it will break
the line wrapping. The solution is to split the input/output line by
line before we minipage which allows us to minipage the first X lines
preceeding the input/output bar. That way, a select amount of lines
force the input/output bar to follow it around.

Owner

But it does not fix the issue when output is not text...but it's a good idea to add this comment.

Collaborator

If you look at the specific non-text blocks, ie display_data, you'll see I don't do any splitting right now, so page breaks are disabled...

@Carreau
Owner

You can also fix upstream to ...user, showing a limit line on codemirror : "be carefull, do not go beyond this line" something like 80 characters...

@Carreau Carreau referenced this pull request in jupyter/nbviewer
Closed

Nbviewer2 #40

@jdfreder
Collaborator

good catch

@Carreau Carreau referenced this pull request
Closed

Embedded images in reveal #104

@minrk
Owner

I've done a read-through, and nothing stands out to me at this point.
This PR is so big that I think what's probably best is to merg it as-is,
and then start looking at smaller work from here on out.

It already seems preferable to current master in pretty much every way.

The only thing I can think of left to be answered is the failing travis tests - @Carreau do you know what's up with that? Are the tests not meant to be running on travis, or is it a real failure?

@Carreau
Owner

Some of the faillure are real faillure, working on if.

@jdfreder there seem to be an issue in remove math:

In[3] from converters import latex_transformer
    lt = latex_transformer.LatexTransformer()
    lt.enabled = True
    lt.remove_math_space('$e$')
Out[3]: '$$'

lt.remove_math_space('$e$abc') -> '$e$bc'

@jdfreder
Collaborator

Yikes, thanks for the heads up.
I'll check it out and report back.
It may have been fixed, in my master branch, I have made a lot of changes since the last merge.

@Carreau
Owner

Ok, I fixed the issues I saw, added test, regenerate the templates and quickly checked the diff of the different references templates.

so, locally on my machine, commit b84afae:

$ nosetests
.........................../Users/bussonniermatthias/nbconvert/tests/tutorial.rst.ref:27: (ERROR/3) Unknown interpreted text role "ref".
/Users/bussonniermatthias/nbconvert/tests/tutorial.rst.ref:48: (ERROR/3) Unknown interpreted text role "ref".
/Users/bussonniermatthias/nbconvert/tests/tutorial.rst.ref:84: (ERROR/3) Unknown interpreted text role "ref".
/Users/bussonniermatthias/nbconvert/tests/tutorial.rst.ref:127: (ERROR/3) Unknown interpreted text role "ref".
/Users/bussonniermatthias/nbconvert/tests/tutorial.rst.ref:157: (ERROR/3) Unknown interpreted text role "ref".
/Users/bussonniermatthias/nbconvert/tests/tutorial.rst.ref:157: (ERROR/3) Unknown interpreted text role "ref".
/Users/bussonniermatthias/nbconvert/tests/tutorial.rst.ref:190: (ERROR/3) Unknown interpreted text role "ref".
............................
----------------------------------------------------------------------
Ran 55 tests in 10.800s

OK
@jakobgager

May I suggest to not automatically wrap all latex within an equation environment, but rather check if the latex code is placed within dollar tags (then use equation environment) or if the environment to be used is given within the latex code (then simply use this one).
This would allow to use other environments as well.
Jakob

Owner

Sure, right now, I'm mostly trying to get this PR landed with the test that pass. Then we could tweak it progressively allow customisation.

If you wish to write a custom filter that detect the environment and wrap if there is none, I'll be happy to merge it.

@Carreau
Owner

Any one to give a last green light to merge or even press the big grey button ?

@jdfreder
Collaborator
@Carreau
Owner

@minrk can you try again ?

@minrk
Owner

docutils.math moved to docutils.utils.math. after fixing that, there's just one failure:

======================================================================
FAIL: test_templates.test_evens('rst', 'tests/ipynbref/IntroNumPy.orig.ipynb', 'tests/template_ref/IntroNumPy_orig.rst')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/minrk/Library/Python/2.7/lib/python/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/minrk/dev/ip/nbconvert/tests/test_templates.py", line 54, in test_profile
    compfiles(output,f.read())
  File "/Users/minrk/dev/ip/nbconvert/tests/test_templates.py", line 17, in compfiles
    cleanfile(strb))
AssertionError: Lists differ: [u'An Introduction to the Scie... != [u'An Introduction to the Scie...

First differing element 130:
integrated between :math:`a=1` and :math:`b=9`.
integrated between :math:`a=1` and :math:`b=9`\ .

Diff is 71038 characters long. Set self.maxDiff to None to see it.
@Carreau
Owner

docutils.math stuff should be fixed.

Last test depends on pandoc version version. I downgraded mine to be as close as possible from Travis one.
If you achieve to build references test that also passes on travis, that would be great.
Personally I gave up trying to have test passing on multiple computer.

@minrk
Owner

Personally I gave up trying to have test passing on multiple computer.

Tests that only pass on one machine aren't exactly tests.

But anyway, I think we are good enough at this point, since nbconvert has never really been tested, we can't call it a regression. I'm going to merge now, so we have a more sensible master.

@minrk minrk merged commit fae0063 into from
@jdfreder
Collaborator

This change broke my ability to export Jose's work. Why are we not writing the images as binary data any more?

Owner
Owner

more precisely, data of resources can be svg/png/jpg/text soem of them should be base64 decoded and wb, other not as they are text... I didn't thought of that.

@Carreau
Owner

Great ! @jdfreder I missed you message about your PR. Yes, you can open it against master now, all IPython core dev will have access to it now.

@damianavila

Yes, finally landed... congratulations for your great work @Carreau... and all who contribute with code and reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.