Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.tex file has no document content #27

Closed
angeloklin opened this issue Jul 19, 2017 · 15 comments
Closed

.tex file has no document content #27

angeloklin opened this issue Jul 19, 2017 · 15 comments

Comments

@angeloklin
Copy link

Hello Chris,
Thanks for your help with the installation issue. All good now.

I am still having problems with content generation.
(1) With a simple notebook as described below, I only get the markdown. If using the -f latex_standard_article it works just aside some very minor formatting issues.
(2) With a notebook of mine, it is not generating any \document content in the .tex file, only header/config details.

There is something not happening on the nbconvert phase.

Please let me know if you can see between the lines or if you need any specific information from my environment.

Thanks in advance.

1

Untitled.tex

. . .
	\begin{document}
\section{Chart 2}\label{chart-2}
	\end{document}

Untitled.ipynb

In [1] from ipypublish.scripts.ipynb_latex_setup import *
In [2] %matplotlib inline
       import matplotlib.pyplot as plt
In [3] for i in range(10):
       print(i)
0
1
2
3
4
5
6
7
8
9
       Chart 2.    # <- Markdown
In [4] plt.plot([1,2,3], [4,2,7])
       plt.show()
(actual chart)

2 $ nbpublish -pdf --pdf-debug notebook.ipynb

notebook.tex

. . .
	\begin{document}
	\end{document}

nppub.log

INFO:main:started ipypublish at Wed Jul 19 10:52:36 2017
INFO:main:logging to: /Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted/20-EDA-tbl_FA_360_Full.nbpub.log
INFO:main:running for ipynb(s) at: 20-EDA-tbl_FA_360_Full.ipynb
INFO:main:with conversion: latex_ipypublish_main
INFO:nbmerge:Reading notebook
INFO:main:getting output format from exporter plugin
INFO:nbexport:running nbconvert
INFO:latex_doc:resolving external file paths in latex_doc metadata to: 20-EDA-tbl_FA_360_Full.ipynb
INFO:main:outputting converted file to: /Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted/20-EDA-tbl_FA_360_Full.tex
INFO:main:dumping external files to: /Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted/20-EDA-tbl_FA_360_Full_files
INFO:main:running pdf conversion
INFO:pdfexport:running: latexmk -bibtex -pdf <outpath>
INFO:pdfexport:latexmk: Latexmk: This is Latexmk, John Collins, 19 Jan. 2017, version: 4.52c.
INFO:pdfexport:latexmk: Rule 'pdflatex': Rules & subrules not known to be previously run:
INFO:pdfexport:latexmk: pdflatex
INFO:pdfexport:latexmk: Rule 'pdflatex': The following rules & subrules became out-of-date:
INFO:pdfexport:latexmk: 'pdflatex'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Run number 1 of rule 'pdflatex'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Running 'pdflatex  -recorder  "/Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted/20-EDA-tbl_FA_360_Full.tex"'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Latexmk: applying rule 'pdflatex'...
INFO:pdfexport:latexmk: This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdflatex)
INFO:pdfexport:latexmk: restricted \write18 enabled.
INFO:pdfexport:latexmk: entering extended mode
INFO:pdfexport:latexmk: 
INFO:pdfexport:latexmk: (/Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted
INFO:pdfexport:latexmk: /20-EDA-tbl_FA_360_Full.tex
INFO:pdfexport:latexmk: LaTeX2e <2017-04-15>
INFO:pdfexport:latexmk: Babel <3.10> and hyphenation patterns for 84 language(s) loaded.
...
INFO:pdfexport:latexmk: Package scrlayer-scrpage Warning: Very small head height detected!
INFO:pdfexport:latexmk: (scrlayer-scrpage)                Using scrlayer-scrpage the head height
INFO:pdfexport:latexmk: (scrlayer-scrpage)                should be at least \baselineskip, which is
INFO:pdfexport:latexmk: (scrlayer-scrpage)                12.0pt currently.
INFO:pdfexport:latexmk: (scrlayer-scrpage)                But head height is currently 0.0pt only.
INFO:pdfexport:latexmk: (scrlayer-scrpage)                You may use
INFO:pdfexport:latexmk: (scrlayer-scrpage)                geometry option `head=12.0pt'
INFO:pdfexport:latexmk: (scrlayer-scrpage)                to avoid this warning.
INFO:pdfexport:latexmk: 
INFO:pdfexport:latexmk: *geometry* driver: auto-detecting
INFO:pdfexport:latexmk: *geometry* detected driver: pdftex
INFO:pdfexport:latexmk: (./20-EDA-tbl_FA_360_Full.aux)
INFO:pdfexport:latexmk: 
INFO:pdfexport:latexmk: Package rerunfilecheck Warning: File `20-EDA-tbl_FA_360_Full.out' has changed.
INFO:pdfexport:latexmk: (rerunfilecheck)                Rerun to get outlines right
INFO:pdfexport:latexmk: (rerunfilecheck)                or use package `bookmark'.
INFO:pdfexport:latexmk: 
INFO:pdfexport:latexmk: )
INFO:pdfexport:latexmk: No pages of output.          <<<<<<<<<<
INFO:pdfexport:latexmk: Transcript written on 20-EDA-tbl_FA_360_Full.log.
INFO:pdfexport:latexmk: Latexmk: References changed.
INFO:pdfexport:latexmk: Latexmk: Log file says no output from latex          <<<<<<<<<<
INFO:pdfexport:latexmk: Latexmk: For rule 'pdflatex', no output was made          <<<<<<<<<<
INFO:pdfexport:latexmk: Rule 'pdflatex': File changes, etc:
INFO:pdfexport:latexmk: Changed files, or newly in use since previous run(s):
INFO:pdfexport:latexmk: '20-EDA-tbl_FA_360_Full.aux'
INFO:pdfexport:latexmk: '20-EDA-tbl_FA_360_Full.out'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Run number 2 of rule 'pdflatex'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Running 'pdflatex  -recorder  "/Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted/20-EDA-tbl_FA_360_Full.tex"'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: === TeX engine is 'pdfTeX'
INFO:pdfexport:latexmk: Latexmk: applying rule 'pdflatex'...
INFO:pdfexport:latexmk: This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdflatex)
INFO:pdfexport:latexmk: restricted \write18 enabled.
INFO:pdfexport:latexmk: entering extended mode
INFO:pdfexport:latexmk: 
INFO:pdfexport:latexmk: (/Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted
INFO:pdfexport:latexmk: /20-EDA-tbl_FA_360_Full.tex
INFO:pdfexport:latexmk: LaTeX2e <2017-04-15>
INFO:pdfexport:latexmk: Babel <3.10> and hyphenation patterns for 84 language(s) loaded.
...
INFO:pdfexport:latexmk: *geometry* driver: auto-detecting
INFO:pdfexport:latexmk: *geometry* detected driver: pdftex
INFO:pdfexport:latexmk: (./20-EDA-tbl_FA_360_Full.aux) )
INFO:pdfexport:latexmk: No pages of output.
INFO:pdfexport:latexmk: Transcript written on 20-EDA-tbl_FA_360_Full.log.
INFO:pdfexport:latexmk: Latexmk: Log file says no output from latex
INFO:pdfexport:latexmk: Latexmk: For rule 'pdflatex', no output was made
INFO:pdfexport:latexmk: Failure to make '20-EDA-tbl_FA_360_Full.pdf'
INFO:pdfexport:latexmk: Latexmk: Use the -f option to force complete processing,
INFO:pdfexport:latexmk: unless error was exceeding maximum runs of latex/pdflatex.
INFO:pdfexport:latexmk: === TeX engine is 'pdfTeX'
INFO:pdfexport:latexmk: Latexmk: Errors, so I did not complete making targets          <<<<<<<<<<
Traceback (most recent call last):
  File "/Users/angeloklin/anaconda2/bin/nbpublish", line 131, in <module>
    nbpublish(filepath, **options)   
  File "/Users/angeloklin/anaconda2/bin/nbpublish", line 64, in nbpublish
    create_pdf=create_pdf, pdf_in_temp=pdf_in_temp, pdf_debug=pdf_debug) 
  File "/Users/angeloklin/anaconda2/lib/python2.7/site-packages/ipypublish/main.py", line 148, in publish
    raise RuntimeError('the pdf export failed, try running with pdf_debug=True')
RuntimeError: the pdf export failed, try running with pdf_debug=True
ERROR:pdfexport:pdf conversion failed: Try running with pdf_debug=True
ERROR:main:pdf export returned false, try running with pdf_debug=True

conda / pip environments

$ conda list | grep -E "(nb|pub)"
ipypublish                0.4.1                     <pip>
nb_anacondacloud          1.2.0                    py27_0  
nb_conda                  2.0.0                    py27_0  
nb_conda_kernels          2.0.0                    py27_0  
nbconvert                 5.1.1                    py27_0  
nbconvert                 5.2.1                     <pip>
nbformat                  4.3.0                     <pip>
nbformat                  4.3.0                    py27_0  
nbpresent                 3.0.2                    py27_0  
widgetsnbextension        2.0.0                    py27_0  

$ pip list | grep -E "(nb|pub)"
ipypublish (0.4.1)
nb-anacondacloud (1.2.0)
nb-conda (2.0.0)
nb-conda-kernels (2.0.0)
nbconvert (5.2.1)
nbformat (4.3.0)
nbpresent (3.0.2)
widgetsnbextension (2.0.0)
@chrisjsewell
Copy link
Owner

chrisjsewell commented Jul 19, 2017

Ok, so this is the basic premise for the output formats:

  1. For latex_standard_article the format rules are; to create a pdf with all notebook cells represented (simplistically) in the final tex/pdf, as per the original nbconvert article format. Essentially, running jupyter nbconvert --to pdf Untitled.ipynb and nbpublish -pdf Untitled.ipynb should give you almost exactly the same tex/pdf.

  2. For latex_ipypublish_main (the default), the formatting rules are:

    a. Output all markdown cells, unless they have the latex_doc.ignore metadata tag
    b. Output no code cells or their output, unless they have a suitable latex_doc metadata tag (see here)

    my use case for this (and why I started building the package!) is that I have a notebook with lots of exploratory code and plots, which I don't want output in the final pdf. So I want to specifically select what cells are output.

So, firstly, try creating Test.ipynb (in a text editor) with the following text. Open it in Jupyter and run all the cells, then save it. (note you don't need to explicitly put %matplotlib inline; import matplotlib.pyplot as plt, since this is done in ipynb_latex_setup)

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from ipypublish.scripts.ipynb_latex_setup import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chart 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot([1,2,3], [4,2,7]);"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Now run nbpublish -f latex_standard_article Test.ipynb and you should get:

	\begin{document}

\title{Notebook}
\maketitle

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}1}]:} \PY{k+kn}{from} \PY{n+nn}{ipypublish}\PY{n+nn}{.}\PY{n+nn}{scripts}\PY{n+nn}{.}\PY{n+nn}{ipynb\PYZus{}latex\PYZus{}setup} \PY{k}{import} \PY{o}{*}
\end{Verbatim}

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}2}]:} \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{l+m+mi}{10}\PY{p}{)}\PY{p}{:}
            \PY{n+nb}{print}\PY{p}{(}\PY{n}{i}\PY{p}{)}
\end{Verbatim}

    \begin{Verbatim}[commandchars=\\\{\}]
0
1
2
3
4
5
6
7
8
9

    \end{Verbatim}

    Chart 2.

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}3}]:} \PY{n}{plt}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{,}\PY{l+m+mi}{2}\PY{p}{,}\PY{l+m+mi}{3}\PY{p}{]}\PY{p}{,} \PY{p}{[}\PY{l+m+mi}{4}\PY{p}{,}\PY{l+m+mi}{2}\PY{p}{,}\PY{l+m+mi}{7}\PY{p}{]}\PY{p}{)}\PY{p}{;}
\end{Verbatim}

\texttt{\color{outcolor}Out[{\color{outcolor}3}]:}\begin{center}
    \adjustimage{max size={0.9\linewidth}{0.9\paperheight}}{Test_files/output_3_0.pdf}
    \end{center}
    { \hspace*{\fill} \\}

	\end{document}

and run nbpublish Test.ipynb and you shoud get

	\begin{document}

Chart 2.

	\end{document}

@angeloklin
Copy link
Author

Hello Chris,
I believe I was misled by my cursory reading of the documentation.

My brain got that the output would be out regardless (unless stated otherwise). Probably because that was what I was looking for. My brain workflow was to have a working version with the actual content and just skip the source code, as the target audience would not be interested in it. So the ipynb here would be a publishable (!? publishable !?) quasi-print quality, without any extras.

But I totally get your point of having test code all around so you would focus on pushing what you do want to come out.

Maybe an idea for your apreciation would be to have a 3rd template that would put the output automatically out, for the lazy people that don't want to mess with metadata cell by cell. Honestly, it would be easier to just use the one line solution with the -f option pointing to an alternative template.

I will check the metadata content again to force the output. It then should work.
After I test it tomorrow, I can close this ticket as well.

Sorry for the confusion.

Thanks!

@chrisjsewell
Copy link
Owner

chrisjsewell commented Jul 19, 2017

Yeh, again thanks its good to get a user perspective.

So yes, as you mention, there are two routes to go;

  1. using latex_ipypublish_main with meta tags in the notebook. If you didn't know, you do View/Cell Toolbar/Edit Metadata to access these (in better visualising/editting of metadata attributes in the Jupyter notebook  #1 I talk about the goal to have a nicer way to do this).

  2. creating a new converter plugin to convert the notebook according to a different set of rules (e.g. including code and/or plots by default). My "dream" (lol) is that, as a community we can create a library of the export plugins for different use cases / output formats.

  • As the default then, I could think about making a latex_ipypublish_draft plugin, with all output put in the pdf by default. The only downside of such a document is that you wouldn't have a nice caption under the plot or label (to reference the plot in other parts of the document)
  • it might be good to have a specific check for .tex files output with nothing between the \begin{document} and \end{document} and output a specific warning in the log.
  • also in the log, it would be good for it to give a summary/verbose output of what cells it is outputting to the document and which it is ignoring, to avoid this kind of confusion.
  • put in the documentation about accessing metadata (View/Cell Toolbar/Edit Metadata)

@chrisjsewell
Copy link
Owner

FYI, for your test notebook, try this :)

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from ipypublish.scripts.ipynb_latex_setup import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chart 2\n",
    "\n",
    "with a nice reference to it \\cref{fig:chart2}."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "latex_doc": {
     "figure": {
      "caption": "the caption for chart 2",
      "label": "fig:chart2",
      "placement": "H"
     }
    }
   },
   "outputs": [],
   "source": [
    "plt.plot([1,2,3], [4,2,7]);"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

@angeloklin
Copy link
Author

Hello Chris,
I think you are with a very nice to do list in your hands.
The log options are useful for some people, but most probably won't use it. Maybe medium priority for those.
I like the draft idea, as it would solve a simpler problem which is just to publish the output but not the input. My guess is that most people wouldn't mind as the main intent is just to hide the code.
Your default serves a more ambitious goal to have a professional look for serious publications (very good and well done!). More configuration but it is a small price for the quality of the final product.

Again, I will just check your instructions tomorrow and will close the ticket if it all goes as planned.
Looking forward to seeing what the version 0.5.1 will bring. The 0.5.0 always has a minor typo or something that need to be fixed. :)

Thanks for your support.

@chrisjsewell
Copy link
Owner

chrisjsewell commented Jul 19, 2017

haha cheers, yeh just keep the feedback coming :)

My last top tip is, if you don't want all the extra files (produced by latexmk) in the converted folder, use nbpublish -pdf -ptemp Test.ipynb. The only drawback, is that latexmk can use these files to work out what changes have been made to the document since it last run and only process those, rather than make the entire document from scratch (particularly handy for making many small changes to long/complex documents).

Also, if you haven't noticed I've also created a new command nbpresent Test.ipnb which will convert and open your notebook as a slideshow.

  • This isn't related to nbpresent (which I've just found out about from you comments) so maybe I should change the command name to avoid conflict.
  • I've just noticed from the Test.ipynb that it outputs "output_type": "stream" in a weird position

@chrisjsewell
Copy link
Owner

chrisjsewell commented Jul 19, 2017

Moved the action points to #28, so I'll close this ticket now

@chrisjsewell
Copy link
Owner

oops actually I'll wait for you to check it tomorrow ta

@chrisjsewell chrisjsewell reopened this Jul 19, 2017
@angeloklin
Copy link
Author

Hello Chris,

Sorry. It just went over my head. :(

I thought I would be able to do it but it still escapes me. I believe I saw the solution in one way and you probably envision in a different one.
I was trying to find a tag to force the output of code, but not the code itself. I went to the code and definitions and only found code, table, figure. If I use code I get the code, but still no output.
I tried also to create a new format, but could not use it with the -f option.

To give you some context on what I would like to do, it is as follows:

  • I have some data from a database, which I converted to a flat file;
  • I am loading the flat file and doing some initial EDA (Exploratory Data Analysis)
  • I have a loop for each column and am I doing some simple investigation, which in essence looks like:
df = pd.read_csv()

for c in df.columns:
  c.describe()
  c.plot()

There is more to the code, but you get the idea. From that, the output would be a repetition of some text and a chart.

I could use the standard output, but it is quite simple. I did like the ipypublish latex style, but it leans more to an academic paper (which is why you created it, so I get it).
So goes back to the suggestion of having a format that by default would print out the markdown and all output without the code and/or have a module template like result analogous to code or content in the standard.

As I said I tried to do it, but requires learning a lot and takes quite some time. I have to go back to my analysis and modelling.

It would help me if you could digest it in smaller bits in a how to kind of format, on what metadata to use, it would be great. If it requires creating a whole new template/style, I would leave that to you to improve your code when you see fit.

In any case, I have learned quite a few new things and tricks in trying to find a solution for my need. Also nice to see how people can help each other. So thanks for that too.

Thanks!

@chrisjsewell
Copy link
Owner

chrisjsewell commented Jul 20, 2017

Ah you're so close!

  1. a plot is just a subcategory of a figure (this is a latex term for basically any graphic that you want to add a caption to)
  2. the metadata acts on the entire cell, so both the input (i.e. the code) and the output (i.e. the plot)

So literally replace code with figure, in that same place, and you will get the plot instead of the code. Or leave the code and add figure below it, and you will get both the code and the plot, e.g.

{
"latex_doc": {
  "code": {},
  "figure": {
    "caption": "Figure caption.",
    "label": "fig:flabel",
    "placement": "H",
    }
}
}

Does that explain it?

@chrisjsewell
Copy link
Owner

metadata_edit

@angeloklin
Copy link
Author

Hello Chris,
Didn't expect to get you awake at this time.

The figure was working before.
My issue now is the 0, 1, 2, 3,... output without the code.

Do you have a trick for that one?

Thanks

@chrisjsewell
Copy link
Owner

haha, bit of a night owl. So just to confirm; the figure is working now and, if I put the gif (above) in the documentation, and altered the following line:
For figures (i.e. any graphics output by the code), enter in cell metadata:
do you think that would make it clearer?

For the 0,1,2,3,.., no I didn't think to add a text meta type for my use case, so I will do that

@angeloklin
Copy link
Author

Hello Chris,
Yes, the image worked since you sent the details the first time. Thanks.

For me, the confusion was that I was looking for the output of the text as well.
The visual clue is nice, but it might make the page really long. Personally, I would be fine with something like: "for this: do that, where that is a more step-by-step instructions' list".
The catch here for you is that you have the big picture and are familiar with the code and the processes. For someone looking for a quick fix one (myself) would prefer to be lead by example (literally) :).

Nevertheless, you have done a great job.

PS: Feel free to close the ticket if you want.

@chrisjsewell
Copy link
Owner

ta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants