Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Allow references to Python variables in Markdown cells #2958

Open
ellisonbg opened this Issue · 34 comments
@ellisonbg
Owner

In PR #2592 @Carreau has come up with a syntax for referencing Python variables in Markdown cells. It uses the Jinja style {{x}} syntax. We love this idea, but there are some things that have to be worked out:

  • What syntax do we use? Are we happy with the {{}}
  • How do we make sure we are processing Markdown in a robust and sane way for latex and literate stuff. We are slowly diverging away from pure Markdown and that is really dangerous. We want the notebook format to work very broadly and having our own Markdown syntax seems like a bad idea.
  • How to handle errors, i.e., undefined variables.
  • What do we want to do about other display formats. Seems like a dangerous path to start to allow non-text formats.
@Carreau Carreau was assigned
@dartdog

so,, is this dead? It would be great as the prior thread pointed out? Sorry to bump wish I had the skilz to implement ( I don't) but I sure would use...

@minrk
Owner

Not dead, just need to get a few things done first.

@notionparallax

I just asked about this on SO, only to find that it was a double duplicate from here and here.

My 2c would be that

  • Jinja syntax would be great
  • It seems from a quick look that markdown doesn't really have a 'pure', but if all the formatting conventions from a well established implementation ( :octocat: github would be a nice one) were followed then adding an injection method isn't actually a markdown issue.
  • could errors be a user configurable thing? An option of [invisible (nothing happens), the word error, or a dirty great stack trace ]
  • As long as it's a text response then it could be injected before it is passed to the markdown renderer, so it could inject html or even more markdown!
@payne92

Not to advocate any particular syntax (I think {{}} is fine), but here's comparable design for R: http://www.rstudio.com/ide/docs/authoring/using_markdown (They use ``` as the delimiter).

Lunamark uses <!--@ for "metadata", see: http://jgm.github.io/lunamark/lunamark.1.html

@bilderbuchi

:+1:, this would be a very useful feature!

@JanSchulz

It would be really nice to have the same syntax as http://blog.rstudio.org/2014/06/18/r-markdown-v2/

@bilderbuchi

I don't know, using the R-style delimiters, i.e. single and triple backticks, respectively, it seems we couldn't choose between just displaying code (like some code snippet/example), and evaluating code and displaying the result (e.g. my_var) - I don't think this is very practical (if I understand it correctly).

@payne92

The use case is writing notebook prose with embedded computed values, such as "The average unemployment for Morgan County in 2014 was 8.4%, down from 10.3% in 2009." In this example, you could imagine the percentages and years being calculated from data.

The broader use case is "smart documents", where you have text and graphics automatically (re) generated (perhaps hiding the code for readability) from data.

I think it's a useful feature.

@JanSchulz

I really like the "knitr" model in rstudio (one markdown document, code blocks get interpreted, values are available in markdown -> gets converted into PDF/html/... which I can publish as an journal article. Different to a notebook as the notebook is based on multiple cells and I'm not sure how to influence whether output is shown or not).

I don't mind how the values are included in the text, but I would find it nice if both R (with knitr/rstudio) and ipython would use the same.

@gauteh
@bilderbuchi

I realize the use case of interpreted code of course, I'm just concerned that we would lose markdown's ability to "just" display code (without interpreting/evaluating it), like this, if the R syntax (i.e. backticks) were chosen.

@JanSchulz

I think in knitr/R markdown you can indicate whether you want to display code (properly highligted, etc) or only the output (plots, tables,...).

I'm coming from an economics background and I don't want to see any code in my papers, so this is a bit different to the (in my opinion) optimized usecase of showing off code.

@payne92

@bilderbuchi oh sorry, I misinterpreted the context of your "don't think this is useful" comment.

And I agree: whatever's implemented should not break existing markdown.

@tlnagy

This would be wonderful and make IPython eat knitr's lunch.

@JanSchulz

https://github.com/ipython-contrib/IPython-notebook-extensions/wiki/python-markdown

I really like this approach: it takes a cell and preprocess it to valid markdown, so no changes to the markdown processor are needed.

@dartdog
@tlnagy

:+1: elegant. I will use as well.

@nehalecky

@JanSchulz, thanks! I really look forward to trying out python-markdown.

@minrk minrk removed the prio-medium label
@spencerlyon2

Is there any chance of getting this into main ipython (jupyter)?

@Carreau
Owner

Is there any chance of getting this into main ipython (jupyter)?

Probably not soon. We need to do a lot of design in the background to have this to work.
Especially we would need an official way to extend markdown syntax instead of just inventing a new one.

@JanSchulz

Actually this (and also knitr/rmarkdown AFAIK) works by having a two stage conversion: first the replacement of any codeblocks with the output of the code and then convert the rest as standard markdown. So this is not an extension of markdown but a preprocessor for a cell.

@juhasch

I think the hard question is how to handle arbitrary invisible Python code that is executed from within a markdown cell. I don't think execution of code can be restricted in a useful implementation.

As long as it is a separate extension you actively have to install and activate, this is a different matter. Also, Python code only gets executed if the notebook is trusted.

I plan to add a tooltip showing the source code if you hover over the output of the python code in a markdown cell, so you can see where this came from.

@Carreau
Owner

Actually this (and also knitr/rmarkdown AFAIK) works by having a two stage conversion: first the replacement of any codeblocks with the output of the code and then convert the rest as standard markdown. So this is not an extension of markdown but a preprocessor for a cell.

Which is custom markdown. That's already what we have with mathjax. We obviously have to store the
un-preprocess markdown in case you re-run the notebook, so any tool that want to use our markdown have to deal with the custom preprocessing.
Pre or post processor, we invent our own syntax, that may, or may not conflict with what people want to do, or will do later.

I think the hard question is how to handle arbitrary invisible Python code that is executed from within a markdown cell. I don't think execution of code can be restricted in a useful implementation.

As long as it is a separate extension you actively have to install and activate, this is a different matter. Also, Python code only gets executed if the notebook is trusted.

I plan to add a tooltip showing the source code if you hover over the output of the python code in a markdown cell, so you can see where this came from.

If we do that, we could restrain to user_variable, ie return the value of user_ns key. That should prevent most execution.

@juhasch

If we do that, we could restrain to user_variable, ie return the value of user_ns key. That should prevent most execution.

You will also want to allow calling functions. A trivial case would be formatting the output or calling a custom repr.

@abalkin

I would love to see something like this implemented. It looks like the most straightforward solution is to run MD cells through a Jinja filter. This has been done before. See dexi, for example.

On the other hand, I don't think this should be enabled by default in all cells. Jinja would increase the complexity of the mark-up considerably and Jinja+Markdown should probably use different highlighting mode than plain MD. Why not implement this as a separate cell type?

@JanSchulz

I think this problem could go away, if it would be possible from code to hide/replace the code cell similar what is done for markdown currently:

-> Add a %%pymarkdown magic, which outputs a text/markdown message and indicates that the source should be invisible/replaced by the output. The magic would then simple do a s&r within the input.

[Edit: Ok, it has different problems, like no syntax highlighting and making no sense in the qtconsole...]

@payne92

Relevant: Mathematica 10 added support for "reports", see: http://www.wolfram.com/mathematica/new-in-10/automated-report-generation/

I haven't (yet) played with it in detail, but it looks like you can use notebook as templates, embedding references to variables and computed output in the notebook text.

@JanSchulz JanSchulz referenced this issue in JanSchulz/knitpy
Open

File extension. #5

@Pipeliner

Suppose we have multiple references in multiple MD cells to the same variable. Which cells should be rendered again when it changes its value?

@cswarth

@Pipeliner I'm not sure what you are getting at. In iPython/Jupyter notebooks each cell must be explicitly rendered (either through Ctrl-return from within the cell or from the Cell->Run All in the menu bar). There is no automatic detection of stale data that I am aware of. Are you thinking of knitr/rmarkdown? knitr/rmarkdown does try to maintain a cache of results from each cell and marks them as dirty when the earlier cells change.

@jgab3103

are there any workarounds for this problem people are using?. I love ipython/jupyter notebooks but it would amazing if we could just do {{ python variable }} in the markdown. Its so handy if you are preparing a notebook with a small sample datasets and then you later want in swap in the full set and have everything update

@jasongrout
Owner

You can use a PlaceProxy widget to render a widget at any html selector on the page. In your markdown, embed a <span id="myid">placeholder</span>, and then later in your code, use something like x=HTML('widget value'), PlaceProxy(child=x, selector='#myid'). Then any time you set x.value='something', the update is reflected in the widget which is displayed in the markdown. See https://github.com/ipython/ipywidgets/blob/1e407cef864363c66a23781b8d560a6ac18b3370/ipywidgets/widgets/widget_box.py#L70 for the definition of PlaceProxy.

Plenty of caveats, of course. if you edit the markdown, the widget disappears. If you open the notebook, the widget may not have been created so it won't show, etc. But it does let you programmatically control an output inside a markdown cell.

Another option is to construct the string and display the markdown as an output, not as a markdown cell. It won't dynamically update, but every time you evaluate the cell the markdown would be updated.

@jgab3103

I didn't want to to the construct the string and put is as output as it just gets so messy having big html strings - neat idea though, I didn't even think of that. But there are definitely a couple of good solutions which do the trick - thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.