Display yaml properties in markdown

[ghost]

Summary

It would be helpful to render yaml key-values in markdown.

As now we have no convenient way to document for example configuration variables. These are common in technical documentation and displaying these variables as links is not really an option. Updating each document is time consuming comparing to the situation where we have these variables explicitly defined.

Current problem

If I change the development environment ip-address I need to go through all the documentation and update each document which refers to this ip-address.

Example solution

application-properties.yaml

environment.development.ip: 127.0.0.1
environment.development.port: 8000

1. Connect to application server admin console and update the application.
2. Test the application:

$ curl  {{ environment.development.ip }} : {{ environment.development.port }} 

would be rendered to:

$ curl 127.0.0.1:8000

Additional goals

Additionally you could create an external yaml file which would have some or all the properties defined.

[d0ugal]

I think this idea has merit, I can see it being very useful. I expect we would want something like this for the 1.0 release. Possibly tied in with #144 somehow.

[ghost]

Middleman implementation (http://middlemanapp.com/advanced/local-data/) could be used as reference.

[d0ugal]

That implementation looks more like what I expect #144 to be. Data to be used from the template, not the markdwon content itself.

[waylan]

I'm thinking this may be a good candidate for a Plugin (see #206) if/when that API lands. What I envision is a plugin that uses a pre-build hook, accepts the Markdown document as a template and passes it through Jinja with the document's context. The resulting Markdown document (with variables filled in from the context) would be returned to Mkdocs for building.

However, I also expect that it would use non-standard variable start and end strings (i.e., perhaps [[ somevar ]] instead of {{ somevar }}). Otherwise, is would be rather difficult to include template snippets in code blocks within the Markdown text (they would need to be manually escaped).

And of course, the output would need to be valid Markdown. Some block level templating stuff may not work and various parts of Jinja would ideally be disabled (which may or may not be possible--I haven't worked the feasibility of that out). As it comes with lots of caveats (and could be a support headache), I think it makes a better candidate for a Plugin rather than something that is supported out-of-the-box.

[huglester]

I would like to join this.
I also agree, that this, indeed would be usefull to have variables

So for example, I can write:

You can visit [Content > Documents bank]({{ variables.base_url }}/admin/photobanks)

And user would be pointed to specific URL. In my use case, every documentation needs to (so called) base uri.
But it is based on the project itself. We could also set up some other variables.

Thanks

[waylan]

I should note that my proposal above may actually work better using one of the Python Mustache Clones (rather than Jinja) for "logic-less templates" -- which would be much more desirable within a Markdown document I would think.

[d0ugal]

+1 to logic-less.

It would be somewhat interesting to see what people could do with the full power of jinja2, but it also sounds super messy.

[waylan]

As a reminder to myself, pystache can accept alternate deliminators to it's parse function (so we can do [[ somevar ]] instead of {{ somevar }}):

renderer = pystache.Renderer()
parsed = pystache.parse(template, deliminators=('[[', ']]'))
output = renderer.render(parsed, context)

[facelessuser]

Why not just change the delimiters in Jinja2? http://jinja.pocoo.org/docs/dev/api/#high-level-api

Before pulling in yet another template library, I think it is very possible to get Jinja2 working. Maybe all of Jinja2's syntax wouldn't be useful in Markdown content, but you can let the user discover that.

block_start_string

The string marking the beginning of a block. Defaults to '{%'.

block_end_string

The string marking the end of a block. Defaults to '%}'.

variable_start_string

The string marking the beginning of a print statement. Defaults to '{{'.

variable_end_string

The string marking the end of a print statement. Defaults to '}}'.

comment_start_string

The string marking the beginning of a comment. Defaults to '{#'.

comment_end_string

The string marking the end of a comment. Defaults to '#}'.

[facelessuser]

I imagine if Jinja2 was used, if a user had conditional logic, it would be wise to keep it inline. But you would have access to filters and such as well so that is pretty cool ;).

[waylan]

@facelessuser as stated in my original proposal, I was intending to do just that (use Jinja). However, I feel like it offers too much for generating Markdown content. Granted that it my personal opinion (I don't even like the additional logic Jinja gives you over Django templates in general use). If it is implemented as a plugin, multiple solutions could exist side-by-side. I expect a Mustache solution would serve most users fine.

[facelessuser]

I see. Yeah, you are probably right. If it is in a plugin, then it isn't like you are adding more dependencies directly into mkdocs. Also, I guess if the inter-markdown templating is limited in the Markdown source, you will also get less noise in the issues about people trying to do something that doesn't work well in Markdown, and that is also a plus.

[geiseri]

I love the idea of mustache! In fact that is what I am using right now to "work around" the issue.

[d0ugal]

@geiseri How do you do that? pre-process markdown before you build it with MkDocs?

[geiseri]

I use pystache and have a python script that will create a branch, read the yml file to get the markdown files, then expand the markdown from a hard-coded dict to new files. Then replace the old files with the new ones and commit/push the branch. Then my readthedocs hook reads that. Crude but effective. I use it do handle branding differences for customer facing documentation.

[facelessuser]

So does the mustache code get parsed before Markdown processing or after? It seems if you parsed after, if you use mustache syntax in headers, you will get bad header ids. Parsing before seems like it would limit where mustache tags could be placed. It seems like even though mustache has a better chance of getting through the Markdown process safely compared to Jinja, it could still muck stuff up.

If you parse before, it seems like you could avoid those pitfalls, but maybe there is a downside with this as well?

[d0ugal]

Then replace the old files with the new ones and commit/push the branch. Then my readthedocs hook reads that.

It sounds like this happens before the Markdown is converted.

[waylan]

I can't think of any good reason to run this on the HTML after converting from Markdown. First of all, it wouldn't need to run on the entire page as everything outside of the Markdown body it in the template already (just use the template for that part). And even if it was run against the (partial) HTML returned by Markdown, I think people would get surprises as they are writing the template in a Markdown document but it is being run in HTML. One thing I've learned maintaining a Markdown library over the years is that not everyone who authors Markdown documents understands HTML. The best way forward in my mind is to have the templating render Markdown before the Markdown is parsed to HTML.

[facelessuser]

Okay, so using mustache didn't have the advantages I though it did, but it still looks cool, just very different. I am not a web developer, so I was just reading about the templates to understand them better, weighing their pros and cons.

I think I understand the differences now and why there are differing camps. I didn't realize that mustache forced you to push your logic down in a lower level underneath the templates. So you can still have logic per se; it just gives you a cleaner (kind of logic-less) template, I thought the reason for picking it was to help side step Markdown parsing issues, but as a pre parser, it won't really matter as both have a way to either escape content or change the brackets midway to suite your needs.

I'm not arguing one way or the other, I think mustache looks like a fine choice as it looks far less limited than I initially thought. I was more just looking into it to learn why some thought one template engine was better than the other. Thanks for indulging me.

[robodude666]

Why do we need to add an additional template syntax for variables in markdown files? Why can't the markdown simply render the html that includes jinja syntax in it, and then when the jinja templates get rendered the config.extra context would be available? That's the current order; it's just for some reason the jinja tags don't get converted with the template.

If someone wants to use jinja logic in their markdown files, let them; if they just want simple variables they could add that too.

Granted, if we had #237 extension support we could have a jinja-preprocess extension.. or some other variable preprocessing system.

[d0ugal]

How would you document a jinja2 example if all jinja2 like code got executed?

[facelessuser]

@robodude666, after experimenting with Jinja2 in markdown source in my own personal project, I don't understand why a different syntax would be preferred except that user X likes mustache better; Jinja seems to work fine.

But as stated before, it sounds like this would be implemented via a plugin API which means you could use Jinja like everything else does, user X could use mustache, and user Y could write a plugin for some other obscure template language that suits their needs.

People are very particular about there language choices. I personally think Jinja should be used everywhere as it is already being used in templates (it also appears to be faster and I don't personally have a problem with it exposing logic in the template). I really don't want to start a template debate; but I don't think there is a problem using either. Maybe I am missing something.

How would you document a jinja2 example if all jinja2 like code code executed?

I am not sure what you mean here, but would you have to document everything? Couldn't you say markdown source supports Jinja2 syntax and link them to Jinja docs? Maybe also say "your mileage may vary"? Explain the user may have to escape certain portions with Jinja raw tags?

In my experiments, I found it most useful to not do templates unless I defined use_template: true in my YAML frontmatter for my given file; keep in mind this is a completely different project. Only in that case would the file be processed with Jinja. That way I didn't have to worry about escaping anything in files that weren't going to use templating.

I am not sure how mkdocs plans to approach all this, but maybe this provides some food for thought.

[facelessuser]

FYI escaping Jinja in markdown wasn't really necessary for me except maybe if I had some LaTeX stuff in my doc, or I wanted to show Jinja or Jinja like syntax, but all you have to do is something like this:

{% raw %}{{ jinja.syntax}}{% endraw %}

I didn't find this to be a big deal though. I haven't played around with trying to show raw tags in raw tags, but I guess that is the only thing that might be annoying.

[d0ugal]

Yeah, having to escape code like that would be really surprising to users and horrible IMO.

[facelessuser]

That's a fair statement, but I don't imagine you would have to do it often. I think you will run into escaping issues no matter what syntax you use. Mustaches solution requires you to change your brackets mid way:

{{=<% %>=}}
some text
<%={{ }}=%>

I find that weird.

[d0ugal]

I'm just looking at the number of places I would need to remember to do that on this page.

[waylan]

@robodude666 You asked why the Jinja template does not already work on the HTML produced by Markdown. The answer is because the Markdown HTML is one of the variables in the context that Jinja inserts into the template. Jinja would need to run twice, once on the HTML and then a second time on the Page Template which would then insert that processed HTML into the page.

See this comment for why I think it is better to run the template against the Markdown text before converting to HTML.

[facelessuser]

What I would really like is templates that only provide the variable part and absolutely nothing else.

Yeah, even with mustache, the logic has to go somewhere. What you really want, is a pretty simple thing to implement without a template language; it's really just find and replace where you use the found key to get the info from your dict. You could easily homegrow your own with very little code.

I don't think any if statements or loops belong in the Markdown source at all.

I think there will be people who differ in your opinion of only having just simple variables; now whether or not mkdocs should entertain them is another topic. But I guess the real question is, does it really matter? If people don't want to use conditional logic, they don't have to. I imagine there are some people who may have legitimate use for some conditional logic.

I'm not sure it should be a concern for mkdocs to police what people use or enforce some idealistic opinion of what should be allowed. Just warn them about potential concerns and add a disclaimer that mkdocs is not going to support issues involving some advanced, wacky notation in their markdown code. If it works for them, why limit them?

Granted this all just my 2 cents, for whatever it is actually worth :).

[waylan]

I think there will be people who differ in your opinion of only having just simple variables;

I think this takes us back to the Plugin API. If the implementation uses the Plugin API, then users can use it, or use some third party implementation, or build their own, whichever suits their needs. However, I think (and this is only my personal opinion) that if MkDocs is going to provide a plugin which offers basic functionality, then it should be the absolutely most basic. Maybe you are right and it should be home-grown search and replace (although you lose filters, which may or may not be an acceptable compromise).

[facelessuser]

Maybe you are right and it should be home-grown search and replace (although you lose filters, which may or may not be an acceptable compromise).

I wasn't exactly sure about all of what you liked and didn't like in traditional template frameworks. But yeah, if that isn't feasible I understand.

I think this takes us back to the Plugin API. If the implementation uses the Plugin API, then users can use it, or use some third party implementation, or build their own, whichever suits their needs.

I agree. If the plugin API allows this to be configured via a plugin, then people can do whatever they want.

However, I think (and this is only my personal opinion) that if MkDocs is going to provide a plugin which offers basic functionality, then it should be the absolutely most basic.

I honestly don't think a template engine's extra features affects MkDocs one way or the other. MkDocs doesn't directly deal with any of that; simple or complex, the engine handles all of that. And Python Markdown shouldn't see any of that either as it should be parsed before passing it through.

Escaping is probably the most important concern, but escaping is always going to be something a user will have to directly confront at some point no matter what template engine is used and what default tag style is chosen. Jinja and mustache both allow changing the default tags if that is a concern, and both have ways of dealing with content too similar to the current tags. I am just having a hard time seeing how a simple template engine makes that big a difference to a complex one in regards to the tax it places on MkDocs.

[waylan]

I just stumbled across this. It provides an interesting solution to supporting variables in Markdown text. One thing I like about it is that all of the tags are in HTML comments so that if/when the page of rendered in any other Markdown system (such as source files on GitHub), the tags are seen as HTML comments and nothing weird happens. I don't think MkDocs would want to copy that feature for feature, but the basics are kind of nice:

<!-- @action data -->
<!-- $variable -->

The first performs an action on the provided data (i.e. include a page) and the second simply prints the value of the variable.

[robodude666]

@waylan That definitely is an interesting solution, however I feel iffy about using HTML comments for anything but comments.

If there's a typo in a variable name or not passed into the context for replacement it'll be harder to tell than replacing {{ $variable }} -- with the exception of links not working either way. In some cases:

This documentation was published on <!-- $date_now -->.

it would still be possible to tell that something was not processed correctly, however it won't be as obvious as seeing {{ $date_now }} rendered on the page. I guess as long as the variable processing has an option to error when not all variables are replaced it probably won't matter.

I'd personally still rather continue using Jinja2 and have the markdown processed first in the context of globally defined variables/plugins, then for the templates.

[facelessuser]

You could always configure jinja2 brackets to look like HTML comments if you wanted to. No sense in implementing a template engine.

[geiseri]

I can live with jinja2 looking like HTML. Then it wouldn't break stuff like github markdown, would it?

[undersound]

+1

Is there any update on this?

[margagn]

• 1 - Any update?

[waylan]

@margagn we are waiting for the Plugin API to happen first (see #206), then this feature will likely be implemented as a plugin.

[hellt]

As I see API has been delivered, what about this feature of getting variables into markdown?

[waylan]

The Plugin API is in the 1.0.dev branch, which is still in development and has not yet been released. When that branch stabilizes and is released, we will then turn our attention to some of the more popular plugin requests (such as this one).

[Ir1d]

Hi, I've noticed a plugin here: https://github.com/rosscdh/mkdocs-markdownextradata-plugin.

Is there a way to implement this without using a plugin?

Thx in advance.

[waylan]

@Ir1d no this will always be a plugin only feature. Even if we provide an official plugin, it will continue to be through a plugin only.

[rkoe]

Just stumbled over this, since I wanted to call some Jinja-macros from Markdown. It turned out that this can basically be added in the Jinja-templates, without any modification to MkDocs.

I now define a whitelist in the Jinja-Templates, which contains all variables/placeholders (plus their mapping to Jinja-macros/variables/functions) which can be used in Markdown, e.g.:

• Markdown:

  Hello World: {{ myfunc }}
  Some other macro: {{ foo.bar }}
  

• Template:

  {%- from "jinja.html" import jinja %} 
  {%- set jinja_allowed = {
          "myfunc": myfunc,
          "foo.bar": foo.bar
  %}
  {{ jinja(page.conent, jinja_allowed) }}
  
  

This works quite well.

But of course it could be improved/simplified if (a) a list of defined macros, (b) a "run-macro-by-name"-function and (c) regular expressions would be accessible in Jinja.

---

my jinja.html:

{# run Jinja-expressions from markdown
  
Often, it's useful to use placeholders or functions in Markdown.
This can be done via the Jinja-template, by defining using placeholders
in Markdown and replacing them with the result of Jinja-macros.

Usage:
- Markdown:

    {{ myexpr }}

- Template:

    {%- from "jinja.html" import jinja %}
    {%- set jinja_allowed = {"myexpr": mymacro, ...} %}
    {{ jinja(page_content, jinja_allowed) }}

Version: 2019-02-02
Author:  Roland Koebler <rk@simple-is-better.org>
#}
{%- macro jinja(s, allowed) %}
    {%- if "{{ " not in s  or  " }}" not in s %}
        {{- s }}
    {%- else %}
        {%- set s = s.replace("<p>{{ ", "{{ ").replace(" }}</p>", " }}") %}
        {%- set parts = s.split("{{ ") %}
        {{- parts[0] }}
        {%- for part in parts[1:] %}
            {%- set expr_rest = part.split(" }}", 1) %}
            {%- if expr_rest|length == 2 %}
                {%- set expr = expr_rest[0].strip() %}
                {%- if expr in allowed %}
                    {{- allowed[expr]() }}
                {%- endif %}
                {{- expr_rest[1] }}
            {%- else %}
                {{- "{{ "+part }}
            {%- endif %}
        {%- endfor %}
    {%- endif %}
{%- endmacro %}

[Evidlo]

@rkoe Awesome! I've modified it to support variables and args/kwargs to functions.

{# run Jinja-expressions from markdown

Often, it's useful to use placeholders or functions in Markdown.
This can be done via the Jinja-template, by defining using placeholders
in Markdown and replacing them with the result of Jinja-macros.

Usage:
- Markdown:

    {{ myvariable }}
    {{ myfunc(http://example.com/foo.png, desc=Description }}

- Template:

    {% macro myfunc(src,desc='') -%}
        <img style="display:block;" src="{{src}}">
        <span>{{desc}}</span>
    {%- endmacro %}

    {% set myvariable = "Hello world!" %}

    {%- from "jinja.html" import jinja %}
    {%- set allowed = {"myfunc": myfunc, "myvariable": myvariable} %}
    {{ jinja(page_content, allowed) }}

Version: 2019-02-28
Author:  Roland Koebler <rk@simple-is-better.org>
Author:  Evan Widloski <evan@evanw.org>
#}
{%- macro render_macros(s, expressions) %}
    {%- if "{{ " not in s  or  " }}" not in s %}
        {{- s }}
    {%- else %}
        {%- set parts = s.split("{{") %}
        {{- parts[0] }}

        {# render each function/variable, then the context up to the next func/var #}
        {%- for part in parts[1:] %}
            {%- set expr_parts = part.split("}}") %}
            {%- set expression = expr_parts[0].strip(" ") %}
            {%- set rest = "".join(expr_parts[1:]) %}

            {# handle functions #}
            {%- if "(" in expression %}
                {%- set func_parts = expression.split("(") %}
                {%- set func_name = func_parts[0] %}
                {%- set arguments = "".join(func_parts[1:]).rstrip(")").split(",") %}

                {# handle args and kwargs #}
                {%- set args = [] %}
                {%- set kwargs = {} %}
                {%- for argument in arguments %}
                    {%- if "=" in argument %}
                        {%- set kwarg_parts = argument.strip(" ").split("=") %}
                        {# just a hack for kwargs[kwarg_parts[0]] = kwarg_parts[1] #}
                        {%- set _ = kwargs.__setitem__(kwarg_parts[0], kwarg_parts[1]) %}
                    {%- else -%}
                        {%- set _ = args.append(argument) -%}
                    {%- endif %}
                {%- endfor %}

                {# call the function #}
                {%- if func_name in expressions %}
                    {{- expressions[func_name](*args, **kwargs) }}
                {%- endif %}

            {# handle regular variables #}
            {%- else %}
                {%- if expression in expressions %}
                    {{- expressions[expression] }}
                {%- endif %}
            {%- endif %}

            {{- rest }}
        {%- endfor %}
    {%- endif %}
{%- endmacro %}

[oprypin]

There are now popular plugins to achieve this:
https://github.com/fralau/mkdocs_macros_plugin
https://github.com/rosscdh/mkdocs-markdownextradata-plugin
So MkDocs itself doesn't need to support Jinja syntax directly, and this issue can be closed.