Process non markdown files

[redtide]

With Jekyll it's possible to process non markdown files if they contains a YAML front matter; IIUC this seems not possible with MKDocs, so I wonder which is the best approach for this.
For example I have an atom.xml that creates feeds from posts, and SVG images processed to create some budges used in the README of a project.
I would prefer to do this from jinja templates but I guess in alternative I could create the files from a python script and make them git-ignored.

[oprypin]

could create the files from a python script

That is properly supported, you could use an on_files event in hooks, but https://github.com/oprypin/mkdocs-gen-files is an even easier-to-use integration, where the script really does look like just a script but is properly integrated into a top-level mkdocs build command.

Other than that, yes, it's true that MkDocs only converts Markdown files to HTML and that's it. And I think you've also been implicitly mentioning the 'macros' plugin which apparently adds Jinja templating to Markdown files but not other files. But that's not something that can be solved here.

[redtide]

but https://github.com/oprypin/mkdocs-gen-files is an even easier-to-use integration, where the script really does look like just a script but is properly integrated into a top-level mkdocs build command.

Thank you! I'll start from that one.

[fralau]

As an additional note, mkdocs-macros is designed to work in the markdown pages.

Mkdocs-macros renders the macros during on_page_markdown event (just before the Markdown rendering).

To include non markdown files, you may use the Jinja2 include directive in your markdown page. Include does render the file included (see the paragraph in mkdocs-macros.)

You can used fenced blocks, if necessary:

```python
{% include 'your_filename.ext' %}
```

[redtide]

Thank you for the reply but it's not clear to me how this would be applied, because I only have to process the non markdown file, it is not included in any markdown page.

[fralau]

I undertand. Actually, you don't need to use any includes or code blocks. It depends on how you define a "markdown file".

By default, Mkdocs will only take files with the .md, .markdown, etc extension, to create html pages.

But in reality, it is agnostic about file format, as long as it is text Typically, if you specify a file in the nav section of the config file that is not markdown, it will take it and attempt to display it.

If it contains macros, mkdocs-macros will render them.

[redtide]

I don't have to add any of those non md files to nav or anywhere else, only process the jinja (or python) content in them, unless it's a hack that doesn't spit nav warnings that I'm not aware of.

[fralau]

The purpose of mkdocs is to convert text files (mostly markdown) into html files with navigation, in order to create a static website. I am not sure what effect you wish to achieve, other than that?

• If you wish to generate a single HTML page, and since mkdocs is built entirely around python-markdown, you could use the command line of python-markdown to achieve the effect you want? You could specify even the markdown extensions that you wish, in order to generate a stand-alone html page.
• If the idea is to generate markdown pages from other sources, that will then be processed by mkdocs, you could generate them with a Python script. The http server provided (mkdocs serve) is equipped with a detector of file changes, so that when you run the script to generate the markdown pages, your browser page would be updated. To create a production build, it's even easier: write a script that runs the production of markdown pages and then mkdocs build.
• In either case, if you want macros, then you could do it with a Python program that uses jinja2 to implement that processing, which sould be easy to do, in 10 lines of code.

To find the best way, it would be necessary to understand exactly what exactly you wish to do, and with which constraints?

[redtide]

I've linked to this discussion from the one I opened on your repo and forgot about this part of this one 🤦

In brief I would like that MKDocs could generate non-markdown/html files from files containing Jinja2 code like Jekyll does via Liquid, so, at least currently, not a MKDocs request but a Macro plugin request. I still confuse MKDocs with Jekyll where templating is part of the application, not a plugin apart.

So if MKDocs generates HTML from MD, I need to generate a SVG/XML file from a SVG/XML file containing Jinja2 code.

It seems Jekyll doesn't care much about the format if it has a frontmatter, where setting layout: null avoids Jekyll to apply the default layout (template in MKDocs) and make it a HTML file, so it just process the template part to manipulate whatever content the file have. Now I'm not sure if this works only with HTML like syntax files like XML or SVG, which are the ones I would like to work with in this case.

[redtide]

FTR I see that mkdocs-gen-files / python hooks could do the job but it's rather complicated than just using something like Jinja2 template.

I guess an atom.xml file would be a better example (reported also here from the discussion in the Macro plugin repository):

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
	<title>{{ site.name }} News</title>
	<link href="{{ '/atom.xml' | absolute_url }}" rel="self"/>
	<icon>{{ '/assets/ico/favicon.ico' | absolute_url }}</icon>
	<link href="{{ '/' | absolute_url }}"/>
	<updated>{{ site.time | date_to_xmlschema }}</updated>
	<id>{{ '/' | absolute_url }}</id>
{% for post in site.posts limit:20 %}
	<entry>
		<title>{{ post.title }}</title>
		<author><name>{{ post.author }}</name></author>
		<link href="{{ post.url | absolute_url }}"/>
		<published>{{ post.date | date_to_xmlschema }}</published>
		<id>{{ post.id | absolute_url }}</id>
		<content type="html">{{ post.content | xml_escape }}</content>
	</entry>
{% endfor %}
</feed>

This is the original Jekyll/Liquid file: given a number of post files (markdown, managed in MKDocs via blog plugin), this produces a feed entry for each.

[oprypin]

So how about running Jinja on the content of the file.

with mkdocs_gen_files.open("foo.xml", "r") as f:
    content = f.read()
content = jinja2.Template(content).render()
with mkdocs_gen_files.open("foo.xml", "w") as f:
    f.write(content)

Alternatively for this particular purpose it seems that extra_templates would fit perfectly?
https://www.mkdocs.org/user-guide/configuration/#extra_templates

[redtide]

Haven't noticed about extra_templates.
I've added a atom.xml file in docs dir and set

extra_templates:
- atom.xml

in mkdocs.yml and

---
template: atom.xml
---

on a atom.md file for testing, but I get a jinja2.exceptions.TemplateNotFound: atom.xml error.
Sorry but how it's the correct setting?

[oprypin]

Delete atom.md, that setting just works by itself

[redtide]

This is what I need for the first part, now I need to see how to extract the other information (post id, author, dat, html content...) but seems like it should work entirely for the svg case, let's see... thank you!

[redtide]

Nope, it doesn't, so unlucky 😭
TypeError: no loader for this environment specified IIUC it's only for xml and html.
Will retry later with mkdocs_gen_files.

[fralau]

I guess an atom.xml file would be a better example (reported also here from the discussion in the Macro plugin repository):

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
	<title>{{ site.name }} News</title>
	<link href="{{ '/atom.xml' | absolute_url }}" rel="self"/>
	<icon>{{ '/assets/ico/favicon.ico' | absolute_url }}</icon>
	<link href="{{ '/' | absolute_url }}"/>
	<updated>{{ site.time | date_to_xmlschema }}</updated>
	<id>{{ '/' | absolute_url }}</id>
{% for post in site.posts limit:20 %}
	<entry>
		<title>{{ post.title }}</title>
		<author><name>{{ post.author }}</name></author>
		<link href="{{ post.url | absolute_url }}"/>
		<published>{{ post.date | date_to_xmlschema }}</published>
		<id>{{ post.id | absolute_url }}</id>
		<content type="html">{{ post.content | xml_escape }}</content>
	</entry>
{% endfor %}
</feed>

This is the original Jekyll/Liquid file: given a number of post files (markdown, managed in MKDocs via blog plugin), this produces a feed entry for each.

@redtide For the example you are submitting, is my understanding correct that it's the output you want? You want the final page to be straight XML? I don't think that should be a problem, except that your output XML will be bracketed between other HTML for the header and footer, navigation, etc. (I assume you intend to put your XML code into a fence block?).

MkDocs-Macros could do it fairly simply, if you copy your XML code/Liquid code directly into a markdown page of the MkDocs project, with the small tweaks so that it's Jinja2-compliant.

What you will need, of course, is the same variables, functions and filters:

• essentially your posts list, containing post objects. I assume you will need to read it from a json or XML file or a database table.
• an xml_escape filter (check whether Jinja2 provides it, or implement it using MkDocs Macros).

See for the gist of how to do that, within the main.py module.

[redtide]

I need an usable XML, so the file with the jinja2 template processed, no fence block or additional html.
The variables are not important, their values are, so IDK in this case how to extract the title, author, date and content if not processing each post' markdown file apart.

[fralau]

Ah, ah. In that case, @oprypin advice is the best: you could manage your own templating with the Jinja2 library, and fetch your own template files (I would go with a completely separate solution from MkDocs' own templating).

For that, no need of a plugin: you can use the hook functions of MkDocs (since version 1.4) with on_post_build.

Or else, MkDocs Macros also allows you to do that with on_post_build.

Between the two, it depends on what you want to do: if you don't need anything else, the principle of economy would be to use MkDoc's standard solution. If you already need MkDocs-Macros for other stuff, then using it for that too, might give you more consistency.

[redtide]

So I've come to this, though I would like to reuse as much mkdocs api as possible but haven't got how to use MkDocsConfig. Anyway... thank you all for the help, got it finally working 😃

generate_atom.py

Click to view

import frontmatter, jinja2, mkdocs_gen_files, os, yaml
from datetime import datetime, timezone
from markdown import markdown
from mkdocs   import utils
#from mkdocs.config.defaults import MkDocsConfig
#config = MkDocsConfig # TypeError: unsupported operand type(s) for +: 'Optional' and 'str'

def post_date(s):
  dt = datetime.strptime(s, "%Y-%m-%d %H:%M:%S")
  dt = dt.replace(tzinfo=timezone.utc)
# return dt.strftime("%Y-%m-%dT%T%z")# without comma in tz
  return dt.isoformat()

with open("mkdocs.yml") as f:
  config = yaml.load(f, Loader=yaml.FullLoader)

with mkdocs_gen_files.open("atom.xml", "r") as f:
  content = f.read()

posts_dir = os.path.abspath("docs/news/posts/")
posts = []
files = [
  os.path.abspath(os.path.join(posts_dir, f))
    for f in os.listdir(posts_dir)
    if os.path.isfile(os.path.join(posts_dir, f))
]
for f in sorted(files, key=lambda x: x, reverse=True):
  with open(f) as file:
    post = frontmatter.load(file)

  post["date"] = post_date(post["date"])
  post["id"]   = "/news/posts/" + os.path.splitext(os.path.basename(f))[0]
  posts.append(post)

content = jinja2.Template(content).render(
# now=utils.get_build_datetime().strftime("%Y-%m-%dT%T%z"),# without comma in tz
  now=datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
  config=config,
  markdown=markdown,
  site_posts=posts
)
with mkdocs_gen_files.open("atom.xml", "w") as f:
  f.write(content)

atom.xml

Click to view

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
	<title>{{ config.site_name }} news</title>
	<link href="{{ config.site_url + '/atom.xml' }}" rel="self"/>
	<icon>{{ config.site_url + '/assets/ico/favicon.ico' }}</icon>
	<link href="{{ config.site_url }}"/>
	<updated>{{ now }}</updated>
	<id>{{ config.site_url }}</id>
{%- for post in site_posts[0:3] %}
	<entry>
		<title>{{ post.title }}</title>
		<author><name>{{ post.author }}</name></author>
		<link href="{{ config.site_url + post.id + '/' }}"/>
		<published>{{ post.date }}</published>
		<id>{{ post.id }}</id>
		<content type="html">{{ markdown(post.content) }}</content>
	</entry>
{%- endfor %}
</feed>

[fralau]

Great! Just curious: did you implement that completely separate from the MkDocs production chain?

(I would have implemented your generate_atom.py program as an Mkdocs on_post_build hook, to make sure that it gets regenerated each time there is a build of the website.)

[redtide]

Sorry, IDK what you mean with "completely separate from the MkDocs production chain", I just added the gen_files plugin in the config and added the python script to it.

Isn't gen_files running at each build? OTOH I use this in a GH Actions CI workflow, so it must create the atom.xml each build, after which the files are deleted and force pushed the next time.

In case someone was wondering why I didn't used some RSS plugin the answer is that I come from Jekyll, which was a PITA with plugins when using gh-pages, so if I can avoid a plugin with a pair of files I like much more this way. (If were me, seeing that the most useful things are not that much I would get rid of plugins and integrate things like this directly in the site generator.)

[oprypin]

@redtide Yes you did great, @fralau just didn't know that mkdocs_gen_files is integrated.

To get the config you should do (instead of reading yaml):

config = mkdocs_gen_files.config

https://oprypin.github.io/mkdocs-gen-files/api.html

[fralau]

It clicked: 💡 you used the gen_file plugin (it was actually mentioned above)!