Pages encoded UTF-8 with BOM not rendered correctly

[stephendrew]

Hello,

I have a simple index markdown file:

# Project

## Overview

And it fails to render correctly in any theme.

Cinder

ReadTheDocs

Am I doing something wrong? If I change it to a level 2 heading, the same thing happens.

I can workaround it by placing a blank line at the start of the file, but then the heading appears too low:

Thanks.

[waylan]

Have you tried including a blank line between your headings? While not strictly necessary in all situations, failing to do so is generally bad form and can result in weird edge cases.

[stephendrew]

Actually, there is a blank line - that was a typo in writing the issue. It is the same either way.

[stephendrew]

It seems to only be occurring on the first page (index.md) in this project, although I seem to recall it doing it on multiple pages in another project (could be wrong).

[marcelkieser]

Same Problem here :( For every document in the repository.

This is the source of the file:

### Bug Template 

'''
[Short description of problem here]

**Reproduction Steps:**

1. [First Step]
2. [Second Step]
3. [Other Steps...]
...

Rendered HTML is

<article class="md-content__inner md-typeset">



  <h1>Bug template</h1>

<p>### Bug Template </p>

Tested with default theme and with material in mkdocs, version 0.16.3, running on Anaconda with Python 2.7.2 on Windows 10.

[facelessuser]

Can you post your mkdocs.yml file? Maybe this is dependent on certain extensions? I copied and pasted your example, but the syntax is weird in your example, and it doesn't look to be the full source of the image you posted. It isn't using backticks for fences, maybe that was a mistake when you posted the example? When I change the code start and end to ```, it works fine for me. But I don't think your source example is right. Regardless, maybe it has nothing to do with the source and is related to extensions?

[facelessuser]

I also can't reproduce the opening post's example in readthedocs. I tired a number of extension combinations and even removed all extensions, and I am not seeing that behavior.

[marcelkieser]

Hmm, sorry, I did only post the header of the document, as I thought the header declaration would suffice. This is the complete document:

### Bug Template 

'''
[Short description of problem here]

**Reproduction Steps:**

1. [First Step]
2. [Second Step]
3. [Other Steps...]

**Expected behavior/content:**

[Describe expected behavior here]

**Observed behavior/content:**

[Describe observed behavior here]

**Screenshots and GIFs**

[Screenshots and GIFs which follow reproduction steps to demonstrate the problem]

**Additional information:**

* Problem started happening recently, didn't happen in an older version: [Yes/No]
* Problem can be reliably reproduced, doesn't happen randomly: [Yes/No]
* Problem happens with all files and projects, not only some files or projects: [Yes/No]
...
'''

Note that I have replaced the backticks in the document with ' so I can paste it correctly ;)

But this document was only the easiest example to use to provide the error description. The behaviour ocurrs for every document in our repository, regardless of the heading level. Every first heading is rendered as a html header (<h1>) following a paragraph containing the actual source heading (<p>### ...). The repository is big and contains about 60 - 100 markdown files in various folders and sub folders, so I can't provide a comprehensive example :(

The yml file is very basic. I have not created any pages mappings because of the mass of documents, I need to generate mkdocs just for every document in the repository with the given folder structure so I leave that unconfigured.

These are the settings currently configured:

site_name: Docs
site_url: url_to_site
repo_url: repo_url
repo_name: Docs
site_description: Docs
site_author: company_name
copyright: company_name
use_directory_urls: false
theme: 'material'

Note that I also used the default theme (which can't be used because of the mass of doucments, material works better for that) with the same result.

I will try to generate the pages mapping automatically, perhaps this changes the result. Will keep you posted about that.

[marcelkieser]

I have tested a bit more and could identify that the problem lies with the file encoding. If the file is encoded in UTF-8 BOM (Windows 😞) then the rendering issue ocurrs. If you use proper UTF-8 there the files are generated as expected.

[facelessuser]

Well, that's good news. At least the issue has been clearly identified. As far as I know, MkDocs doesn't allow you to specify encoding, so as long as you use proper UTF-8, you should be fine. Maybe this should be more clearly stated in the documentation if it isn't already.

[facelessuser]

MkDocs could read files with encoding utf-8-sig which should strip out BOM if it is found, and if it is not found, it would read the file as a normal utf-8. I personally think this would end a lot of confusion in this regard. Issues like this are always unclear to debug, so just using an encoding that handles with and without BOM may be the easiest path forward to avoid issues like this in the future.

I also think it is perfectly reasonable for MkDocs to just clarify that they only accept utf-8 without BOM. But I suspect people will run into this again and again and have to have it explained to them; it may just be easier to use utf-8-sig and not worry about it.

[waylan]

If the file is encoded in UTF-8 BOM (Windows 😞) then the rendering issue ocurrs. If you use proper UTF-8 there the files are generated as expected.

That makes sense. Can't believe I didn't think of asking about encoding before.

I also think it is perfectly reasonable for MkDocs to just clarify that they only accept utf-8 without BOM. But I suspect people will run into this again and again and have to have it explained to them; it may just be easier to use utf-8-sig and not worry about.

You may have a point. I suspect this has not been done given the following in the Python docs:

To increase the reliability with which a UTF-8 encoding can be detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls "utf-8-sig") for its Notepad program... In UTF-8, the use of the BOM is discouraged and should generally be avoided.

First, I can't imagine anyone actually using Notepad to edit their files. Is there any other editors which actually do that by default? I've never encountered one.

That said, using utf-8-sig would just 'do the right thing' whether there is a BOM or not. In fact, a strict reading of the above is that writing the BOM to utf-8 is discouraged, not accounting for it when reading. As the docs mention:

On encoding the utf-8-sig codec will write 0xef, 0xbb, 0xbf as the first three bytes to the file. On decoding utf-8-sig will skip those three bytes if they appear as the first three bytes in the file.

So we could read using utf-8-sig and then write only with utf-8. That way, we can just ignore any BOM.