Markdown in raw HTML stops working after first raw HTML tag

[fpw]

Hello,

I'm using Markdown and the "extra" extension to support Markdown in raw HTML div elements with the attribute markdown='1' as explained on the example page: https://pythonhosted.org/Markdown/extensions/extra.html

However, as soon as a raw HTML tag without the "markdown" attribute occurs inside the elment, Markdown will not be processed anymore after the end of that element. If you put any Markdown code after the "Raw html blocks may also be nested." text of the example page, it will not be processed even though it is still in the "markdown='1'" div.

Short example with 3 Markdown uses where the second one is not processed:

<div markdown="1">

Markdown is *active* here.

<div name="RawHtml">
Raw html blocks may also be nested.
</div>

Markdown is *not* active anymore here.

</div>

Markdown is *active again* here.

[fpw]

Additional info: The code actually works as expected on Linux with Python 2.7.6 and python-markdown 2.4.1 but not on OS X with Python 2.7.8 and latest python-markdown from git.

[waylan]

Additional info: The code actually works as expected on Linux with Python 2.7.6 and python-markdown 2.4.1 but not on OS X with Python 2.7.8 and latest python-markdown from git.

Well that's odd. And if that's the case, I'm afraid I have no way of reproducing the error myself, which means I'm not likely to debug it. Especially if it is in the HTML parsing code. That's the most complex part of the parser. Hopefully someone who can reproduce the error will take an rack at it. Thanks for the report.

[facelessuser]

I get this as well, but I also get in 2.4.1. I am not running any Python less than 2.7.8...that would seem odd if the Python version was a real factor, but maybe. If I get some time, I may look at it.

[fpw]

I've tested it again with Python 2.7.6 and am not getting it on Linux with the latest revision, so it seems to depend on the Python version I guess.

[facelessuser]

Oh...that's annoying, but at least that helps narrow down the issue.

[waylan]

@facelessuser what OS are you running? My assumption was that the different OS, not the different python version was what was causing the issue. @fpw reported the problem only on OSX (which I can't replicate as I currently don't have access to such a system). I'm just trying to narrow down the exact situation in which the error occurs.

[fpw]

@waylan It is OS independent, but seems to depend on the Python version. I'm getting the unexpected behavior with Python 2.7.8 on both OS X and Linux, but not with Python 2.7.6 on either OS X or Linux.

[facelessuser]

I have reproduced on OSX. I have also reproduced it on windows with a portable python 2.7.6. So I should say, I haven't not been able to reproduce it. I have been able to patch it for my scenarios, but I am still trying to understand how @waylan and @fpw can have it work with different environments.

[fpw]

As far as I understood, @waylan hasn't tested it himself yet. And as this is the first time I'm using a Python project, maybe my testing is wrong - I've cloned the repository from Git and then just ran python bin/markdown_py -x extra while 2.4.1 was still installed. The script looks like it just imports Markdown, maybe it imports 2.4.1 because I haven't actually installed the version I cloned? If that's how Python packages work, I'm sorry for the confusion, I'm a newbie when it comes to Python :)

If that's the case, the bug was introduced after 2.4.1 and exists for all Python versions and OSs in the current branch.

[facelessuser]

I would create a file called test.py with the following content:

import markdown

text = '''<div markdown="1">

Markdown is *active* here.

<div name="RawHtml">
Raw html blocks may also be nested.
</div>

Markdown is *not* active anymore here.

</div>

Markdown is *active again* here.
'''

html = markdown.markdown(text, extensions=['markdown.extensions.extra'])

print(html)

And then drop it in the Python Markdown 2.5.X folder at the same level as the markdown folder. Then open a terminal at that location and run the script:

python test.py

Since the test.py current working directory contains the markdown library and the current working directory takes priority over the system library path, it it will import that one instead of the installed one.

The way the markdown_py script works, it will first look in the bin folder to see if it can find markdown, then it will walk the rest of the system paths looking for markdown and will find the installed version. So probably not the best way to run it.

[facelessuser]

I was able to isolate the problem in preprocessors.py in _nested_markdown_in_html. It appears it wasn't separating the child html tags from the parent content well. Fixing it's indexing caused the tail text to get processed proper.

Now I need to run more exhaustive tests, but at least for the case you posted, I am now getting this:

<div>
<p>Markdown is <em>active</em> here.</p>
<div name="RawHtml">
Raw html blocks may also be nested.
</div>

<p>Markdown is <em>not</em> active anymore here.</p>
</div>
<p>Markdown is <em>active again</em> here.</p>

I did this early this morning, so I haven't had a chance to look into it anymore today. Maybe tonight, I will have a chance, maybe not. Depends what I have going on.

I was mainly concerned that I couldn't get get 2.5.1 to work on 2.7.6 Python, but maybe it was universally broken for 2.5.1.

[fpw]

@facelessuser Thanks, your test script allowed me to track down the bug. It is indeed not dependent on the OS or the Python version, sorry again for the previous confusion. I've now tested all releases and the bug was introduced between 2.4-final and 2.4.1-final and is still present in the current branch. The Debian package for 2.4 is called 2.4-1, hence my mistake for reporting this as working in 2.4.1. Using your script I've now double-checked all statements in this post :)

[facelessuser]

Fantastic, that makes more sense. I couldn't see any code that looked like it would break for only Python > 2.7.6. I think I may check out the changes that were made between those points and match them up with what I patched in 2.5.1 and see if it all makes sense. I probably need to run some tests on some more complicated HTML scenarios to make sure I am not fixing one thing and breaking something else though.

[facelessuser]

Sure enough, it was this commit that broke it bdb1bda. But I need to understand what the intention was behind the original change so that both the original issue is fixed and the this new issue is fixed.

[facelessuser]

Issue #308 is the origin of this. I will make sure we are not regressing with my changes before I make a pull request.