Presence of new line characters in inline attribute list throws exception

[rkrp]

Presence of new line characters \n (without any whitespace around) in markdown attribute list causes exception while converting to HTML. The problem is in attr_list extension.

Case 1

from markdown import markdown

md = """
![text](http://link){foo="a"\nbar="b"}
"""
print markdown(md, extensions=['attr_list'])

This results in ValueError: too many values to unpack in _handle_double_quote

Case 2

This produces incorrect results when there is a class name or id present in the attribute list.

For example,

from markdown import markdown

md = """
![text](http://link){foo="a"\n.nomedia bar="b"}
"""
print markdown(md, extensions=['attr_list'])

produces the following output:

<p><img _.nomedia="
.nomedia" alt="text" bar="b" foo="a" src="http://link" /></p>

Possible Fix

Replacing the new line characters with a single space fixes the problem. It also provides the expected output for the two cases.

diff --git a/markdown/extensions/attr_list.py b/markdown/extensions/attr_list.py

index 683bdf8..83c2c01 100644
--- a/markdown/extensions/attr_list.py
+++ b/markdown/extensions/attr_list.py
@@ -139,6 +139,7 @@ class AttrListTreeprocessor(Treeprocessor):
             else:
                 # inline: check for attrs at start of tail
                 if elem.tail:
+                    elem.tail = elem.tail.replace('\n', ' ')
                     m = self.INLINE_RE.match(elem.tail)
                     if m:
                         self.assign_attrs(elem, m.group(1))    

After applying the above patch, the results are obtained as

Case 1

![text](http://link){foo="a"\nbar="b"}

<p><img alt="text" bar="b" foo="a" src="http://link" /></p>

Case 2

![text](http://link){foo="a"\n.nomedia bar="b"}

<p><img alt="text" bar="b" class="nomedia" foo="a" src="http://link" /></p>

EDIT: Added code highlighting

[waylan]

Thanks for the report. I agree that we should never raise an exception based on bad input. However, I'm not sure this is actually a bug in this case, or, if it is, that your suggested solution is the right way to go. I'm inclined to say that the newline makes is not a valid attribute list and it should not be seen as one.

And then there is the escaping issue. If you were to save your sample (with the literal \n) to a text file on the file system, and then feed that file into Markdown, the problem would not exist. it would be the same as doing the following:

>>> t = '''
... ![text](http://link){foo="a"\\n.nomedia bar="b"}
... '''
>>> markdown.markdown(t, extensions=['attr_list'])
u'<p><img _n.nomedia="\\n.nomedia" alt="text" bar="b" foo="a" src="http://link" /></p>'

Notice the escaped newline character (\\n). The output is clearly wrong (in that it is not what the author intended), but it doesn't break anything. That said, in the file someone could put an actual line-break in the middle of an attribute list:

>>> t = '''
... ![text](http://link){foo="a"
... .nomedia bar="b"}
... '''
>>> markdown.markdown(t, extensions=['attr_list'])
u'<p><img _.nomedia="\n.nomedia" alt="text" bar="b" foo="a" src="http://link" /></p>'

And that produces almost the same result. Most notably, no error is raised.

The fact is, the bug can only be produced in the python console as a user error where the user fails to properly escape a character. In fact, consider this:

>>> t = '![text](http://link){foo="a"\n.nomedia bar="b"}'
>>> markdown.markdown(t, extensions=['attr_list'])
u'<p><img _.nomedia="\n.nomedia" alt="text" bar="b" foo="a" src="http://link" /></p>'

The only difference between this and the original example is that the input is in single quotes rather than quote literals. Using \n in a quote literal is known to have weird side effects and is only reproducible within the Python console. In a quote literal, newline characters need an extra level of escaping to properly reflect input from any other source.

However, as I noted above, the output is still not what the author intended. That is acceptable as the user failed to follow the syntax properly. When the Markdown parser gives you different HTML than you expected, the first step should be to go back and correct the error in the Markdown you authored. Only when the Markdown input is of the proper syntax (but the output is wrong) do we have a bug, which is not the case here.

And yes, actually raising an exception is bad in a server environment (for example). However, in this case the exception can only be raised on the Python console (and nowhere else), so the exception will never be seen in the real world. Hardly a concern.

That said, if a fix were to be applied, it would be to cause the parser to fail to recognize anything with a newline character in it as an attribute list. In other words, the input:

![text](http://link){foo="a"
.nomedia bar="b"}

should probably output:

<p><alt="text" src="http://link" />{foo="a"
.nomedia bar="b"}</p>

[waylan]

Hmm, after posting my reply above, it occurred to me that I never actually tried the example provided in the original report:

>>> t = '''
... ![text](http://link){foo="a"\n.nomedia bar="b"}
... '''
>>> markdown.markdown(t, extensions=['attr_list'])
u'<p><img _.nomedia="\n.nomedia" alt="text" bar="b" foo="a" src="http://link" /></p>'

I'm not able to reproduce the reported error.

@rkrp, which version of Python are you using and on which OS?

[mitya57]

@waylan I can reproduce @rkrp's error with Markdown 2.6.6 and Python 3.5.1 on Linux.

Also, I never heard anything about side-effects of using \n in a quote literal. For me, it is just a normal line break:

>>> a = """foo\nbar"""
>>> b = """foo
... bar"""
>>> a == b
True

And, as expected, I can reproduce it with normal line break too:

>>> md = """
... ![text](http://link){foo="a"
... bar="b"}
... """
>>> markdown(md, extensions=['attr_list'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/markdown/__init__.py", line 494, in markdown
    return md.convert(text)
  File "/usr/lib/python3/dist-packages/markdown/__init__.py", line 375, in convert
    newRoot = treeprocessor.run(root)
  File "/usr/lib/python3/dist-packages/markdown/extensions/attr_list.py", line 144, in run
    self.assign_attrs(elem, m.group(1))
  File "/usr/lib/python3/dist-packages/markdown/extensions/attr_list.py", line 149, in assign_attrs
    for k, v in get_attrs(attrs):
  File "/usr/lib/python3/dist-packages/markdown/extensions/attr_list.py", line 66, in get_attrs
    return _scanner.scan(str)[0]
  File "/usr/lib/python3.5/re.py", line 376, in scan
    action = action(self, m.group())
  File "/usr/lib/python3/dist-packages/markdown/extensions/attr_list.py", line 35, in _handle_double_quote
    k, v = t.split('=')
ValueError: too many values to unpack (expected 2)

[mitya57]

This happens with inputs that don't contain newlines too:

>>> md = "![text](http://link){foo=a=b}"
>>> markdown(md, extensions=['attr_list'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dmitry/upstream/Python-Markdown/markdown/__init__.py", line 494, in markdown
    return md.convert(text)
  File "/home/dmitry/upstream/Python-Markdown/markdown/__init__.py", line 375, in convert
    newRoot = treeprocessor.run(root)
  File "/home/dmitry/upstream/Python-Markdown/markdown/extensions/attr_list.py", line 144, in run
    self.assign_attrs(elem, m.group(1))
  File "/home/dmitry/upstream/Python-Markdown/markdown/extensions/attr_list.py", line 149, in assign_attrs
    for k, v in get_attrs(attrs):
ValueError: too many values to unpack (expected 2)

(this time I tested with latest master rather than 2.6.6 release).

[rkrp]

Thanks for the reply. I am sorry, I guess I wasn't very clear in my original report. Let me give the context. The markdown document, I am trying to convert is an output from pandoc.

Pandoc produces markdown document with newline characters in between the {} where the inline attributes are specified. For example, consider the following document (test.md),

![](something.png){.aligncenter .wp-image-15 .size-large width="720"
height="390"}

Running

md = open('test.md').read()
print markdown(md, extensions=['attr_list'])

This results in

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    print markdown(md, extensions=['attr_list'])
  File "/usr/lib/python2.7/site-packages/markdown/__init__.py", line 494, in markdown
    return md.convert(text)
  File "/usr/lib/python2.7/site-packages/markdown/__init__.py", line 375, in convert
    newRoot = treeprocessor.run(root)
  File "/usr/lib/python2.7/site-packages/markdown/extensions/attr_list.py", line 144, in run
    self.assign_attrs(elem, m.group(1))
  File "/usr/lib/python2.7/site-packages/markdown/extensions/attr_list.py", line 149, in assign_attrs
    for k, v in get_attrs(attrs):
  File "/usr/lib/python2.7/site-packages/markdown/extensions/attr_list.py", line 66, in get_attrs
    return _scanner.scan(str)[0]
  File "/usr/lib/python2.7/re.py", line 336, in scan
    action = action(self, m.group())
  File "/usr/lib/python2.7/site-packages/markdown/extensions/attr_list.py", line 35, in _handle_double_quote
    k, v = t.split('=')
ValueError: too many values to unpack

Since, it is not specified in maruku from where the IAL syntax was inspired from and also, since Pandoc inserts new line characters in the middle of IAL, I assumed that it is a valid markdown syntax.

[rkrp]

For a document, such as

![text](http://link){foo="a"
.nomedia bar="b"}

The expected output is

<img alt="text" bar="b" class="nomedia" foo="a" src="http://link" />

But, the obtained output is,

<img _.nomedia="
.nomedia" alt="text" bar="b" foo="a" src="http://link" />

Please note the difference in the class attribute between the expected and actual result.

@waylan I am using Markdown 2.6.6, Python 2.7.11 in Linux [Arch]

[rkrp]

@waylan No disrespect intended. I am a student who is looking forward to contributing to opensource projects.

I was working with Pelican which uses both Python-Markdown and Pandoc to convert documents from one format to another.

Pelican was throwing the above specified exception all over my console since, it was encountering newline characters in IAL in the documents generated (around 100 in number) by Pandoc. I found that the issue was in attr-list extension and then patched it up to strip off the newline characters. It was a quick and dirty solution which worked well for my case. I understand that it might not be applicable for all the cases. And that duct-tape code is not manageable in the long run.

[waylan]

Okay, sorry about the weird escaping thing. Not sure what I was thinking there. Anyway, I am able to reproduce it now.

>>> t = '''
... ![text](http://link){foo="a"
... bar="b"}
... '''
>>> markdown.markdown(t, extensions=['attr_list'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "markdown/__init__.py", line 494, in markdown
    return md.convert(text)
  File "markdown/__init__.py", line 375, in convert
    newRoot = treeprocessor.run(root)
  File "markdown/extensions/attr_list.py", line 144, in run
    self.assign_attrs(elem, m.group(1))
  File "markdown/extensions/attr_list.py", line 149, in assign_attrs
    for k, v in get_attrs(attrs):
  File "markdown/extensions/attr_list.py", line 66, in get_attrs
    return _scanner.scan(str)[0]
  File "/home/waylan/.virtualenvs/md/lib/python2.7/re.py", line 323, in scan
    action = action(self, m.group())
  File "markdown/extensions/attr_list.py", line 35, in _handle_double_quote
    k, v = t.split('=')
ValueError: too many values to unpack
>>> t = '''
... ![text](http://link){foo="a"
... .baz bar="b"}
... '''
>>> markdown.markdown(t, extensions=['attr_list'])
u'<p><img _.baz="\n.baz" alt="text" bar="b" foo="a" src="http://link" /></p>'

What is interesting is that when the newline is followed by a dot class, the error does not occur. However, if the newline is followed by a key-value pair, then the error occurs. Apparently, I only checked the former earlier and that is why I couldn't reproduce the problem.

And to be clear, for a document, such as

![text](http://link){foo="a"
.nomedia bar="b"}

The only acceptable output is:

<p><img alt="text" src="http://link" />{foo="a"
.nomedia bar="b"}</p>

It is not:

html<img alt="text" bar="b" class="nomedia" foo="a" src="http://link" />

An attribute list must not contain a newline. Anything which looks like an attribute list that contains a newline is not an attribute list.