ElementTree and minidom don't prevent creation of not well-formed XML

[ods]

BPO	5166
Nosy	@ods, @scoder, @jwilk, @florentx, @vadmium, @ben-spiller, @ben-spiller

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-27.11:39:42.662>
created_at = <Date 2009-02-06.11:13:43.638>
labels = ['expert-XML', '3.8', 'type-feature', 'library', '3.7']
title = "ElementTree and minidom don't prevent creation of not well-formed XML"
updated_at = <Date 2019-04-27.11:39:42.643>
user = 'https://github.com/ods'

bugs.python.org fields:

activity = <Date 2019-04-27.11:39:42.643>
actor = 'scoder'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-27.11:39:42.662>
closer = 'scoder'
components = ['Library (Lib)', 'XML']
creation = <Date 2009-02-06.11:13:43.638>
creator = 'ods'
dependencies = []
files = []
hgrepos = []
issue_num = 5166
keywords = []
message_count = 11.0
messages = ['81259', '89685', '89699', '95684', '95689', '101158', '111603', '258343', '324922', '328040', '340981']
nosy_count = 12.0
nosy_names = ['effbot', 'ods', 'scoder', 'strangefeatures', 'jwilk', 'eli.bendersky', 'flox', 'nvetoshkin', 'santoso.wijaya', 'martin.panter', 'benspiller', 'Ben Spiller']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue5166'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[ods]

ElementTree and minidom allow creation of not well-formed XML, that
can't be parsed:

>>> from xml.etree import ElementTree
>>> element = ElementTree.Element('element')
>>> element.text = u'\0'
>>> xml = ElementTree.tostring(element, encoding='utf-8')
>>> ElementTree.fromstring(xml)
[...]
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1,
column 9

>>> from xml.dom import minidom
>>> doc = minidom.getDOMImplementation().createDocument(None, None, None)
>>> element = doc.createElement('element')
>>> element.appendChild(doc.createTextNode(u'\0'))
<DOM Text node "">
>>> doc.appendChild(element)
<DOM Element: element at 0xb7ca688c>
>>> xml = doc.toxml(encoding='utf-8')
>>> minidom.parseString(xml)
[...]
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, colum

I believe they should raise some exception when there are characters
not allowed in XML (http://www.w3.org/TR/REC-xml/#NT-Char) are used in
attribute values, text nodes and CDATA sections.

[effbot]

For ET, that's very much on purpose. Validating data provided by every
single application would kill performance for all of them, even if only a
small minority would ever try to serialize data that cannot be represented
in XML.

[ods]

Every blog engine I've even seen so far pass through comments from
untrusted users to RSS/Atom feeds without proper validation causing
broken XML in feeds. Sure, this is a bug in web applications, but DOM
manipulation packages should prevent from creation broken XML to help
detecting errors earlier.

[strangefeatures]

I'm also of the opinion that this would be a valuable feature to have. I
think it's a reasonable expectation that an XML library produces valid
XML. It's particularly strange that ET would output XML that it can't
itself read. Surely the job of making the input valid falls on the XML
creator - that's the point of using libraries in the first place, to
abstract away from details like not being able to use characters in the
0-32 range, in the same way that ampersands etc are auto-escaped.
Granted, it's not as clear-cut here since the low-range ASCII characters
are likely to be less frequent and the strategy to handle them is less
clear. I think the sanest behaviour would be to raise an exception by
default, although a user-configurable option to replace or omit the
characters would also make sense. If impacting performance is a concern,
maybe it would make sense to be off by default, but I would have thought
that the single regex that could perform the check would have relatively
minimal impact - and it seems to be an acceptable overhead on the
parsing side, so why not on generation?

[ods]

Here is a regexp I use to clean up text (note, that I don't touch
"compatibility characters" that are also not recommended in XML; some
other developers remove them too):

# http://www.w3.org/TR/REC-xml/#NT-Char
# Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] |
# [#x10000- #x10FFFF]
# (any Unicode character, excluding the surrogate blocks, FFFE, and
FFFF)
_char_tail = ''
if sys.maxunicode > 0x10000:
_char_tail = u'%s-%s' % (unichr(0x10000),
unichr(min(sys.maxunicode, 0x10FFFF)))
_nontext_sub = re.compile(
ur'[^\\x09\\x0A\\x0D\\x20-\\uD7FF\\uE000-\\uFFFD%s]' %
_char_tail,
re.U).sub
def replace_nontext(text, replacement=u'\uFFFD'):
return _nontext_sub(replacement, text)

[nvetoshkin]

What about this example?
>>> from xml.dom import minidom
>>> doc = minidom.Document()
>>> el = doc.createElement("Test")
>>> el.setAttribute("with space", "False")
>>> doc.appendChild(el)
<DOM Element: Test at 0xba1440>
>>>
>>> #nahhh
... minidom.parseString(doc.toxml())
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "C:\Python26\lib\xml\dom\minidom.py", line 1928, in parseString
    return expatbuilder.parseString(string)
  File "C:\Python26\lib\xml\dom\expatbuilder.py", line 940, in parseString
    return builder.parseString(string)
  File "C:\Python26\lib\xml\dom\expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 33

>>

Is it worth making another bug report?

[BreamoreBoy]

In msg89685 it's stated that this behaviour is deliberate for ET. Could somebody please comment on the minidom aspects.

[vadmium]

bpo-12129 is open about this sort of problem with xml.dom (which would also apply to minidom I think).

If someone wants to suggest a clarification for the Element Tree documentation, that might work. But I tend to agree about not bogging down the implementation.