xml.etree.ElementTree accepts control chars.

[mmaker]

BPO	18850
Nosy	@scoder, @tiran, @bitdancer, @mmaker, @serhiy-storchaka
Superseder	bpo-5166: ElementTree and minidom don't prevent creation of not well-formed XML
Files	test.py inject.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-09-02.21:19:38.862>
created_at = <Date 2013-08-27.09:50:51.437>
labels = ['expert-XML', 'type-feature', 'library']
title = 'xml.etree.ElementTree accepts control chars.'
updated_at = <Date 2013-09-02.21:19:38.861>
user = 'https://github.com/mmaker'

bugs.python.org fields:

activity = <Date 2013-09-02.21:19:38.861>
actor = 'eli.bendersky'
assignee = 'none'
closed = True
closed_date = <Date 2013-09-02.21:19:38.862>
closer = 'eli.bendersky'
components = ['Library (Lib)', 'XML']
creation = <Date 2013-08-27.09:50:51.437>
creator = 'maker'
dependencies = []
files = ['31482', '31484']
hgrepos = []
issue_num = 18850
keywords = []
message_count = 30.0
messages = ['196271', '196277', '196278', '196280', '196284', '196286', '196288', '196294', '196296', '196298', '196313', '196315', '196328', '196360', '196361', '196362', '196364', '196365', '196366', '196367', '196368', '196369', '196373', '196379', '196441', '196488', '196510', '196756', '196797', '196808']
nosy_count = 7.0
nosy_names = ['mmokrejs', 'scoder', 'christian.heimes', 'r.david.murray', 'eli.bendersky', 'maker', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '5166'
type = 'enhancement'
url = 'https://bugs.python.org/issue18850'
versions = ['Python 3.4']

[mmaker]

Got from irc;
<JerryKwan> python bug in xml.etree.ElementTree, from version 2.7 to 3.2 http://www.reddit.com/r/Python/comments/1l6cta/python_bug_in_xmletreeelementtree/

I think we should keep consistency with lxml and forbid control chars in advance.
<https://en.wikipedia.org/wiki/Valid_characters_in_XML#XML_1.1\>

Attaching the test file proving the issue, slightly modified to work on 2.7 and 3.x

[scoder]

This is a bit tricky in ET because it generally allows you to stick anything into the Element properties (and that's a feature). So catching this at tree building time (as lxml.etree does) isn't really possible.

However, at least catching it in the serialiser should be possible and would help. Regexes for well-formed tag names and text could do the job.

[mmaker]

you mind if I try by myself to provide patch and unittest in the next few days?

[scoder]

Go for it. That's usually the fastest way to get things done.

[bitdancer]

Unless it is a security issue, this seems like the kind of fix that shouldn't be applied to maintenance releases.

[mmaker]

I suppose it is, David, if in 2 minutes flat I can change your terminal name.

[bitdancer]

In that case, the fix needs to be applied to 3.2 and 2.6 as well. Or at least considered for application. It could be that this will break working (though dangerous) programs. I'll leave it to folks more knowledgeable in this particular area than I to decide where the severity/backward compatibility line lies in this case.

[serhiy-storchaka]

See also bpo-7727. Almost any other XML generation code (xml.sax.sautils.XMLGenerator, xml.dom.minidom.Element.writexml(), etc, but not plistlib.PlistWriter) has the same problem.

The problem with filtering control characters is that it will significantly slowdown XML generation.

[scoder]

Michele, could you elaborate how you would exploit this issue as a security risk?

I mean, I can easily create a (non-)XML-document with control characters manually, and the parser would reject it.

What part of the create-to-serialise process exactly is a problem here?

[mmaker]

Michele, could you elaborate how you would exploit this issue as a security risk?
Sure. What I meant in my message is: assume you have a script that simply stores each message it receives (from stdin, from a tcp stream, whatever) inside an xml tree like
'<text>{message1}</text><text>{message2}<text>',
and prints the tree on SIGINT.

What I would expect is the xml document not to allow control chars, as "restricted and discouraged", and consistent with lxml.
What instead happens is that the control chars are not handled, and thus anybody can send control chars in my terminal. Changing the terminal title is a trivial example of those.

For sure an echo server may have the same issue, but the premises are different, because I expect to print just a byte stream.
Mentioning this fact in the documentation may be a possible solution, but I believe more that keeping consistency with lxml is the right way.

I mean, I can easily create a (non-)XML-document with control characters manually, and the parser would reject it.
False? The parser is *not* rejecting control chars.

What part of the create-to-serialise process exactly is a problem here?
ElementTree.tostring().