xml.etree.ElementTree does not read xml-text over page bonderies

[roland]

BPO	4100
Nosy	@birkenfeld
Files	bug.py bug.xml bug.py fix_cross_boundary_on_ElementTree_v2.patch test_v2.py ElementTree_iterparse_doc.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/birkenfeld'
closed_at = <Date 2009-01-01.11:46:58.240>
created_at = <Date 2008-10-10.14:53:37.836>
labels = ['type-bug', 'docs']
title = 'xml.etree.ElementTree does not read xml-text over page bonderies'
updated_at = <Date 2009-01-01.11:46:58.238>
user = 'https://bugs.python.org/roland'

bugs.python.org fields:

activity = <Date 2009-01-01.11:46:58.238>
actor = 'georg.brandl'
assignee = 'georg.brandl'
closed = True
closed_date = <Date 2009-01-01.11:46:58.240>
closer = 'georg.brandl'
components = ['Documentation']
creation = <Date 2008-10-10.14:53:37.836>
creator = 'roland'
dependencies = []
files = ['11762', '11763', '11764', '11769', '11770', '11925']
hgrepos = []
issue_num = 4100
keywords = ['patch']
message_count = 6.0
messages = ['74635', '74645', '74677', '75447', '75451', '78696']
nosy_count = 4.0
nosy_names = ['effbot', 'georg.brandl', 'ocean-city', 'roland']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4100'
versions = ['Python 2.6', 'Python 2.5', 'Python 3.0']

[roland]

xml text fields are not read properly when it is encountered in a
'start' event.

During a 'start'-event elem.text returns None, if the text string cross
a page boundary of the file. (this is platform dependent and a typical
value is 8K (8192 byte)).

This line cause an error if the page size is 8192.
<a>this is a text where X has position 8192 in the file</a>

In most cases this erroneous behaviour can be avoid when elem.tree
always returns the proper value at the 'end'-event.

Two files are submitted:
bug.py: An excerpted file that produced an error with the submitted xml
file.

bug.xml: An xml file, a little bit more then 8200 bytes. In can of the
page size is greater than 8K.. file should be enlarged. Important is
however that the text should cross the page boundary. Tags and
attributes and attribute values as well are OK

I might have misunderstood the documentation of etree, because there are
situations that I have not tested.
/roland

[ocean-city]

Minimum script to reproduce this issue is "bug.py" I've attached.
And I think this issue can be fixed with
"fix_cross_boundary_on_ElementTree.patch". I'll attach the test case for
this issue as "test.py". (I wanted to intergrate test into
test_xml_etree_c.py, but it uses doctest which I don't know about)

/////////////////////////
// Cause of issue

TreeBuilder#start() and TreeBuilder#end() are handlers driven by
self._parser.feed(data) in iterparse.next(), and iterparse stores
elements returned by these functions.

But element is not initialized at the moment. No one can determine
element.text when start tag is found, and element.tail when end tag is
found vise versa. We can say "the element is initialized" when
encountered next element or TreeBuilder is closed.

So, iterparse's _events queue may contain uninitialized elements, so my
patch waits until the element will be initialized.

[roland]

We had to be careful how we should handle this.

http://effbot.org/zone/element-iterparse.htm

A note on this site says following :

Note: The tree builder and the event generator are not necessarily
synchronized; the latter usually lags behind a bit. This means that when
you get a “start” event for an element, the builder may already have
filled that element with content. You cannot rely on this, though — a
“start” event can only be used to inspect the attributes, not the
element content. For more details, see this
<ref>http://mail.python.org/pipermail/xml-sig/2005-January/010838.html\</ref>.

I do understand that it might be so that elem.text is undefined at start.

I have not investigated how iterparse handle this situation over boundaries:

<a> text <b> text </b> text </a>

[effbot]

Roland's right - "iterparse" only guarantees that it has seen the ">"
character of a starting tag when it emits a "start" event, so the
attributes are defined, but the contents of the text and tail attributes
are undefined at that point. The same applies to the element children;
they may or may not be present.

If you need a fully populated element, look for "end" events instead.

[ocean-city]

I propose to note this behavior on document. I'll attach the patch. (I
just inserted your comment into document)