xml.sax and xml.dom fetch DTDs by default

[akuchling]

BPO	2124
Nosy	@loewis, @akuchling, @devdanzin

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-01-13.21:29:50.177>
created_at = <Date 2008-02-15.14:42:54.826>
labels = ['expert-XML', 'performance']
title = 'xml.sax and xml.dom fetch DTDs by default'
updated_at = <Date 2013-02-28.01:32:10.423>
user = 'https://github.com/akuchling'

bugs.python.org fields:

activity = <Date 2013-02-28.01:32:10.423>
actor = 'rsandwick3'
assignee = 'none'
closed = True
closed_date = <Date 2012-01-13.21:29:50.177>
closer = 'loewis'
components = ['XML']
creation = <Date 2008-02-15.14:42:54.826>
creator = 'akuchling'
dependencies = []
files = []
hgrepos = []
issue_num = 2124
keywords = []
message_count = 31.0
messages = ['62430', '62431', '62446', '62452', '62475', '62478', '62483', '62486', '62512', '62515', '62787', '62790', '62975', '81080', '81091', '81102', '81103', '81104', '81106', '81108', '81111', '81114', '81122', '110894', '110899', '151191', '151195', '151196', '151197', '151204', '183197']
nosy_count = 10.0
nosy_names = ['loewis', 'akuchling', 'pboddie', 'exarkun', 'ajaksu2', 'vdupras', 'damien', 'BreamoreBoy', 'Brian.Visel', 'rsandwick3']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue2124'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[akuchling]

The W3C posted an item at
http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic
describing how their DTDs are being fetched up to 130M times per day.

The Python parsers are part of the problem, as
noted by Paul Boddie on the python-advocacy list:

There are two places which stand out:

xml/dom/xmlbuilder.py
xml/sax/saxutils.py

What gives them away is the way as the cause of the described problem is
that
they are both fetching things which are given as "system identifiers" - the
things you get in the document type declaration at the top of an XML
document
which look like a URL.

If you then put some trace statements into the code and then try and parse
something using, for example, the xml.sax API, it becomes evident that by
default the parser attempts to fetch lots of DTD-related resources, not
helped by the way that stuff like XHTML is now "modular" and thus employs
lots of separate files in the DTD. This is obvious because you get
something
like this printed to the terminal:

saxutils: opened http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-inlstyle-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-framework-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-datatypes-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-qname-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-events-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-attribs-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml11-model-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-charent-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-lat1.ent
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-symbol.ent
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-special.ent
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-text-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-inlstruct-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-inlphras-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-blkstruct-1.mod
saxutils: opened http://www.w3.org/MarkUp/DTD/xhtml-blkphras-1.mod

Of course, the "best practice" with APIs like SAX is that you define
your own
resolver or handler classes which don't go and fetch DTDs from the W3C all
the time, but this isn't the "out of the box" behaviour. Instead,
implementers have chosen the most convenient behaviour which arguably
involves the least effort in telling people how to get hold of DTDs so that
they may validate their documents, but which isn't necessarily the "right
thing" in terms of network behaviour. Naturally, since defining specific
resolvers/handlers involves a lot of boilerplate (and you should try it in
Java!) then a lot of developers just incur the penalty of having the
default
behaviour, instead of considering the finer points of the various W3C
specifications (which is never really any fun).

Anyway, I posted a comment saying much the same on the blog referenced
at the
start of this thread, but we should be aware that this is default standard
library behaviour, not rogue application developer behaviour.

[akuchling]

Here's a simple test to demonstrate the problem:

from xml.sax import make_parser
from xml.sax.saxutils import prepare_input_source
parser = make_parser()
inp = prepare_input_source('file:file.xhtml')
parser.parse(inp)

file.xhtml contains:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"\>
<html xmlns="http://www.w3.org/1999/xhtml" />

If you insert a debug print into saxutils.prepare_input_source,
in the branch which uses urllib.urlopen(), you get the above list of
inputs accessed: the XHTML 1.1 DTD, which is nicely modular and pulls in
all those other files.

I don't see a good way to fix this without breaking backward
compatibility to some degree. The
external-general-entities features defaults to 'on', which enables this
fetching; we could change the default to 'off', which would save the
parsing effort, but would also mean that entities like é weren't
defined.

If we had catalog support, we could ship the XHTML 1.1 DTDs and any
other DTDs of wide usage, but we don't.

[loewis]

On systems that support catalogs, the parsers should be changed to
support public identifiers, using local copies of these DTDs.

However, I see really no way how the library could avoid resolving the
DTDs altogether. The blog is WRONG in claiming that the system
identifier is not a downloadable URL, but a mere identification (the
namespace is a mere identification, but our parsers would never try to
download anything). The parser needs the DTDs in case external entities
occur in the document (of course, download could be delayed until the
first reference occurs).

[vdupras]

The blog page talked about 503 responses. What about issuing a warning
on these responses? Maybe it would be enough to make developers aware of
the problem?

Or what about in-memory caching of the DTDs? Sure, it wouldn't be as
good as a catalog or anything, but it might help for the worst cases?

[pboddie]

(Andrew, thanks for making a bug, and apologies for not reporting this
in a timely fashion.)

Although an in-memory caching solution might seem to be sufficient, if
one considers things like CGI programs, it's clear that such programs
aren't going to benefit from such a solution. It would be interesting to
know what widely deployed software does use the affected parsers,
though. A Google code search might be helpful.

I think that the nicest compatible solution would be to have some kind
of filesystem cache for the downloaded resources, but I don't recall any
standard library caching solution of this nature. Things like being able
to write to a known directory, perhaps using the temporary file APIs
which should work even as a "very unprivileged" user, would be useful
properties of such a solution.

[akuchling]

What if we just tried to make the remote accesses apparent to the user,
by making a warning.warn() call in the default implementation that was
deactivated by a setFeature() call. With a warning, code will continue
to run but the user will at least be aware they're hitting a remote
resource, and can think about it, even if they decide to suppress the
warning.

We should also modify the docs to point this out; it's not likely to
help very much, but it's still worth doing.

[loewis]

-1 on issuing a warning. I really cannot see much of a problem in this
entire issue. XML was designed to "be straightforwardly usable over the
Internet" (XML rec., section 1.1), and this issue is a direct
consequence of that design decision. You might just as well warn people
against using XML in the first place.

[vdupras]

-1 on the systematic warnings too, but what I was talking about is a
warning that would say "The server you are trying to fetch your resource
from is refusing the connection. Don't cha think you misbehave?" only on
5xx and 4xx responses, not on every remote resource fetching.

[devdanzin]

Martin, I agree that simply not resolving DTDs is an unreasonable
request (and said so in the blog post). But IMHO there are lots of
possible optimizations, and the most valuable would be those darn easy
for newcomers to understand and use.

In Python, a winning combo would be an arbitrary (and explicit) FS
"dtdcache" that people could use with simple a drop-in import (from a
third-party module?). Perhaps the cache lives in a pickled dictionary
with IDs, checksums and DTDs. Could also be a sqlite DB, if updating the
dict becomes problematic.

In that scenario, AMK could save latter W3C hits with:

from xml.sax import make_parser
from dtdcache.sax.saxutils import prepare_input_source # <- dtdcache
parser = make_parser()
inp = prepare_input_source('file:file.xhtml', cache="/tmp/xmlcache")

It might be interesting to have read-only, force-write and read-write
modes. Not sure how to map that on EntityResolver and DTD consumers (I'm
no XML user myself).

Regarding the std-lib, I believe effective caching hooks for DTDs trump
implementing in-memory or sqlite/FS. IMNSHO, correct, accessible support
for catalogs shouldn't be the only change, as caching should give better
performance on both ends.

[loewis]

In Python, a winning combo would be an arbitrary (and explicit) FS
"dtdcache" that people could use with simple a drop-in import (from a
third-party module?).

It's indeed possible to provide that as a third-party module; one would
have to implement an EntityResolver, and applications would have to use
it. If there was a need for such a thing, somebody would have done it
years ago.

It might be interesting to have read-only, force-write and read-write
modes.

Please take a look at catalogs - they are a read-only repository for
external entities (provided those entities have a public identifier,
which the W3C DTDs all have).

Regarding the std-lib, I believe effective caching hooks for DTDs trump
implementing in-memory or sqlite/FS.

So then nothing needs to be done - the hooks have been in place since
Python 2.0.

[akuchling]

The solution of adding caching, If-Modified-Since, etc. is a good one,
but I quail in fear at the prospect of expanding the saxutils resolver
into a fully caching HTML agent that uses a cache across processes. We
should really be encouraging people to use more capable libraries such
as httplib2 (http://code.google.com/p/httplib2/), but this is slightly
at war
with the batteries-included philosophy.

So, I propose we:

• add warnings to the urllib, urllib2, saxutil module docs that parsing
  can retrieve arbitrary resources over the network, and encourage the
  user to use a smarter library such as httplib2.
• update the urllib2 HOWTO to mention this.

I'm willing to do the necessary writing.

[loewis]

I may have lost track somewhere: what does have urllib* to do with this
issue?