Update the XML security documentation

[hedsnz]

Documentation

The XML security documentation could do with some clarifications.

Some classes of vulnerability are mentioned once with no further explanation

The documentation states:

An attacker can abuse XML features to carry out denial of service attacks, access local files, generate network connections to other machines, or circumvent firewalls.

It then goes on to list the various types of DoS (e.g., billion laughs), and how they are mitigated with Expat >= 2.6.0. However, the three other classes of vulnerability that are mentioned - "access local files, generate network connections to other machines, or circumvent firewalls" - are not discussed further. It's not clear whether up-to-date versions of Python, which include libexpat >= 2.6.0, are vulnerable to those attacks. There's no discussion on how you might verify whether your code is vulnerable to those attacks, how to prevent them, etc.

Of course, Python isn't responsible for ensuring that anyone writing Python is doing so safely, but I think it's unusual to mention a series of (quite serious) vulnerabilities without any further indication of whether they're legitimate concerns, and potentially what to do about them.

Clarify the role of libexpat

It is mentioned that:

Expat versions lower that 2.6.0 may be vulnerable to “billion laughs”, “quadratic blowup” and “large tokens”. Python may be vulnerable if it uses such older versions of Expat as a system-provided library. Check pyexpat.EXPAT_VERSION.

It would be useful to add (from my understanding, which may be wrong):

• libexpat is the default XML parser. If you do not do anything unusual, then you will be using libexpat for parsing XML.
• libexpat is bundled with Python.
• Therefore (I assume), specific Python versions ensure that libexpat >= 2.6.0 is used. If that's true, it may be helpful to to add that specific versions of Python are not vulnerable to those attacks, if they're downloaded from python.org directly, i.e., if they're not sourced from some third-party who has compiled Python with a different version.

Clarify system-provided vs bundled libexpat

Related to the previous point.

A system-provided library is mentioned, but it's not discussed in what circumstances that would be used instead of the Python-bundled libexpat. Presumably the default is to use Python-bundled libexpat, and I haven't been able to find a way to use a system version at runtime; is that only possible if you're compiling Python from source? If that's the case, it would be useful to include a note about that.

I'm happy to create a PR that includes some of these clarifications, but would appreciate getting some feedback on my assumptions first, since I obviously don't know much about this.

Linked PRs

• gh-139313: Improve docs on XML security #139460

[picnixz]

It would be useful to add (from my understanding, which may be wrong):

• we already say that: "The Expat parser is included with Python, so the xml.parsers.expat module will always be available."
• libexpat (the sources) are not always bundled with Python; it depends on the build configuration. This is explained in https://docs.python.org/3/using/configure.html#cmdoption-with-system-expat.
• libexpat >= 2.6.0 is not necessarily used.

We can now explain that Expat 2.7.2 and later should be safe against some attacks where the allocated memory is astronomically large but that wouldn't trigger the billion laughs protection.

A system-provided library is mentioned, but it's not discussed in what circumstances that would be used instead of the Python-bundled libexpat

It's only if you compile Python from source. But this is some "advanced" usage. It's a "use at your own risk" feature because we can't guarantee that the system expat lib will have the expected protections (we don't do a "version check").

Of course, Python isn't responsible for ensuring that anyone writing Python is doing so safely, but I think it's unusual to mention a series of (quite serious) vulnerabilities without any further indication of whether they're legitimate concerns, and potentially what to do about them.

Indeed, and I don't want to do it in the docs. We could add a link to the known XML vulnerabilities in general (e.g., https://cheatsheetseries.owasp.org/cheatsheets/XML_Security_Cheat_Sheet.html) but it would be too much to explain if Expat is vulnerable to each of them.

@hartwork do you have a list of which vulnerabilities (in the list provided by OWASP) that Expat is vulnerable to/safe against?

[hartwork]

@hartwork do you have a list of which vulnerabilities (in the list provided by OWASP) that Expat is vulnerable to/safe against?

@picnixz I'm not sure I understand the core question yet. Could you elaborate or put it differently?

@hedsnz would you be up for jumping on a voice call in English or German with me on this? I'm happy to try answer and better understand your key questions, but feel like this could end up with a lot of typing if we stick to writing. if that's an option, my profile has my e-mail and we can find a time that works well for you.
PS: Are you familiar with https://github.com/tiran/defusedxml/blob/main/README.md ? It touches up on many related topics, e.g. even hash flooding (that Expat is safe from since it started using SipHash).

[picnixz]

Could you elaborate or put it differently?

To be precise, do you know which vulnerabilities Expat could be affected by among those listed in https://cheatsheetseries.owasp.org/cheatsheets/XML_Security_Cheat_Sheet.html? I'm not fond of providing an exhaustive list of vulnerabilities, just the common ones (and mainly those whose consequences is a DoS).

PS: Are you familiar with tiran/defusedxml@main/README.md ? It touches up on many related topics, e.g. even hash flooding (that Expat is safe from since it started using SipHash).

Yes, but we actually removed this table (I think) in #135294.

[hartwork]

Could you elaborate or put it differently?

To be precise, do you know which vulnerabilities Expat could be affected by among those listed in https://cheatsheetseries.owasp.org/cheatsheets/XML_Security_Cheat_Sheet.html?

@picnixz for the denial of service ones, the latest release of Expat provides the best protection of all releases and the common attack variants are covered, but just today I received an e-mail that tells me that we are not done on the denial-of-service front yet. Regarding attacks accessing local or remote files, Expat does not do either by itself and the user of Expat would need to bring and register their own handler (via XML_SetExternalEntityRefHandler) to "make it vulnerable". So e.g. what OWASP lists as Reflected File Retrieval and Server Side Request Forgery is not possible by default, unless some code on top of Expat explicitly "makes" a new hole like that and then the vulnerability is not in Expat but it's application.

Does that answer the question?

I find that https://docs.python.org/3/library/pyexpat.html#xml.parsers.expat.xmlparser.ExternalEntityRefHandler does not mention implications of implementing such a handler, so maybe the docs of that function would be the key place to warn about things.

I'm not fond of providing an exhaustive list of vulnerabilities, just the common ones (and mainly those whose consequences is a DoS).

I understand.

PS: Are you familiar with tiran/defusedxml@main/README.md ? It touches up on many related topics, e.g. even hash flooding (that Expat is safe from since it started using SipHash).

Yes, but we actually removed this table (I think) in #135294.

Good to know!

There is a typo in there that Git main still has. How do I create a mini pull request without a news file and a related GitHub issue? I failed when I tried last time. Would be great to know. Do you happen to have a link on how that is done?

[hartwork]

CC @hannob

[picnixz]

Does that answer the question?

Yes.

I find that docs.python.org/3/library/pyexpat.html#xml.parsers.expat.xmlparser.ExternalEntityRefHandler does not mention implications of implementing such a handler, so maybe the docs of that function would be the key place to warn about things.

Yes that would be great.

There is a typo in there that Git main still has. How do I create a mini pull request without a news file and a related GitHub issue? I failed when I tried last time. Would be great to know. Do you happen to have a link on how that is done?

Use an old issue (you can reuse the one that removed the table) and ping me. We'll add a skip news label. In the future, if it's really just a simple typo fix but you can't find an appropriate issue, just make your PR and add some fix typo in the title so that triagers cann add skip news and skip issue labels. There is no "automated" way to do it though.

[hartwork]

@picnixz thank you!

[hartwork]

@picnixz does this work? #139335

[hedsnz]

Thanks so much for all the quick responses.

• we already say that: "The Expat parser is included with Python, so the xml.parsers.expat module will always be available."

Respectfully, I disagree that saying that the Expat parser is included with Python implies that (bundled) Expat is the default XML parser. Those are two different things; the former might imply the latter to some people but it didn't necessarily follow for me. For example, it seems plausible for software to use a system library as the default, but also include a bundled version of the library as a fallback if the system version is not available. To that point, the current documentation states:

Python may be vulnerable if it uses such older versions of Expat as a system-provided library.

So we've mentioned that Python could feasibly use a system-provided version of Expat, but we haven't described how that situation might occur. It's possible to find that information if you look at the Python build documentation that you link to, but (IMO) that's not particularly easily discoverable if you aren't familiar with this; and given that the version of Expat is critical to the viability of these potential attacks, I think it's an important point to make in the XML security section.

Regarding attacks accessing local or remote files, Expat does not do either by itself and the user of Expat would need to bring and register their own handler (via XML_SetExternalEntityRefHandler) to "make it vulnerable". So e.g. what OWASP lists as Reflected File Retrieval and Server Side Request Forgery is not possible by default, unless some code on top of Expat explicitly "makes" a new hole like that and then the vulnerability is not in Expat but it's application.

Thank you, this is exactly the kind of information that I'm hoping we can include in Python's XML security documentation. Or at least a link to relevant Expat documentation of this nature.

Putting everything together, here are some tentative changes:

Existing section

It is important to note that modules in the xml package require that there be at least one SAX-compliant XML parser available. The Expat parser is included with Python, so the xml.parsers.expat module will always be available.

Proposed update

It is important to note that modules in the xml package require that there be at least one SAX-compliant XML parser available. Python uses Expat as the default XML parser. Expat is included with Python, so the xml.parsers.expat module will always be available.

Existing section

An attacker can abuse XML features to carry out denial of service attacks, access local files, generate network connections to other machines, or circumvent firewalls.

Expat versions lower that 2.6.0 may be vulnerable to “billion laughs”, “quadratic blowup” and “large tokens”. Python may be vulnerable if it uses such older versions of Expat as a system-provided library. Check pyexpat.EXPAT_VERSION.

Proposed update

An attacker can abuse XML features to carry out denial of service attacks, access local files, generate network connections to other machines, or circumvent firewalls.

The viability of these attacks can depend on the version of Expat used. Given that it's possible to build Python using a system version of Expat, rather than the Python-bundled version, check pyexpat.EXPAT_VERSION to see the version of Expat used by Python.

For example, Expat versions lower than 2.6.0 may be vulnerable to “billion laughs”, “quadratic blowup” and “large tokens”. It is also possible to configure Expat to parse external entities, which may allow attackers to access local files or generate network connections; however, this is not possible when using the default handler.

See <link to Expat docs perhaps?> for more detail on XML security using Expat.

I'm not particularly sold on the above wording, but the main point I'm trying to get across is that "...access local files, generate network connections to other machines, or circumvent firewalls" could use some further explanation. And I think @hartwork's explanation above is pretty much exactly what I was hoping to clarify.

@hartwork - happy to jump on a call if the above is still unclear, let me know. Also, thanks for bringing up defusedxml. That package has great documentation about potential XML attacks, but - as someone without much XML security experience - it also probably adds some confusion to the mix. For example, a bunch of SAST and linting tools still recommend replacing standard Python xml with defusedxml, but it's not clear to me that defusedxml was originally intended to be something that people relied on (instead of, e.g., a great piece of documentation about XML security considerations and some practical mitigations). In particular, it's not clear which (if any) of the discussed attacks are still possible with the latest versions of Python and Expat.

Thanks!

[hartwork]

Thanks so much for all the quick responses.

• we already say that: "The Expat parser is included with Python, so the xml.parsers.expat module will always be available."

Respectfully, I disagree that saying that the Expat parser is included with Python implies that (bundled) Expat is the default XML parser.

@hedsnz I agree.

Those are two different things; the former might imply the latter to some people but it didn't necessarily follow for me. For example, it seems plausible for software to use a system library as the default, but also include a bundled version of the library as a fallback if the system version is not available.

I agree. (There seems to be no universal standard whether systemwide or bundled would be default. From a security point of view, I would argue you always want to (1) use systemwide Expat everywhere and (2) but sure that that single copy is not missing any security updates.)

To that point, the current documentation states:

Python may be vulnerable if it uses such older versions of Expat as a system-provided library.

So we've mentioned that Python could feasibly use a system-provided version of Expat, but we haven't described how that situation might occur. It's possible to find that information if you look at the Python build documentation that you link to, but (IMO) that's not particularly easily discoverable if you aren't familiar with this; and given that the version of Expat is critical to the viability of these potential attacks, I think it's an important point to make in the XML security section.

I'm personally not sure how far this should go, e.g. if these docs start to talk about things like Homebrew and that (most?) Linux distro packages of CPython will rip out and/or bypass any bundles — Debian and Gentoo for sure —then at some point, it will turn into a book and no longer be able to quickly answer questions because of volume. So there has to be a sweet spot of "enough" somewhere and maybe that point needs a concrete persona (or personas) of reader(s) in mind and what they may already know or not know. Or maybe parts can be offloaded elsewhere through a long-term-available and quality content link but I'm not sure where.

Existing section

It is important to note that modules in the xml package require that there be at least one SAX-compliant XML parser available. The Expat parser is included with Python, so the xml.parsers.expat module will always be available.

Proposed update

It is important to note that modules in the xml package require that there be at least one SAX-compliant XML parser available. Python uses Expat as the default XML parser. Expat is included with Python, so the xml.parsers.expat module will always be available.

• Regarding "default parser" — is there even any other XML parser in CPython other than Expat? If not, I'm not sure "default XML parser" would be spot-on concept-wise. If there are others, what are they? I didn't find libxml2 in here from a quick look.
• This adaptation does not say that use of the bundle (rather than systemwide) would be default. my understanding was that that's something you want to address.

Existing section

An attacker can abuse XML features to carry out denial of service attacks, access local files, generate network connections to other machines, or circumvent firewalls.

Expat versions lower that 2.6.0 may be vulnerable to “billion laughs”, “quadratic blowup” and “large tokens”. Python may be vulnerable if it uses such older versions of Expat as a system-provided library. Check pyexpat.EXPAT_VERSION.

Proposed update

An attacker can abuse XML features to carry out denial of service attacks, access local files, generate network connections to other machines, or circumvent firewalls.

The viability of these attacks can depend on the version of Expat used. Given that it's possible to build Python using a system version of Expat, rather than the Python-bundled version, check pyexpat.EXPAT_VERSION to see the version of Expat used by Python.

Two thoughts on how the replacement is missing parts of the picture:

• There are distros that backport security fixes where pyexpat.EXPAT_VERSION will be misleading. Examples are RHEL (RedHat Enterprise Linux) and Debian stable. For SUSE I would need to check to be sure.
• As mentioned above, in many (or most?) context the distributor builds Python rather than the end user. Currently the addition seems to serve completeness but not necessarily help a specific user group. Those compiling manual will need familiarize themselves with the build system in more detail anyway, just my 2 cents.

For example, Expat versions lower than 2.6.0 may be vulnerable to “billion laughs”, “quadratic blowup” and “large tokens”. It is also possible to configure Expat to parse external entities, which may allow attackers to access local files or generate network connections; however, this is not possible when using the default handler.

The last sentence I find not so fortunate, and I would flip the general approach around a la "unless you do X, you're secure".
Maybe I should note that once we're at sentence level, review should move to a pull request if you're open to create one.

@hartwork - happy to jump on a call if the above is still unclear, let me know.

I sent a message to you on the call subject via https://heds.nz/about/ just now.

[picnixz]

Respectfully, I disagree that saying that the Expat parser is included with Python implies that (bundled) Expat is the default XML parser

I'm ok for "Python uses Expat as the default XML parser" as it's a small change.

So we've mentioned that Python could feasibly use a system-provided version of Expat, but we haven't described how that situation might occur.

The default build settings are the safest. It would be a footgun to actually tell users that are worried about security that they could also configure something by themselves. So no, I don't want to (re-)mention the system-wide possible configuration (and it's only possible if you build it from source, which is not the case for most users, and impossible on some platforms).

Note that it's not the only way to actually have another expat version; one could also provide an ABI-compatible copy of the extension module (extension modules are like shared libraries). Thus, the best is to make them aware of the version check (your updated paragraph is fine but I'll remove the mention to system-wide expat). Historically, users asked a bundled version of Expat because the system-wide installation was not updated on time.

however, this is not possible when using the default handler.

In general, defaults are meant to be safe. And if they are not, then we mention it. Not the other way around. Otherwise the docs would blow up. I would prefer changing the docs for the external entity handler as Sebastian suggested:

I find that docs.python.org/3/library/pyexpat.html#xml.parsers.expat.xmlparser.ExternalEntityRefHandler does not mention implications of implementing such a handler, so maybe the docs of that function would be the key place to warn about things.

---

Keep in mind that the policy of the Python docs is not to make users understand all security issues, neither to be overly verbose (especially if the information can be found elsewhere), but rather make them aware of their existence and the some good practices (e.g., checking the expat version is the only practice I would recommend in genreal; don't use your system expat which may be outdated). For users that want more, we can add references but I don't want this page to become a full-fledged article.