Deny path *.<extension> blocking all indexing for a site

[m-i-l]

A user entered a deny path of *.xml, which seems a reasonable way to express "don't index any XML files". However, the deny parameter on LinkExtractor interprets this is such a way that nothing on the site is indexed at all. I think this is because the regex is interpreted as *. which means everything, although escaping it with *. doesn't appear to resolve the issue.

[m-i-l]

For now I'm catching and replacing *.xml with .xml$ which works, although it might need a more robust solution.

[m-i-l]

Another user has set deny paths including '*.json' and '*.atom' which has blocked indexing again. Reopening. Going to need a more robust solution.

[m-i-l]

Implemented more robust solution using the regex exclusion_value = re.sub('^\*\.(\w+)$', r'.\1$', exclusion_value) to replace '.xml' with '.xml$', '.json' with '.json$' etc.