replaceExternalLinks removes wrong closing square bracket with nested internal link

[spyysalo]

To replicate:

replaceExternalLinks('[http://example.org [[internal|link]] text]')

result:

'[[internal|link] text]'

expected:

'[[internal|link]] text'

This causes further breakage in downstream processing.

Internal links within external links appear with some frequency in the raw Wikipedia data and are reasonably resolved by MediaWiki. For example,

[http://www.palaeos.org/Proteobacteria Proteobacteria information from [[Palaeos]].]

occurs in the source for https://en.wikipedia.org/wiki/Proteobacteria#External_links .

(Found when trying to fix #54, which previously masked this issue.)

[spyysalo]

I looked at this a bit more, and the issue appears to be this regex: https://github.com/attardi/wikiextractor/blob/master/WikiExtractor.py#L2039, in particular the part

([^\]\x00-\x08\x0a-\x1F]*?)\]

which eats left square brackets (only ] appears in the negated character class) and terminates on the first closing bracket.

It should be possible to fix with the following pattern (here writingBADCHARS for \x00-\x08\x0a-\x1F for readability)

((?:[^][BADCHARS]|\[\[[^][BADCHARS]*\]\])*?)\]

Would you consider a pull request implementing this?

[attardi]

Alternatively one may look for the first ] which is not followed by another ], i.e.:

'](?!])'

Please try it and let me know.