Import Wikipedia abstracts

[sylvinus]

We will end up importing the whole Wikipedia dumps soon enough, however a simple first step would be to import the abstracts, after #10 is finished.

That could allow us to add good descriptions, possibly of better quality than DMOZ. That would be helpful for commonsearch/cosr-results#1 for instance.

We could also possibly start including all wikipedia urls in the results, even if we didn't index their whole content yet.

https://meta.wikimedia.org/wiki/Data_dumps

There seems to be a combined abstract.xml file ("Recombine extracted page abstracts for Yahoo"), is this the one we should use?
https://dumps.wikimedia.org/enwiki/20160204/

[Tpt]

The "Recombine extracted page abstracts for Yahoo" quality seems very low (a lot of summary are actually infobox invocations) and is polluted with redirection. And I don't manage to find an other dump with article abstracts.

The relevant REST API call to get summary and image for an article is the /page/summary operation of the REST API: https://rest.wikimedia.org/fr.wikipedia.org/v1/?doc#/

[sylvinus]

Thanks for looking into it! Too bad for the abstracts dump.

We can't really use the REST API because we want all articles at once.

I guess we'll have to use the larger dumps. The solutions I can think of:

• Use the zim HTML dumps http://www.kiwix.org/wiki/Content_in_all_languages
• Use the wikipedia XML dumps and transform them into HTML with Parsoid. It seems that Parsoid relies on a Wikimedia webservice for template expansion though?
• Wait for the official Wikipedia HTML dumps to come through: https://phabricator.wikimedia.org/T93396
• Use the Elasticsearch dumps directly http://dumps.wikimedia.org/other/cirrussearch/20160307/

Any other ideas?

[wumpus]

I've downloaded the SQL dumps and used them directly, it's quick to download and not so hard to parse, although you do need to get smart about the HTML which is in the "text". (I can contribute perl code that does it...) blekko got pretty good "abstracts" by just extracting the first couple of sentences.

[sylvinus]

There has been some recent activity on Wikimedia's side on the HTML dumps: https://phabricator.wikimedia.org/T133547

I hope they can go through so that we can index not only the abstracts but the whole articles as well.

In the meantime, either the SQL or the ZIM dumps seem like the best bets. @wumpus feel free to share your perl code, I'd be curious to see the hacks you did :)

[wumpus]

Hack alert!

extract-raw-wikipedia-articles.pl.txt

[sylvinus]

Hack alert indeed!! It's almost as scary as I imagined ;-)

I just merged support for pluggable document sources so it should be very easy to write one that reads the .zim files and see how it behaves.