API-only export option (without Special:Export)

[nemobis]

Wanted for various reasons. Current implementation: --xmlrevisions, false by default. If the default method to download wikis doesn't work for you, please try using the flag --xmlrevisions and let us know how it went.
https://groups.google.com/forum/#!topic/wikiteam-discuss/ba2K-WeRJ-0

Previous takes:
#195
#280

[nemobis]

Does not yet work for Wikia, partly because they return a blank page for exportnowrap used in getXMLHeader(). Have to use wikitools there as well?

  File "./dumpgenerator.py", line 2195, in <module>
    main()
  File "./dumpgenerator.py", line 2187, in main
    createNewDump(config=config, other=other)
  File "./dumpgenerator.py", line 1756, in createNewDump
    generateXMLDump(config=config, titles=titles, session=other['session'])
  File "./dumpgenerator.py", line 717, in generateXMLDump
    for xml in getXMLRevisions(config=config, session=session):
  File "./dumpgenerator.py", line 792, in getXMLRevisions
    for page in result['query']['allrevisions']:
KeyError: 'query'
No </mediawiki> tag found: dump failed, needs fixing; resume didn't work. Exiting.

[nemobis]

Before even downloading the first revisions, there is some wiki where the export gets stuck in an endless loop of "Invalid JSON response. Trying the request again" or similar message:

Analysing http://www.haplozone.net/wiki/index.php
Trying generating a new dump into a new directory...
Retrieving the XML for every page from the beginning
Invalid JSON, trying request again
Invalid JSON, trying request again

[nemobis]

Now tested with a 1.12 wiki,http://meritbadge.org/wiki/index.php/Main_Page , courtesy https://lists.wikimedia.org/pipermail/wikitech-l/2018-May/090004.html : 27cbdfd 680145e

[nemobis]

For Wikia, the API export works without exportnowrap: http://00eggsontoast00.wikia.com/api.php?action=query&prop=revisions&meta=siteinfo&titles=Main%20Page&export&format=json

But facepalm, where the API help says "Export the current revisions of all given or generated pages" it really means that any revision other than the current one is ignored: http://00eggsontoast00.wikia.com/api.php?action=query&revids=3|80|85&export is the same as http://00eggsontoast00.wikia.com/api.php?action=query&revids=85&export

[nemobis]

Here we go: 7143f7e

It's very fast on most wikis, because it makes way less requests if your average number of revisions per page is less than 50.

The first dump produced with this method is: https://archive.org/download/wiki-ferstaberindecom_f2_en/ferstaberindecom_f2_en-20180519-history.xml.7z

[nemobis]

And now also Wikia, without the allrevisions module: https://archive.org/details/wiki-00eggsontoast00wikiacom

The XML built "manually" with --xmlrevisions is almost the same as usual (at the cost of making at least one request per page), but it's missing parentid and at the moment minoredit.

[nemobis]

Analysing http://nimiarkisto.fi/w/api.php
Trying generating a new dump into a new directory...
Loading page titles from namespaces = all
Excluding titles from namespaces = None
29 namespaces found
    Retrieving titles in the namespace 0
.Traceback (most recent call last):
  File "./dumpgenerator.py", line 2288, in <module>
    main()
  File "./dumpgenerator.py", line 2280, in main
    createNewDump(config=config, other=other)
  File "./dumpgenerator.py", line 1844, in createNewDump
    getPageTitles(config=config, session=other['session'])
  File "./dumpgenerator.py", line 416, in getPageTitles
    for title in titles:
  File "./dumpgenerator.py", line 292, in getPageTitlesAPI
    allpages = jsontitles['query']['allpages']
KeyError: 'query'

[nemobis]

In testing this for Wikia, remember that the number of edits on Special:Statistics isn't always truthful (this is normal on MediaWiki). For instance http://themodifyers.wikia.com/wiki/Special:Statistics says 2333 edits, but dumpgenerator.py exports 1864, and that's the right amount: entering all the titles on themodifyers.wikia.com/wiki/Special:Export and exporting all revisions gives the same amount.

Also, a page with 53 revisions on that wiki was correctly exported, which means that API continuation works; that's something!

[nemobis]

Not sure what's going on at http://zh.asoiaf.wikia.com/api.php

Traceback (most recent call last):
  File "./dumpgenerator.py", line 2308, in <module>
    main()
  File "./dumpgenerator.py", line 2300, in main
    createNewDump(config=config, other=other)
  File "./dumpgenerator.py", line 1864, in createNewDump
    getPageTitles(config=config, session=other['session'])
  File "./dumpgenerator.py", line 429, in getPageTitles
    for title in titles:
  File "./dumpgenerator.py", line 252, in getPageTitlesAPI
    config=config, session=session)
TypeError: 'NoneType' object is not iterable
tail: cannot open 'zhasoiafwikiacom-20180521-wikidump/zhasoiafwikiacom-20180521-history.xml' for reading: No such file or directory
No </mediawiki> tag found: dump failed, needs fixing; resume didn't work. Exiting.

http://zhpad.wikia.com/api.php seems to eventually fail as well

[nemobis]

Next step: implementing resuming. I'll probably take the readTitles() part out of getXMLRevisions() to make things clearer.

I think it would be the occasion to make sure that we log something to error.log when we catch an exception or call sys.exit(1), so that it's easier to inspect failed dumps and see what happened when they stopped. I have almost 4k interrupted Wikia dumps.

[nemobis]

Later I'll post a series of errors.log from failed dumps.

For now I tend to believe that, when the dump runs to the end, the XML really is as complete as possible. For instance, on a biggish wiki like http://finalfantasy.wikia.com/wiki/Special:Statistics :

$ grep -c "<revision>" finalfantasywikiacom-20180523-history.xml
1638424
$ grep -c "<page>" finalfantasywikiacom-20180523-history.xml
311259

That's over a million "missing" revisions compared to what Special:Statistics says, which however cannot really be trusted. The number of pages is pretty close.

On the other hand, it could be that the continuation is not working in some cases... In clubpenguinwikiacom-20180523-history.xml, I'm not sure I see the 3200 revisions that the main page ought to have.