Don't download dump files that are not done yet

[vrandezo]

E.g. using wdtk-example, it is right now already downloading the April 20 dump, but the dump is not fully generated yet. Can we first check whether a dump is complete and available before starting to download it?

[mkroetzsch]

I thought that we are doing this. The script always fetches the maxrevid before attempting a download, and it only starts if this id can be found. There are also some kind of checks for the "done" status independent of this, but maybe there is a gap there in one case. What kind of dump are you talking about, daily or current or full?

[vrandezo]

It was the current one.

[guenthermi]

I got the same problem by testing my json-serializer example code. At first I got the error that there is no maximal revision id. Later it downloaded the incomplete dump and reported that it finished the processing after downloading and processing only 160 MB of the dump file (wikidatawiki-20140420-pages-meta-current).

[mkroetzsch]

Confirmed. We have code that checks the md5sums to see if a dump is done, but the download does not use this and relies on the maxrevid alone. My assumption was that the maxrevid is not published before the dump is done, but that seems to be wrong.

[vrandezo]

Thx!