Download script

[newsch]

Closes #12

Remaining work:

• Add handling for error conditions
• Document config and error codes

[biodranik]

"Latest dumps are already downloaded"?

[newsch]

If URLS is empty, then none of the specified languages could be found for the latest dump.

If a newer dump isn't available, it will still check the sizes of the last downloaded dump, and exit with 0.

[biodranik]

Good! The goal is to make a cron script that will update files automatically when they are published (and delete old files).

Another question: should previously generated HTML and other temporary files be deleted before relaunching the wikiparser? Does it make sense to cover it in the run script?

[newsch]

They shouldn't need to be.

The temporary files are regenerated each time.
The generated HTML will be overwritten if it is referenced in the new planet file.

If an article isn't extracted from the dump due to #24 or something else, then having the old copy still available might be useful.

But if the HTML simplification is changed, and older articles are no longer referenced in OSM, then they will remain on disk unchanged.

[biodranik]

Need to test it on a server )

[biodranik]

nit: Can array be used here without a warning?

[newsch]

To convert it to an array with the same semantics it would need to suppress another warning:

# shellcheck disable=SC2206 # Intentionally split on whitespace.
LANGUAGES=( $LANGUAGES )

[newsch]

I've looked a little into parallel downloads with programs in the Debian repos:

GNU parallel or GNU xargs works, but you lose the progress bar from wget and no indication of how the downloads are doing:

for url in $URLS; do echo "$url"; done | xargs -L 1 -P 3 wget --no-verbose --continue --directory-prefix "$DOWNLOAD_DIR"

aria2c returned protocol errors:

aria2c -x2 -s2 -c -d ~/Downloads/aria-test \
    https://dumps.wikimedia.org/other/enterprise_html/runs/20230801/dewiki-NS0-20230801-ENTERPRISE-HTML.json.tar.gz \
    https://dumps.wikimedia.org/other/enterprise_html/runs/20230801/eswiki-NS0-20230801-ENTERPRISE-HTML.json.tar.gz

08/17 10:35:22 [NOTICE] Downloading 1 item(s)

08/17 10:35:23 [ERROR] CUID#9 - Download aborted. URI=https://dumps.wikimedia.org/other/enterprise_html/runs/20230801/eswiki-NS0-20230801-ENTERPRISE-HTML.json.tar.gz
Exception: [AbstractCommand.cc:351] errorCode=8 URI=https://dumps.wikimedia.org/other/enterprise_html/runs/20230801/eswiki-NS0-20230801-ENTERPRISE-HTML.json.tar.gz
  -> [HttpResponse.cc:81] errorCode=8 Invalid range header. Request: 130154496-14855176191/29631479446, Response: 130154496-14855176191/25080519092

axel only seems to parallelize a single download

wget2 works great out of the box:

wget2 --progress=bar --continue --directory-prefix "$DOWNLOAD_DIR" $URLS

[newsch]

Do you want the script to handle this?

If it will be running on a cron job, then it might be good to keep 2 copies around.
Otherwise the script could delete the last dump as wikiparser is using it?

[biodranik]

1. Aren't files that were open before their deletion on Linux still accessible?
2. Dumps are produced regularly, right? We can set a specific schedule.
3. Script may have an option to automatically delete older dumps.

[newsch]

1. Aren't files that were open before their deletion on Linux still accessible?

You're right, as long as run.sh is started before download.sh deletes them, it will be able to access the files.

2. Dumps are produced regularly, right? We can set a specific schedule.

Yes, they're started on the 1st and the 20th of each month, and finished within 3 days it looks like.

3. Script may have an option to automatically delete older dumps.

👍

[newsch]

I've added a new option:

-D      Delete all old dump subdirectories if the latest is downloaded

[biodranik]

wget2 works great out of the box:

The default behavior can be like this: use wget2 if it's available, and fall back to a single-threaded download while mentioning a speedup with wget2.

Another important question is if it's ok to overload wiki servers with parallel downloads. Can you please ask them to confirm? Maybe they have a single-threaded policy?

[newsch]

Looks like 2 parallel downloads is the max:

If you are reading this on Wikimedia servers, please note that we have rate limited downloaders and we are capping the number of per-ip connections to 2. This will help to ensure that everyone can access the files with reasonable download times. Clients that try to evade these limits may be blocked.

There are at least two mirrors that host some of the latest enterprise dumps:

• (US) https://dumps.wikimedia.your.org/other/enterprise_html/runs/
• (Sweden) https://mirror.accum.se/mirror/wikimedia.org/other/enterprise_html/runs/

[biodranik]

Good, let's track how fast mirrors are updated. We may hardcode or put into readme links to URLs/mirrors and use what is better/faster.

[newsch]

Both of the mirrors have the 2023-08-20 dumps up already.

[biodranik]

Thanks!

1. wget2 doesn't resume interrupted downloads.
2. Don't forget to squash all commits before the merge )

[biodranik]

Will wikiparser generator properly find/load newer versions from the latest dir without specifying explicit file names?

[newsch]

For the run.sh script, you'll provide a glob of the latest directory:

./run.sh descriptions/ planet.osm.pdf $DUMP_DIR/latest/*

It doesn't have any special handling for the $DUMP_DIR layout.

[newsch]

wget2 doesn't resume interrupted downloads

What kind of interruption? It should be able to handle network drops and temporary errors.

[biodranik]

Can spaces be added here?

[newsch]

I haven't seen an example with spaces in the name. All of the browser user agents use CamelCase instead of spaces.

[biodranik]

It would be great to test all these PRs on the server with a real data.