[1.34.0] mirrorbrain detects and rejects downloads through aria2, despite disguise as wget

[hartwork]

Hi!

I'm using aria2 to download source tarballs (of GNOME Meld) with fetchcommandwrapper. The precise call to aria2 is this:

/usr/bin/aria2c -d /usr/portage/distfiles -o meld-3.18.3.tar.xz --allow-overwrite=true --max-tries=5 --max-file-not-found=2 --user-agent=Wget/1.19.1 --split=1 --max-connection-per-server=1 --uri-selector=inorder https://download.gnome.org/sources/meld/3.18/meld-3.18.3.tar.xz

Interestingly, the GNOME server runs mirrorbrain, detects aria2c somehow (despite the cheap disguise as wget) and returns metalink XML rather than the .xz file I requested. Does not happen with actual wget 1.19.5. Bumping the disguise user agent version to 1.19.5 does not help or change anything I would notice.

Because I'm only using a single connection and a single split, that use of aria2 seems perfectly ethical to me and don't really appreciate being blocked for something ethical. I wonder if you know about this behaviour and what you think about it.

Attachments:

• meld-3.18.3.tar.xz.xml.txt — Metalink XML returned by GNOME's mirrorbrain

[hartwork]

PS: Is there a way to make aria2 "follow" Metalink content in this very case?

[tatsuhiro-t]

aria2 by default follows metalink. Doesn't aria2c https://download.gnome.org/sources/meld/3.18/meld-3.18.3.tar.xz work for you?

[hartwork]

PS: Found the docs on --follow-metalink=true|false|mem with Default: true but it seems broken or I misunderstand it. It's also rejected when passed on the comment line:

# /usr/bin/aria2c -d /usr/portage/distfiles -o meld-3.18.3.tar.xz --allow-overwrite=true --max-tries=5 --max-file-not-found=2 --user-agent=Wget/1.19.5 --split=1 --max-connection-per-server=1 --uri-selector=inorder --follow-metalink=mem https://download.gnome.org/sources/meld/3.18/meld-3.18.3.tar.xz
/usr/bin/aria2c: unrecognized option '--follow-metalink=mem'
Usage: aria2c [OPTIONS] [URI | MAGNET | TORRENT_FILE | METALINK_FILE]...
See 'aria2c -h'.

I don't see why, please advise.

[hartwork]

aria2 by default follows metalink. Doesn't aria2c https://download.gnome.org/sources/meld/3.18/meld-3.18.3.tar.xz work for you?

It downloads a .meta4 file for me, see:

# aria2c https://download.gnome.org/sources/meld/3.18/meld-3.18.3.tar.xz

11/17 15:58:31 [NOTICE] Downloading 1 item(s)

11/17 15:58:32 [NOTICE] Download complete: /tmp/tmp.HShTfBCXCY/meld-3.18.3.tar.xz.meta4

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
10f3d8|OK  |        n/a|/tmp/tmp.HShTfBCXCY/meld-3.18.3.tar.xz.meta4

Status Legend:
(OK):download completed.

[hartwork]

I just found that my system aria2 was compiled with --disable-metalink. If I recompile --enable-metalink, things look a lot better, even --follow-metalink=mem is recognized and seems to work well, pretty cool.
Could it be that despite --disable-metalink configuration aria2 sent some Accept header to the server that still asks for Metalink? My guess would be that --disable-metalink needs to affect one place more in the code. What do you think?

[tatsuhiro-t]

You are right that Accept header for metalink is sent regardless of whether metalink feature is built-in or not.

[hartwork]

Cool, thanks for the quick fix!