Output to stdout

[seanfisk]

aria2 is awesome! However, one thing I am missing is output to stdout, mainly so things like this can work:

# download and untar the latest version of Boost
wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.gz -O - | tar -xz

I am comfortable with C++, so I have no problem making a pull request to add it. But I wanted to gauge how involved this feature might be. It seems simple, but it might be more complex than I imagine, since all visible output now has to go to stderr or directly to the tty. So I figured I'd ask about it first.

[tatsuhiro-t]

The big issue is that aria2 can download multiple files simultaneously and out-of-order in its original file position. In that condition, output to stdout is somewhat hard to use.

[seanfisk]

OK, so it sounds like it would be pretty difficult and possibly in conflict with typical use of the tool. A couple other things I noticed that would make it difficult to implement:

• Progress is printed to stdout by default. Obviously this wouldn't be possible if the file is being output
  to stdout, so that would have to change (presumably to stderr or /dev/tty). That might be a breaking change for others' scripts if they expected progress to be output to stdout.
• The -o option isn't considered a path, it's the basename for the downloaded file.

I tried a named pipe (FIFO) approach, but the issue is that aria2 has no way of writing to a file descriptor that is known before the command is executed. Yes, you can make sure that foo.1 doesn't happen and predict the file name, but you can't create a file that aria2 then writes to. That makes it "impossible" to do the simultaneous download/untar with the current aria2 and standard tools (it's probably possible, but I'm no UNIX whiz).

Maybe this isn't the right fit for this particular tool.

[AnselmD]

I am interested in it. It is possible to determine outside from aria, how much from the "head" of the file is downloaded? Is is possible to to determine it with the controle file of aria2?
http://aria2.sourceforge.net/manual/en/html/technical-notes.html#control-file-aria2-format

[seanfisk]

@AnselmD Possibly, but I think that if this feature was implemented, it should be implemented internally rather than externally.

[yarikoptic]

I would like to chime another 1c for the ability to specify output file name. ATM, I see no ability to download from multiple URLs into a differently named file. Usecase -- use by git-anex see http://git-annex.branchable.com/todo/extensible_addurl/#comment-40a1d58630f56dd744d56dc56a68770e

[tatsuhiro-t]

To specify different file name for each downloads, currently one has to use -i file option.
http://aria2.sourceforge.net/manual/en/html/aria2c.html#cmdoption-i
Inside it you can set options per download item: http://aria2.sourceforge.net/manual/en/html/aria2c.html#input-file

[yarikoptic]

On Fri, 05 Dec 2014, Tatsuhiro Tsujikawa wrote:

To specify different file name for each downloads, currently one has to
use -i file option.
http://aria2.sourceforge.net/manual/en/html/aria2c.html#cmdoption-i
Inside it you can set options per download item:
http://aria2.sourceforge.net/manual/en/html/aria2c.html#input-file

sorry if I am blind but I don't see an option for specifying the output
filename among those options ... could you please point me further or
just provide an example? Thanks in advance!

[tatsuhiro-t]

For example, suppose you created the following file:

http://localhost/foo/bar
    out=buzz
http://localhost/index.html
    out=file.html

and save it to list.txt.

Then aria2c -ilist.txt makes aria2 download 2 files and file from http://localhost/foo/bar is saved as buzz and file from http://localhost/index.html is saved as file.html.
In list.txt, you can write URI per line for each download. Options for each download can be specified after the URI line and preceded by one or more spaces. In the above example, out option is specified, which is the same as --out option in command-line. You can also write multiple URIs per line delimited by TAB to download a resource from multiple URIs.

[ghost]

That's the major showstopper for me as well. I think it would be rather easy to determine internally if we're currently downloading single file or multiple and enable "--output2stdout" (or whatever you might call it) only for single-file downloads. There're already options which work differently for single- and multi- file downloads (--download-result for example).
I really hope one day aria2c will be able to catch up with wget's features.

[icodeforlove]

I needed this as well because we were pulling 500GB, up to 1TB snapshot files from cloud storage to a machine which did not have double the physical space. Resumability is not required as if the pipe breaks the whole operation is broken, this does not mean that the underlying process cant keep retrying on parts (and it does this).

I threw together https://github.com/icodeforlove/npm_fatpipe.

It's only for piping output, so if you are looking for resumable features I'd stick to aria2. As for how the concurrency is handled It downloads multiple parts, and pipes to stdout in order. One of the big issues you will run into is handling back pressure, so having the ability to fine tune this is very useful.

Maybe something like this can be added into aria2c, if not you can use fatpipe for this edgecase.

[jalr]

I want to regularyly restore compressed database backups. A drop-in replacement for the curl $dumpurl | lzop -dc | mysql would solve my problems I currently have with aborted downloads.

[sebma]

Do you plan to implement this feature so that we can pipe the output to html parsers such as pup ?

[mikepurvis]

Just adding another voice here; I'd like to be able to specify arbitrary streams for my downloads, especially when talking to the daemonized aria2 client. As a quick trial of attemping to set it to write directly to stdout, we end up having issues when it tries to do things like seek/flush the file:

$ aria2c https://github.com/aria2/aria2/raw/master/AUTHORS --dir / -o dev/stdout --allow-overwrite

09/11 16:41:16 [NOTICE] Downloading 1 item(s)

09/11 16:41:16 [NOTICE] CUID#7 - Redirecting to https://raw.githubusercontent.com/aria2/aria2/master/AUTHORS

09/11 16:41:16 [ERROR] Exception caught
Exception: [DefaultBtProgressInfoFile.cc:211] errorCode=1 Failed to write into the segment file /dev/stdout.aria2

09/11 16:41:16 [ERROR] Error when trying to flush write cache
Exception: [AbstractDiskWriter.cc:335] errorCode=17 Failed to seek the file /dev/stdout, cause: Illegal seek

09/11 16:41:16 [ERROR] CUID#7 - Download aborted. URI=https://github.com/aria2/aria2/raw/master/AUTHORS
Exception: [AbstractCommand.cc:403] errorCode=17 URI=https://raw.githubusercontent.com/aria2/aria2/master/AUTHORS
  -> [DownloadCommand.cc:124] errorCode=17 Write disk cache flush failure index=0

09/11 16:41:16 [NOTICE] Download GID#ee8c0e9b5e8a9d7a not complete: /dev/stdout

So the first order of business here is probably a stream-friendly output mode which avoids doing these things.