Archive stdin or fifo

[jborg]

(From http://librelist.com/browser//attic/2014/1/2/feature-request-archive-stdin-or-fifo/)

Hi,

first of all thanks to Jonas for developing attic. I am using attic
since 4 weeks on Debian and that's a great tool for doing backup with
deduplication. I am excited about the dedup ratio (original size vs.
unique data) achieved in my single repository although throwing in
different backup sources (tar, mysqldump) from different machines.

I would like to send a feature request here because i dont't have a
github account:
Like (all) the other Unix/Linux tools attic should be able to open
stdin or a fifo for backup and dedup the stream until eof. And of
course it should be possible to extract these archives to stdout.

Greetings,
Matthias.

[jborg]

This could be used to backup any type of streamed data.
Attic deduplicates data across the entire repository regardless of file and positoin.

[tomee]

I would really like that feature, too. I intend to use attic to archive virtual machine images.
The perfect implementation for me would be some kind of "create/write into a virtual file" to avoid the necessity to have as many repositories as VM images to be backed up. E.g. cat /dev/vg/mysnap | attic create --write-into-file=mysnap repo::today -
The "mysnap" file would contain the raw contents of the stream. And there could be several such files in one repository (versioned/deduplicated).

[ThomasWaldmann]

see PR #208.

[ThomasWaldmann]

Can someone explain the use case for a fifo?

[danmbox]

I'd like to add a use case -- backing up a Postgresql database, by piping pg_dumpall / pg_dump -Fc

[Ernest0x]

Can someone explain the use case for a fifo?

You can have a generic attic script that calls tasks before running the backup tasks (pre-run) and tasks after running the backup (post-run). That way you do not have to change your script to arrange the pipes each time you want a different pipe to attic. You will just create a pre-run script that possibly creates the fifo and then attaches to it as input, and perhaps a post-run script that does any clean up you may want. That way you can just add to your generic attic script configuration the pre-run and post-run scripts and also add the fifo path in the includes (maybe with a prefix to let attic know that it should open it as a fifo and read it, e.g. "fifo:/path/to/fifo").

[danmbox]

Regarding the stream:

• there should be an option to name the stream (e.g. stdin=postgre.sql)
• if the format is tar, Attic could detect individual filenames from tar data (like zbackup does)

i.e. we could have --stdin-name and --stdin-format

[ThomasWaldmann]

@danmbox why do you need/want a fifo for pg_dumpall that and not just a simple pipe (via attic stdin)?

@Ernest0x so do I understand it right, the fifo file is just like a meeting point for your different (script) commands so you do not have to arrange normal pipes between them?

[danmbox]

@ThomasWaldmann I want a pipe alright, and I also want to name the incoming file (so that it isn't simply stored as "stdin"), just as backing up directories preserves filenames. See my comments at PR #208

[Ernest0x]

@ThomasWaldmann Yes. It would help at abstracting piping logic out of scripting attic, which architecturally is not a bad idea imho. Also, fifos created and used by other tools may already exist in your systems and you may also want to attach to and read them to a backup archive.