Support backup to raw full files instead of zip

[kenkendk]

From spama...@ajserver.com on April 03, 2011 17:39:41

Support backing up to actual files instead of zip'ed files, i.e. test.doc is send to the server is test.doc. This may even have been in an earlier version as I would imagine it is a logical stepping stone to send data.

My problem is when my wife calls from the road or parents call because they accidentally deleted a file they have difficulty finding where to restore (not computer savvy at all). I however have access to the server could go find it and email it back to them, simple easy administrator level access. We currently are using DeltaCopy however I find Duplicati to be far superior in every way except that I cannot easy send people back their lost files as the server administrator.'

For issues of compression during transfer I use rsync args: --compress-level=9 and this does a great job.

Again I am only asking for an option to be able to write the actual files through to the server, obviously this would be limited to no encryption and perhaps only where it would be used in the case.

I would like to assist in the development of duplicati however I am not much of a programer, sysadmin and software tester is what I do professional. I can also offer build boxes/hardware to some extent. Please let me know how I can assist the project.

Thanks,
Aaron Johnson
aaron at same domain as my Google account

Original issue: http://code.google.com/p/duplicati/issues/detail?id=394

[kenkendk]

From kenneth@hexad.dk on April 04, 2011 04:39:49

After reading this a few times, I think you are asking something similar to "can Duplicati work like RSync and store files?".

The answer to that question is no, see the FAQ for details: https://code.google.com/p/duplicati/wiki/FAQ It basically boils down to requiring server side software, which receives data and handles various operations. Since Duplicati is designed to work with "dumb" backends, such as S3 or FTP, it is not possible to require a server side program.

You could in theory store the zip files as folders, but it would require many transfers (one for each file) during each backup. Besides, the data inside is not
a trivial copy of the file, but only a diff, making it somewhat difficult to extract the desired version of the file, even if it is not compressed.

If you do have access to the server, you could install Duplicati on the server, and then run it there, which will give you fast access to the data files, and a UI for selecting the file to restore. Alternatively, you can just run it on your local machine and connect to the data storage and restore it like that.

Does that answer/solve your request?

As for assisting the the project, I currently have what I need in terms of hardware for testing/building. I could use a range of servers, eg. ftp/ssh/webdav for running automated tests against, but it is not a priority for now.

If you do software testing, I would like to know of ways to perform automated GUI testing. I currently have automated testing for the core program with various options, and it seems to catch most mistakes that I make. But I have not yet found a way to make automated GUI test, so if you have some suggestions for that, they would be much appreciated.

Owner: kenneth@hexad.dk
Labels: -Type-Defect Type-Enhancement

[kenkendk]

From ioncot...@gmail.com on July 04, 2011 09:29:58

from rsync manual page I understand rsync works excellent with local source AND destination.
combining this with mountability of any of your backends as windows local drives or linux mountpoints, I see no reason to not use rsync.

[kenkendk]

From kenneth@hexad.dk on July 04, 2011 10:39:26

Yes, you can mount the folder, but that does not mean that you can avoid the file transfer. Even if your target is a mounted local drive you will still have to write the entire new file on the remote server, it is just abstracted by the mounting process. It would also require that each backend can do a rename operation (which may be possible).

In the end, you would only get the benefit that you store the recent copy, not the transfer benefit, which I consider the most important factor with rsync. If you do mount the folder, is there anything that Duplicati could add compared to just using rsync (or one of the many GUIs)?

[kenkendk]

From rst...@gmail.com on September 13, 2013 14:02:15

We just changed the storage engine in a way that makes it even more difficult to open backups on the server and send the files to someone else. It is still possible to access a backup from a clean computer and restore files.

The new engine now can restore files to their original destination and me will spend some time to make the UI easier to use and open it for remote access. So, you could login to your parent's computer and restore files for them. This is all planned for the new UI.

More information about the new storage:
Test build: http://www.duplicati.com/news/theduplicati20storageengineishere How it works: https://duplicati.googlecode.com/files/Block-basedstorageformat.pdf To summarize: We will never implement your wish to write files to a server but Duplicati might still have a solution for your problem. :-)

Status: Done

[rkarlsba]

Duplicati should support non-compressed storage. Most of my data is compressed already, so having it go through gzip for just a .1% of compression is just waste.

How can I do this?

[mwtrigg]

Another potential reason: using Amazon Cloud Drive. If I'm backing up picture files, they're all free (do not count against storage limits) when in their native format; however in their compressed format, there is a cost.

[phendryx]

👍 I literally just bought the 100GB for $11.99/yr plan, knowing 95% of my backups are photos. My other backups would fit well under 100GB, but pictures, not so much. The wife loves her 90 slightly different pictures of an unfocused moon :-|.

edit: I have Prime (who doesn't?), this photo storage is "free".

[feutl]

I just do not understand why people would want files being stored 1:1 on the backup location, because there are so many tools out there doing exactly this, and duplicati is great because of being secure and using remote locations securely.

BUT what I really would love to see, and I back @rkarlsba is a way to run the backup without compression. I backup jpg, mpg and other already compressed stuff, my NAS is not happy about the additional compression layer without a usable result.

Found this
#2671 as well from @rkarlsba - perhaps it is possible to just put the disable compression to a simple switch option.

[kenkendk]

First, Duplicati has a list of non-compressible file extensions, and it includes .jpg:
https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Main/default_compressed_extensions.txt

Any file on the list will be stored without attempting compression.
You can use your own file (same format as the default one) and supply it with --compression-extension-file.

If you really don't want compression, you can add --zip-compression-method=store or alternative --zip-compression-level=0.

[suoko]

In case of OD4B, this sw is a nice solution where official client is missing
If it synced file by file instead of zipping, it would be much appreciated

[the-lazy-fox]

Totally agree with you guys, a raw saving model would be great as hell!

[rkarlsba]

Well, this ticket is only just short of six years old, so give it another six years, someone might bother to implement it?

[drwtsn32x]

@rkarlsba - it will never happen.

The point of using Duplicati is to get you versioning, deduplication, flexible retention, etc.

If you want a synchronization tool use something like rclone - it is pretty amazing

[rkarlsba]

I know other tools, that's not the point here. The point is that the forced zip compression of things like video files is just stupid, since they can't be compressed anyway.

[drwtsn32x]

@rkarlsba - the main reason the files are ZIP on the back end is because they've been chunked, deduplicated, and chunks repackaged into volumes. Even if the ZIP compression were disabled, you'd still have chunks/volumes. Duplicati is not a sync tool and will never back up files in native format on the back end.

[rkarlsba]

Normal files can also be chunked quite easily, it's just called read() and write() or even something like dd

[drwtsn32x]

The chunking is to facilitate deduplication. You can't have deduplication and still have files in native format.

[kees-z]

The point is that the forced zip compression of things like video files is just stupid, since they can't be compressed anyway.

There is no forced compression. File extensions specified in default_compressed_extensions.txt will be added to the Zip archives without any using any compression.

[rkarlsba]

still, the point is, again, that chunking is very easy to do in the code, so there's no reason, really, to zip it

[kees-z]

In Duplicati, the default size of a chunk (block) is 100 KB. These blocks are packaged in a zip archive (only compressable file types are actually compressed) to reduce the number of files at the remote storage location. Without using archives containing quite a number of blocks, we would end up with tons of files in a single folder at the backend.

[drwtsn32x]

@rkarlsba not sure how else to explain this.... but Duplicati will never be a synchronization type program that keeps files in native format. It is inherently incompatible with its core feature, deduplication.

[james-powis]

I don't even remember why I star'ed this... but that was a pretty solid F-You from @drwtsn32x ... I can say that Zipping the hell out of something is very different than deduplication... I know uncompressed output is a very real use case for a lot of my use cases at all of my last employers going back a decade...

[drwtsn32x]

@james-powis I certainly didn't mean to come across that way and apologies to @rkarlsba if that is how it sounded.

Was just trying to emphasize that Duplicati is not a sync tool and will never put files in native format on the back end.

[drwtsn32x]

If Duplicati didn't zip at all then all the raw chunks would be on the back end. @kees-z did a nice job explaining.