extremely inefficient way of handling sparse files

[mbiebl]

Todays backup of a 30G Linux partition with ~15G of data took several hours (actually, I aborted it after several hours). In the past, this process has been rather quick (matter of minutes).

Running fsarchiver -v shows the following:

            xattr:file=[/var/log/lastlog], attrid=0, name=[security.selinux], size=37
            xattr:lgetxattr(/var/log/lastlog,security.selinux)=37
            xattr:lgetxattr(/var/log/lastlog,security.selinux)=37: [security.selinux]
-[00][ 83%][REGFILE ] /var/log/lastlog
oper_save.c#171,createar_obj_regfile_unique(): backup_obj_regfile_unique(file=/var/log/lastlog, size=1168000000292)

/var/log/lastlog is so-called a sparse file, its actual size is

$ du -hs /tmp/fsa/20190123-191847-0001b2f9-00/var/log/lastlog 
292K	/tmp/fsa/20190123-191847-0001b2f9-00/var/log/lastlog

$ stat /tmp/fsa/20190123-191847-0001b2f9-00/var/log/lastlog 
  File: /tmp/fsa/20190123-191847-0001b2f9-00/var/log/lastlog
  Size: 1168000000292	Blocks: 584        IO Block: 4096   regular file
Device: 806h/2054d	Inode: 145655      Links: 1
Access: (0664/-rw-rw-r--)  Uid: (    0/    root)   Gid: (   43/    utmp)
Access: 2019-01-22 19:40:36.006267137 +0100
Modify: 2019-01-22 19:40:36.006267137 +0100
Change: 2019-01-22 19:40:36.006267137 +0100
 Birth: -

Looks like fsarchiver tries to process 1TB of data.

https://linux.die.net/man/8/lastlog

[mbiebl]

https://bugzilla.redhat.com/show_bug.cgi?id=951564

[mbiebl]

After excluding the following files, backup times are back to normal:

$ ls -la tallylog lastlog faillog 
-rw-r--r--. 1 root root  128000000032 Dez  7 20:02 faillog
-rw-rw-r--. 1 root utmp 1168000000292 Jan 23 19:58 lastlog
-rw-------  1 root root  256000000064 Dez  7 20:02 tallylog

$ du -hs tallylog lastlog faillog 
68K	tallylog
292K	lastlog
36K	faillog

[fdupoux]

You are right, there is currently no support for sparse files. fsarchiver just performs standard read operations on the fly and will consider all logical bytes as actual data. Unfortunately I don't think there is any easy solution to add support for sparse files. It would probably require a change in the file format and a lot of code changes.

[SolarDuck]

This seems to contradict:

https://forums.fsarchiver.org/viewtopic.php?f=17&t=1009&p=2971&hilit=sparse#p2971

Which states:

When fsarchiver reads a sparse file from the disk, it's just a normal file where empty parts are just zero bytes. As a consequence the compression will be extremely good for all these parts (something like 99% I guess). Fsarchiver also checks the space used by the file on the disk to know whether or not it's a sparse file, and then it sets a flag in the archive. When you restfs, it will recreate the file as a sparse file if the flag is set, else it will just be a normal file with a lot of zero bytes inside.

At the end, the archive will be very small anyway since big blocks with zero bytes have an excellent ratio.

The above is dated: Sat Feb 21, 2004 12:12 pm
And
http://www.fsarchiver.org/changelog/

for 0.6.7 (2010-01-31):

Added support for sparse files (sparse file on the original disk will be recreated as sparse file)

What gives?

Thank you,