zbackup gc keeps adding 2x file size?

[cgm999]

[root@hw1 zbackup]# ls -lh index
total 216M
-rw------- 1 root root 132K Apr 11 02:58 50f87e1dad04b9dff1b153b08d79b6b548cce18c24019d90
-rw------- 1 root root 216M Apr 10 08:35 878f27901bf28cad4ecae1e1d58190fe4118548b972d7607

FIRST GC adds 216M index

[root@hw1 zbackup]# zbackup gc --non-encrypted $PWD
Loading index...
Loading index file 878f27901bf28cad4ecae1e1d58190fe4118548b972d7607...
Loading index file 50f87e1dad04b9dff1b153b08d79b6b548cce18c24019d90...
Index loaded.
Using up to 40 MB of RAM as cache
Using up to 16 thread(s) for compression
Checking used chunks...
Checking backup 003...
Checking backup 001...
Checking backup 002...
Checking backup 004...
Checking backup 005...
Checking bundles...
Loading index...
Loading index file 878f27901bf28cad4ecae1e1d58190fe4118548b972d7607...
Loading index file 50f87e1dad04b9dff1b153b08d79b6b548cce18c24019d90...
Index loaded.
Cleaning up...
Garbage collection complete
[root@hw1 zbackup]# ls -lh index
total 432M
-rw------- 1 root root 132K Apr 11 02:58 50f87e1dad04b9dff1b153b08d79b6b548cce18c24019d90
-rw------- 1 root root 216M Apr 10 08:35 878f27901bf28cad4ecae1e1d58190fe4118548b972d7607
-rw------- 1 root root 216M Apr 11 05:18 fa996559bbf6d44bc671908c42a3a2e1c21096e4d1c115c1

SECOND GC adds 432M index

[root@hw1 zbackup]# zbackup gc --non-encrypted $PWD
Loading index...
Loading index file fa996559bbf6d44bc671908c42a3a2e1c21096e4d1c115c1...
Loading index file 878f27901bf28cad4ecae1e1d58190fe4118548b972d7607...
Loading index file 50f87e1dad04b9dff1b153b08d79b6b548cce18c24019d90...
Index loaded.
Using up to 40 MB of RAM as cache
Using up to 16 thread(s) for compression
Checking used chunks...
Checking backup 003...
Checking backup 001...
Checking backup 002...
Checking backup 004...
Checking backup 005...
Checking bundles...
Loading index...
Loading index file fa996559bbf6d44bc671908c42a3a2e1c21096e4d1c115c1...
Loading index file 878f27901bf28cad4ecae1e1d58190fe4118548b972d7607...
Loading index file 50f87e1dad04b9dff1b153b08d79b6b548cce18c24019d90...
Index loaded.
Cleaning up...
Garbage collection complete
[root@hw1 zbackup]# ls -lh index
total 863M
-rw------- 1 root root 432M Apr 11 05:19 41f8614034da0cfaf76766ef28f590e1dc9521b3383d656b
-rw------- 1 root root 132K Apr 11 02:58 50f87e1dad04b9dff1b153b08d79b6b548cce18c24019d90
-rw------- 1 root root 216M Apr 10 08:35 878f27901bf28cad4ecae1e1d58190fe4118548b972d7607
-rw------- 1 root root 216M Apr 11 05:18 fa996559bbf6d44bc671908c42a3a2e1c21096e4d1c115c1

Every new gc run will add a new file 2x size of prev one.

[zogith]

I am also seeing this issue. Every GC doubles the index file size, so having a periodic gc to clear out old backups causes excessive growth in repository size, making zbackup unfeasible to use for backups. This is with zbackup 1.4.1 from the Ubuntu .deb files.
Is there a fix for this? Or is there a better practice than periodic run of "gc" to clean out expired data from the repository?

[zogith]

Using the current version checked out from GIT rather than the Ubuntu release 1.4.1 seems to fix it - though I had to also apply the patch at #72 - more info as my ongoing backups accumulate.
EDIT - on further testing the current github version doesn't fix it: If I repeatably back up /var/cache on my Ubuntu system, delete the oldest backup, keeping 10 backups alive, I get the following index sizes:

index size
364K /tmp/zbackup-test/index
724K /tmp/zbackup-test/index
1.5M /tmp/zbackup-test/index
2.9M /tmp/zbackup-test/index
5.7M /tmp/zbackup-test/index
1.2M /tmp/zbackup-test/index
2.2M /tmp/zbackup-test/index
4.4M /tmp/zbackup-test/index
8.8M /tmp/zbackup-test/index
18M /tmp/zbackup-test/index
36M /tmp/zbackup-test/index
70M /tmp/zbackup-test/index
140M /tmp/zbackup-test/index
280M /tmp/zbackup-test/index
560M /tmp/zbackup-test/index
1.1G /tmp/zbackup-test/index
2.2G /tmp/zbackup-test/index
4.4G /tmp/zbackup-test/index

[zogith]

More repeatable example, with script:
using contrived example:

DIRBACKEDUP=/var/cache/
TESTZBACK=/store1/tmp/zbackup-test
zbackup init --non-encrypted $TESTZBACK  

tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/001
tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/002
tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/003
tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/004
tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/005
tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/006
tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/007
tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/008
tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/009
tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $TESTZBACK/backups/010

while [ 1 -eq 1 ]
do
   OLDESTBACK=`ls -rt $TESTZBACK/backups/* | head -1`
   echo removing $OLDESTBACK
   rm $OLDESTBACK
   echo garbage collecting.
   zbackup gc --non-encrypted --cache-size 1024mb $TESTZBACK

   NEWESTBACK=`ls -t $TESTZBACK/backups/ | head -1`
   CURRENTBACK=$TESTZBACK/backups/00`expr $NEWESTBACK + 1`
   echo currentback is $CURRENTBACK
   tar cf - $DIRBACKEDUP | zbackup backup --non-encrypted $CURRENTBACK

   echo index size
   du -hc $TESTZBACK/index
   du -hc $TESTZBACK/index >> /tmp/zbackup-test-index-size.$$.txt
done

I get result size file:

212K    /store1/tmp/zbackup-test/index
212K    total
420K    /store1/tmp/zbackup-test/index
420K    total
832K    /store1/tmp/zbackup-test/index
832K    total
1.7M    /store1/tmp/zbackup-test/index
1.7M    total
3.3M    /store1/tmp/zbackup-test/index
3.3M    total
6.5M    /store1/tmp/zbackup-test/index
6.5M    total
13M /store1/tmp/zbackup-test/index
13M total
26M /store1/tmp/zbackup-test/index
26M total
52M /store1/tmp/zbackup-test/index
52M total
103M    /store1/tmp/zbackup-test/index
103M    total
206M    /store1/tmp/zbackup-test/index
206M    total
412M    /store1/tmp/zbackup-test/index
412M    total
823M    /store1/tmp/zbackup-test/index
823M    total
1.7G    /store1/tmp/zbackup-test/index
1.7G    total
3.3G    /store1/tmp/zbackup-test/index
3.3G    total
6.5G    /store1/tmp/zbackup-test/index
6.5G    total

[zogith]

I think I have a crude fix - I say crude as I did this without any deep understanding of the code, but setting indexModified to true when doing gc so that the index is rewritten rather than added to when doing gc() seems to fix it: e.g.

edit: only works in some cases, still debugging.
edit: diff deleted, it was junk, sorry.

[DuncanRobertson]

This change seems to sort out this problem, at least for my usage case of zbackup:
DuncanRobertson@83a0fa8
It forces the rewrite of the index when a gc() being done.

[Vlad1mir-D]

This bug was fixed in d6cb627 (and a0ed4a4 - backport for 1.4).
I'll release 1.4.3 with bugfix as soon as I'll close #82.