testing compression

[flibustenet]

I'm trying to test compression but i don't see any compression.

$ kopia repository create filesystem --path /ocean/kz
$ kopia policy set --compression=gzip-best-compression /ocean/kz
$ kopia policy show /ocean/kz
Policy for wilk@thinkpad:/ocean/kz:
...
Compression:
  Compressor: "gzip-best-compression" (defined for this target)
  Compress files regardless of extensions.
  Compress files of all sizes.

$ kopia snapshot ~/projets/flibuste

The repository has exactly the same size without setting compression (580M for 617M of source, 259M with targz).
Did I missed some configuration ?

I also tried setting compression with --global

edit: I build kopia with today clone

[jkowalski]

you need to set policy on ~/projects/... and not on the repository.

What kinds of files are you compressing? Many files these days are not really compressible at all.
Try a directory full of source code, such as Kopia itself.

To verify the compression do kopia snapshot ls to find the directory entry (starts with k<hash>) then do kopia ls -l k<hash> - if you see any entries with Z in them - those are compressed.

[flibustenet]

It's ok with setting policy on the source. I've tried with kopia source and it's ok also.

But i found that with a sql dump as text, very compressible (153M possibly gziped to 10M) there is an issue.

$ ls -lh /tmp/data
total 153M
-rw-r--r-- 1 wilk wilk 153M janv.  8 16:30 dump.sql

Without compression

$  kopia snapshot /tmp/data
uploaded snapshot 034c... (root k8b...) in 4.407741852s
$ du -shc /ocean/kk
153M	/ocean/kk

With compression

uploaded snapshot 70... (root k9b6...) in 1m9.976920549s
$ du -shc /ocean/kk
311M	/ocean/kk

I cleared cache and create new repository each time

edit: also mem usage is going very high (800M)

[jkowalski]

Weird, I'll investigate this - I downloaded a 163MB SQL dump from https://launchpad.net/test-db/+download and I can also see this.

[flibustenet]

I can confirm that it fix the issue \o/. You've done it very rapidly.
My repository (617M) is going to 262M which is less than with borg 305M. But slower (kopia 34s, borg 16s). On second snapshot kopia is a lot faster than borg...

I will test it now with my home on a daily basis to compare with real data.

Thanks !

[jkowalski]

btw you can see which compression algo is the fastest with:

kopia benchmark compression --data-file ~/employees_db/load_salaries.dump --repeat 1

[flibustenet]

Changing the algo, it's a lot faster with s2. Still a little bit slower than borg (lz4) but, for me, i don't think it need to be investigated more than that.