Please clean up old cached images

[martinpitt]

I haven't used lxd that many times yet, and already I have quite a sizable image cache:

$ sudo du -hs /var/lib/lxd/images/
2,7G    /var/lib/lxd/images/
$ lxc image list
+-------+--------------+--------+-----------------------+--------+---------+-------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |      DESCRIPTION      |  ARCH  |  SIZE   |          UPLOAD DATE          |
+-------+--------------+--------+-----------------------+--------+---------+-------------------------------+
|       | 147b4fdcc090 | no     | Ubuntu xenial (amd64) | x86_64 | 67.04MB | Nov 24, 2015 at 10:52am (CET) |
|       | 1b20851236d5 | no     | Ubuntu xenial (amd64) | x86_64 | 66.87MB | Dec 1, 2015 at 6:47am (CET)   |
|       | 23f3395b9409 | no     | Ubuntu trusty (amd64) | x86_64 | 64.57MB | Dec 1, 2015 at 7:27am (CET)   |
|       | 25fcf43cb929 | no     | Ubuntu trusty (amd64) | x86_64 | 64.56MB | Nov 26, 2015 at 3:49pm (CET)  |
|       | 3e9577f7a954 | no     | Ubuntu wily (amd64)   | x86_64 | 73.36MB | Dec 1, 2015 at 6:40am (CET)   |
|       | b35f2aa010db | no     | Ubuntu xenial (amd64) | x86_64 | 67.06MB | Nov 25, 2015 at 8:29am (CET)  |
+-------+--------------+--------+-----------------------+--------+---------+-------------------------------+

It's of course very desirable to cache the latest version of a particular image. But it would be nice to automatically clean up older versions of the same image type,so that they don't pile up. It seems every image inflates to roughly 400 MB in /var/lib/lxd/images/.

FTR, these were created with something like lxc launch images:ubuntu/trusty/amd64 t1 where images: is https://images.linuxcontainers.org:8443.

Thanks!

[martinpitt]

Alternatively, is there some "cache prune" command that one could run from cron? I'm planning to use this for production autopkgtest, and they will create and remove containers all the time (and thus quickly pile up huge caches).

[stgraber]

Hmm, images should be automatically flushed 10 days after they were last used (configurable), seems like there's something off there.

@hallyn

[stgraber]

@martinpitt for autopkgtest, you probably want to use "lxd-images import ubuntu SERIES ARCH --alias SOME-NAME --sync" which will get you the official cloud images rather than the unofficial images our Jenkins builds. The --sync combined with the cronjob we ship with LXD in Ubuntu will also mean that the image will be updated for you every hour by cron.

[stgraber]

(and yes, we know it's a bit confusing right now with the two ways to get Ubuntu images, we will merge simplestream support directly in LXD in the next couple of months which will make the lxd-images script go away and hopefully simplify things a bit)

[martinpitt]

TBH, your images are much better than the cloud images -- they have tons of stuff on them which I don't want and which is even detrimental for testing. I have a big list of apt-get purge in my image preparation scripts for creating my own base images from the cloud images. It seems at some point I'll need to do that anyway for lxd then (once you drop your images).

Thanks for the hint with importing! However, will that actually give me any benefit over the lazy downloading with specifying a remote image source? The nice thing there is that I get exactly the images that are needed, and I don't need any extra maintenance/cron jobs to sync images explicitly.

[martinpitt]

Hmm, images should be automatically flushed 10 days after they were last used (configurable),

Ah, it hasn't been 10 days yet, just 7 :-) I'll watch this and report back in 4 days when the oldest image gets 11 days old. Thanks!

[stgraber]

You can also change the value:

lxc config set images.remote_cache_expiry 2

I've never really seen this work here but that's mostly likely due to me never having images that are unused and old enough to be pruned. Looking at the code, just reducing the expiry should trigger an immediate expiry run.

[stgraber]

And no, if you're happy with the super minimal LXC images, you can keep using them.
As images.linuxcontainers.org appears as a LXD server, expiry is automatically setup for you and you don't need a cron job or anything.

I just thought you'd prefer the cloud images as I believe that's what's currently used for the VM based autopkgtest setup and would make sense as a way to provide a consistent environment across arches.

[stgraber]

(but for Xenial, there sure is the benefit that our own images are actually built daily, which is not yet the case with the cloud images :))

[martinpitt]

I just thought you'd prefer the cloud images as I believe that's what's currently used for the VM based autopkgtest setup

As a basis, yes, as that's all I get to start from. But I adjust them quite a bit (http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/tree/setup-commands/cloud-vm-setup) to downsize them again.

[srkunze]

TBH, your images are much better than the cloud images -- they have tons of stuff on them which I don't want

My 2 cents: I totally agree.

[martinpitt]

I close this for now. I'm quite sure I was just missing the 10 day cleanup period. If I really still have the old images next week I'll reopen. Thanks for the hint about lowering the cache time, that's going to be useful!

[martinpitt]

Hmm, I checked today again and I now have images which are more than 10 days old:

|              | b35f2aa010db | no     | Ubuntu xenial (amd64) | x86_64 | 67.06MB  | Nov 25, 2015 at 8:29am (CET)  |
|              | 147b4fdcc090 | no     | Ubuntu xenial (amd64) | x86_64 | 67.04MB  | Nov 24, 2015 at 10:52am (CET) |
| xenial/amd64 | 322b76d2e1d4 | no     |                       | x86_64 | 122.00MB | Dec 4, 2015 at 9:45am (CET)   |

So reopening.

[stgraber]

@hallyn any idea what may be going wrong?

[hallyn]

@martinpitt

could you shut down lxd, then do

sudo sqlite3 /var/lib/lxd/lxd.db

SELECT value FROM config WHERE key='images.remote_cache_expiry';

I think what's going wrong is we're not setting a default of 10 like we meant to do.

[hallyn]

Oh, I see, we're supposed to cache there being no value and return 10 in that case.

[hallyn]

There are a few little errors there. I will push a fix later today plus a real test case.

[hallyn]

Note that the specs/configuration.md file says of the expiry time:

Number of days after which an unused cached remote image will be flushed

There are other bugs in the code anyway (expiry date was calculated wrong) which i fixed locally, but this description would not apply to the lxd-images imported images because they are not marked as cached, and presumably are not unused.

Do we want to change this to say that any image created before current date - expiry-days is deleted?

[stgraber]

No, lxd-images indeed aren't cached images and that's fine.

lxd-images itself has a --sync flag which does the auto-update and garbage collection.

With lxd-images going away soon, the LXD caching mechanism will then be used for the cloud images too.

@martinpitt is using the LXC images I believe.

[hallyn]

ok thx, then i'll post what i have to fix.

[martinpitt]

It seems you already have this figured out, but for the record:
``
sqlite> SELECT value FROM config WHERE key='images.remote_cache_expiry';

There is no hit at all. `SELECT * FROM config` only has keys `core.https_address` and `core.trust_password`. I think that confirms what @hallyn said.

> @martinpitt is using the LXC images I believe.

Yes, e. g. `ubuntu/xenial/amd64` from https://images.linuxcontainers.org:8443. They are really nice!