Ability to cache !Sh build commands

[cablehead]

Just checking - is the best way to discuss new ideas? Would a mailing list be better?

So this is very much a feature request...

I find myself wanting to be able to cache !Sh sections.

I.e., something like you have a !Sh section that takes a while to build, but is stable. You then want to build on top of that with additional !Sh sections. If you add multiple !Sh sections, could a cache be taken after each one, and only rebuilt if that !Sh section changes?

[tailhook]

Well, I'm not sure I understand you. But let me explain how cache works in vagga. Unlike in docker which caches each step, we have two things:

1. Cache of packages for each package manager (and other downloads)
2. Fully built containers with version hash

The (1) is probably not relevant for discussion.

The hash for container version is constructed from the !Sh section itself. I.e. if you don't change text of shell command, vagga will think that container doesn't change, and will not rebuild.

In case you use some external scripts or files in !Sh script and want container to depend on it, use !Depends "filename" command, which does nothing except uses file contents in the hash of the container version.

[cablehead]

As a concrete example, alpine doesn't have a package to install openresty. I have this recipe to install it:

https://github.com/cablehead/vagga-alpine-openresty/blob/master/vagga.yaml

It takes a couple of minutes to compile. I'd like to be able to freeze this down, and then build additional things on top. If that makes sense? So I guess yeah, getting close to docker's idea of caching after each step.

[tailhook]

The problem is being like a docker is that we can do that fast only on btrfs. Is it ok for you? Well, snapshots are fast, but still for each snapshot we need to keep temporary files around.

There are couple of alternatives I'm thinking of:

1. We can make repeated compilation very fast by using ccache.
2. We can do !BuildPackage directory command which either builds a package or uses already built one. Alpine packages is very easy to create. This has an advantage that you will have a nicely organized packages, instead of directories with bunch of stuff (i.e. temporary directories with build files kept in snapshots)
3. We can have !Container xxx command which you can use as starting point of another container instead of !Alpine v3.1

What do you think?

[cablehead]

I really like the idea of number 3.. but, I'll definitely defer to you on this one. Particularly I don't have experience building alpine packages, but would like to learn! Haha.

I'm in the middle of reworking that repo I linked above to have recipes for other things I'm working with:

https://github.com/cablehead/vagga-alpine

Thinking out loud, if someone wanted to reuse these recipes, what would be the easiest thing? Would say turning these recipes into something that creates an alpine package for reuse be better, or to use them as is to create a container, and then reference that container with !Container xxx

[cablehead]

How would these recipes look reworked to be usable with a BuildPackage directive?

[tailhook]

I would say for BuildPackage they need just to be alpine packages, like this: http://wiki.alpinelinux.org/wiki/APKBUILD_examples:Simple

The advantage is: they are usable not only in vagga but generally for alpine users.

(In fact I want to implement all three things :) , but not snapshotting so far)

[cablehead]

The combination of those 3 features would be really powerful. Yeah, snapshotting could lead to things potentially turning into a mess and would be unnecessary with these features (or even 1 or 2 of them)

[tailhook]

Another thought about reusing recipes, is that when they are done as containers they are not composable (you can't apply two).

Still "!Container" command is probably the easiest thing to do, so I'm going to start with it :)

[cablehead]

That's true, while 1 and 2 would be composable. Is there anything I can do that would be of help? I still owe you some patches for the installation docs too :)

[tailhook]

You can try "subcontainers" branch. It doesn't build dependent container automatically, you need to "vagga _build name" first (or run any command in it). It lacks docs and versioning is quite glitchy so far.

Also it's has refactoring of internals, so I need to test it much more.

[cablehead]

Awesome: https://gist.github.com/cablehead/9675e5dc85b33ba2d441

[tailhook]

Ok, the !Container now in master. I've put a package into separate vagga-testing repository until I cut a next release (in case you use debian/ubuntu repo).

[tailhook]

Well the !Container is in stable long time ago. And we have #82, #83 for other two options. So I'm closing this now.