Caching research

[binoculars]

After some tweaks and enabling caching, I was able to get the build down to 10 minutes for an added library (libsnappy) and 1 minute for non-Dockerfile changes. Updating x265 took about 17 minutes. I think this is the way to go for speed.

What I did to my repo:

• Move most ARGs to the top
• Inherit ARGs only where needed
• Enable caching for GitHub actions docker builds
• Parallelize the builds by breaking them up into multi-stage
• Use COPY command to copy only files needed from previous stages
• Added several new libraries

[wader]

After some tweaks and enabling caching, I was able to get the build down to 10 minutes for an added library (libsnappy) and 1 minute for non-Dockerfile changes. Updating x265 took about 17 minutes. I think this is the way to go for speed.

Interesting, thanks for trying things out!

What I did to my repo:

• Move most ARGs to the top

It actually used to be like that but at some point i decided it was nice to not spread things out if possible.

• Inherit ARGs only where needed
• Enable caching for GitHub actions docker builds

This i should really do. Do you know what limits there are on cache sizes? some expire time etc? thinking if there are some limits that would make caching not as useful in practice.

• Parallelize the builds by breaking them up into multi-stage

Is nice but as i mentioned earlier not sure yet what i think about the increased complexity of the Dockerfile.

• Use COPY command to copy only files needed from previous stages
• Added several new libraries

Haven't used ladspa and frei0r myself, they seem to use .so files? do you know if building with musl libc could make this a problem? are glibc built plugins compatible?

Noticed libcaca was removed? i remember there was some x11 issue etc?

libvo-amrwbenc and snappy would be nice to add

About installing alpine package or build from source: I've chosen to built from source if the library has updates for features or security, especially decoders, that would take time to end up in alpine. Otherwise also if there are some other issue lik drags in deps we don't want or not built with optimisation/hardening. But i don't follow any strict rules.

[wader]

Wops accidentlly submitted above comment before i was done writing... editing now :) ok done editing

[binoculars]

The cache limits for gha are currently 10 GB and 7 days which seems pretty reasonable.

Not sure about the .so files—I haven’t tested these features yet, although I had frei0r in an old build.

Yes libcaca failed on the build but not configure, so I tried pushing that while testing it failed so I had to remove it.

[wader]

Adding caching #253. Not as useful without more stages but much better than nothing

[wader]

BTW know if there are any security concern with using a shared cache between PR and other builds? if content addressable i would guess not?

[binoculars]

Not that I'm aware of. Works like a charm if you only build master branch, but I'm just too paranoid to let it build without a manual review first.

[wader]

Ok, will probably sync some other things from your fork when i get time, thanks again for working on it! let's keep in touch!