How to speed up image generation?

[AlenaSviridenko]

Hi folks,

We are constantly trying to improve image generation speed, because we provide a lot of pre-installed software, and along with adding a new one, we should try not to significantly increase image generation time. We already implemented some ideas with parallel tools installation and changes in image specs, and are actively looking into other ideas.
Effective image generation time is important for testing changes and for those customers who build image based on our scripts.

This is an open discussion, any ideas are warmly welcomed 🍕

[catthehacker]

One thing is to make module installation parallel.
current:
https://github.com/actions/virtual-environments/blob/4de7f89a42529d48d04cc7a7eb85b570b709448c/images/linux/scripts/installers/azpowershell.sh#L15-L18
new:

parallel --jobs 0 --halt soon,fail=1 'pwsh -Command "Save-Module -Name Az -LiteralPath /usr/share/az_{} -RequiredVersion {} -Force -Verbose"' ::: "${versions[@]}"

I'm pretty sure there could be more actions that can be parallelised in this repo, unfortunately it's quite hard to read all the scripts where everything is mixed together with multiple different downloading tools (since that takes most of the time).

Another option would be (if you care about anywhere from seconds to minutes, I have not measured performance gain) to not run PowerShell, since, lets be honest here, its start-up and performance is just not great in comparison to bash.
Either commit to it completely or don't use it at all.

I've never used packer so I don't know it's full capabilities/quirks but it's been quite an effort to port one image to Docker and it took lots of time debugging whole thing. It would have been easier if it was done in pure bash or pure PowerShell.

[rdilts74]

Understand

[maxim-lobanov]

Speed up .NET Core installation: actions/runner-images#2367

[jordimassaguerpla]

What about using docker? Images are much smaller than vms and you can reuse the existing layers/base images. Have you thought about it?

[catthehacker]

Images are definitely not smaller, unless you think about compressed image which is stored in registry.
Layers can be reused only when previous layers have not been modified so either you re-run without cache or you copy the build scripts before each step (DON'T).
Part of the software and software installation is dependent on systemd which is not available in Docker, e.g.: previous Ubuntu versions have very weird package dependencies which break build mid-through because systemd is not available so the same toolkit/installation scripts can't be used in Docker.
There is much different environment handling between full blown VM with OS vs Docker container.
All of that above is massive pain to make and maintain Docker image/s based on this repo.

[jordimassaguerpla]

I see you already thought about it ;)

[catthehacker]

@jordimassaguerpla I've been working on (this) for few months already where most of the software should work.

[xloem]

Adding a note that podman supports systemd much better than docker, and appears designed to be fully compatible with docker.

[catthehacker]

Will check it out

[beryl-factset]

Has anyone done a performance audit?

• What actions consume the most time?
• (package download, package installation, cleanup after the install?)
• is there a package or provider that disproportionately affects the time?

[miketimofeev]

@beryl-factset we have added a step that shows provisioners duration. That can be useful for the analysis.
Examples:
Windows 2019 build
Ubuntu 20 build

[beryl-factset]

@miketimofeev
thanks, I'll dig more into it.

[conradj3]

A bit of a crazy idea, but since we spend a decent amount of time downloading. I had a thought of checking binary hashes for installs against a Ultra/Premium storage disk (Cache) that gets attached to the image packer machine at build. I haven't really put to much engineering thought into it, but Scripts ideally would run with parallelism (if applicable) and prior to downloading they look for a hash signature hit. If found, it runs the install from the mounted Disk.

[lzandman]

My guess is that applications that use a simple copy deploy scheme (i.e. download the files and copy them to their install location; no additional install work needed) could easily be installed in parallel. But applications that are installed using package managers (NPM / choco) and/or Windows MSI packages probably can't be installed in parallel, due to the fact that those package managers need to resolve dependencies. Unless a package manager itself offers a parallel install feature (I think Yarn does this), you're pretty much stuck.

I do wonder if it would be feasible to separate some of the downloads from their accompanying install phase. Visual Studio seems like a large download. Why not start downloading those files immediately, while the system is busy installing other apps?

[beryl-factset]

Hi, we regularly see build breaks, it's nothing consistent. (one time it might be an error with a chocolatey install, the next 5 times it works fine, then a Haskell download error.)
I'm looking at ways to reduce the breaks (and speed up the builds), I found that packer supports a PACKER_CACHE_DIR (https://developer.hashicorp.com/packer/docs/configure#configuration-reference) and was thinking about attaching a cache to the builders. Has anyone tried that?
My concern is that it won't cache things downloaded though the build scrips, or is only used for plugins, so I'm checking to see anyone has tried it.

[fahhem]

For me, the vast majority of the build time is in the one large shell provisioner that takes ~60 scripts and runs them one at a time. I built a small (~100 line) Python script that takes that list of scripts and runs them in a semi-parallel manner. Firstly, it takes as arguments any script-to-script dependencies (such as install-azure-devops-cli.sh must run after install-azure-cli.sh finishes), then it creates a topological sort of the packages and runs them via a thread pool limited to half the number of cores.

I give the packer VM 8 cores and 16GB of RAM, so it's running 4 scripts in parallel, which mainly is about running apt-get in parallel with other operations (like testing). When I run it single-threaded, it takes ~48 minutes, but when I run 4 at a time it takes ~22 minutes on my machine. Not 4x faster, but pretty good! More threads don't speed it up any more, since it seems it's spending most of its critical path on apt operations.

If there's interest from the maintainers in it, I'd be happy to create a PR for it

[fahhem]

PR opened at actions/runner-images#11467 to start the process of adding parallelism. I don't want to throw a large PR at this repo, so I'll send it in a few PRs instead.