Build process parallel processing

[ccjjmartin]

Checklist

• Affected Issues have been mentioned in the Closing issues section
• Documentation has been written/updated.
• Changelog entry has been written

Changelog Entry

Improvement - Build process parallel processing (#981)

Closing issues

closes #981

[ccjjmartin]

@rocketeerbkw After this is merged updating the Jenkins test scripts with this make build-list | xargs -n 1 -P 5 make would likely speed the CI environment up on the build portion of things. My initial testing of this showed processes taking around 20 minutes drop to 2-3 minutes. I am interested in what the results are like on a macbook pro.

[ccjjmartin]

I made the assumption that the make build-list command is outdated and should match what the build command is doing.

[ccjjmartin]

Looks like Jenkins only takes 18 seconds if the images are already built so there may not be a huge gain on that side of things. This may only be on fresh builds: https://lagoon-ci.amazeeio.cloud/blue/organizations/jenkins/lagoon/detail/PR-982/1/pipeline/17

[rocketeerbkw]

@ccjjmartin the parallelization technique does not work for me. Instead of running 5 "threads" with one make build each, it runs 5 "threads" of all builds, so it basically builds each image 5 times which is not only not faster than normal, but a lot slower ;)

➜ lagoon git:(upgrade-deps) ✗ make build-list | xargs -n 1 -P 5 make
docker build --quiet --build-arg LAGOON_VERSION=development --build-arg IMAGE_REPO=lagoon -t lagoon/commons -f images/commons/Dockerfile images/commons
docker build --quiet --build-arg LAGOON_VERSION=development --build-arg IMAGE_REPO=lagoon -t lagoon/commons -f images/commons/Dockerfile images/commons
docker build --quiet --build-arg LAGOON_VERSION=development --build-arg IMAGE_REPO=lagoon -t lagoon/commons -f images/commons/Dockerfile images/commons
docker build --quiet --build-arg LAGOON_VERSION=development --build-arg IMAGE_REPO=lagoon -t lagoon/commons -f images/commons/Dockerfile images/commons
docker build --quiet --build-arg LAGOON_VERSION=development --build-arg IMAGE_REPO=lagoon -t lagoon/commons -f images/commons/Dockerfile images/commons

Maybe this is a OS difference with make + xargs?

[tobybellwood]

We've been working separately on image building and parallelisation - will release in the coming weeks.