Automate build system

[smlx]

Checklist

• Affected Issues have been mentioned in the Closing issues section
• Documentation has been written/updated.
• Changelog entry has been written

The idea behind this change is that the image names are already encoded in directory names, and build dependencies between images are already defined in the Dockerfiles themselves. So let's parse that information to generate build targets and dependencies automatically, and avoid the error-prone process of manual Makefile editing. These generated make rules are then written to .docker.mk, which is included in the main Makefile. This will generate a build:<name> target for each image.

Note that this new system doesn't touch any files like the existing build system, but docker image caching is pretty fast - a second run of make build:all -j$(getconf _NPROCESSORS_ONLN) takes <20s on my system.

Here are the new commands:

help                                 Display this help section
minishift/start                      Create and start local minishift cluster, and configure for Lagoon
build:s3-save                        Save container images to S3
build:s3-load                        Load container images from S3
build:push-minishift                 Push any available built images to the local minishift registry
publish:amazeeio-baseimages          Push base images to the amazeeio org on docker hub
publish:amazeeiolagoon-baseimages    Push base images to the amazeeiolagoon org on docker hub
publish:amazeeiolagoon-serviceimages Push service images to the amazeeio org on docker hub
build:all                            Build all lagoon images
build:list                           List all image build targets
build:pull                           Pull all third-party images that Lagoon bases its images on, and write report to pull-report.json

Most of these are functionally identical to the targets they replace, but there are a few new ones..

• The old minishift target is now split into minishift and minishift/start. This is done becaues the former actually creates a ./minishift file which is a dependency of other targets. The latter does the actual configuration of minishift for local lagoon development.
• build:pull pulls all the base images that lagoon builds on into the local docker image cache, and generates a pull-report.json containing the image names/tags and hashes of these base images. This could be published as a release artefact to make it easier to know exactly which image versions make up a lagoon release.
• help (the default target) prints a simple display of the major Makefile commands.

I've also made some small changes to simplify the layout of the lagoon images, consolidating Dockerfiles where possible (e.g. php, ELK). The layout could probably be simplified further, which would then allow further simplification of the build scripts.

Changelog Entry

Improvement - automate build system

Closing issues

Closes #981, #1376, #1482, #1499.
Supersedes #982.

[tobybellwood]

I really like this.

Looking at the section used to build multi-version images, we can probably look to extend that a little to additionally specify the alpine version for backwards compatibility

PHP_VERSIONS := 5.6 7.0 7.1 7.2 7.3 7.4
NODE_VERSIONS := 6 8 9 10 12
PYTHON_VERSIONS := 3.7
SOLR_VERSIONS := 5.5 6.6 7.5

We'd need an elegant way to add an ALPINE_VERSION to correspond to specific PHP/NODE/PYTHON etc versions - this could be part of a different piece of work to pin image versions better and avoid suprises

PYTHON_VERSIONS := 2.7|3.10 3.7|3.11

[smlx]

I've split these PRs out of this branch for easier merging: #1503, #1516, #1520, #1524, #1525, #1526

Once those are in I'll rebase on master and this one should be ready to review.

[smlx]

Closing this since I think it's been superseded by https://github.com/uselagoon/lagoon-images