use github actions to build docker images

[sidey79]

Build process via github actions

• Added a workflow which builds the fhem docker image for serveral architectures via buildx supported plattforms
• Upload for test of the images currently provided to github container registry if there is a secret token provided,

disabled travis CI builds

• Due to missing travis credits, the travis yaml file is renamed to not force any more builds

Changes in Dockerfile

• ARG statements are initialized with default params and as late as possible to avoid cache invalidation

• COPY statements combined to reduce layers

• RUN statements are sorted to get most caching for different builds

• Reference version of base dockerimage and also reference apt version to invalidate cache if new version is needed

• Limited nodejs support for plattforms where it is supported, some more work can be done here using a already existing nodejs image, but may it is better to run this in a separate docker server

• Usage of ARCH variable has changed, because ARCH itself isn't a common variable from buildx instead TARGETARCH is used

• Integrationtest script runs against amd64 image

• build script now uses buildx, buildx is installed if needed

• Unittests are executed against amd64 image

• Added dependabot script to detect changed base image

• Build tow images vairiants using different layer options via separate action step

• Build now uses the slim image from debian as docker base image

• Build time reduced due to caching of fhem-svn and docker build cache

• Upload docker image to github container registry

Todos not possible in this PR

• Finish upload image to docker hub., Needs to be done in the target repo with the needed rights
• Update Readme to instruct about available features per supported platform

Relates to: #28

[jpawlowski]

These lines make sure these environment variables will still be available after "FROM". They shall not be removed to make sure this information can be accessed later on during runtime (e.g. by dockerinfo module)

[sidey79]

These are no ENV variables, these are ARG, which are available only during Build process, but not part of the image.
And because i haven't found any location where these ARG are used, i removed them to clean up orphans.

ARG variables are not persisted into the built image as ENV variables are

[jpawlowski]

these ARGs are there so people can use them when building a local image themselves.

[jpawlowski]

you have to repeat BASE_IMAGE and BASE_IMAGE_TAG after FROM, otherwise they will not be available for other commands anymore (e.g. if used by somebody adding some customizations to the build process)

[sidey79]

If someone want's to modify the build process ( dockerfile ) he can add it where he needs it i think.
I removed it, because currently it's nowhere used, but it woud break caching as soon as it is defined with a different value.

[jpawlowski]

It seems new variables TARGETARCH and TARGETPLATFORM are used inconsistantly. This shall be corrected before merging into dev branch.

[jpawlowski]

Also make sure the content variable is not tied to be available on Github Actions only as otherwise local builds on personal devices may break.

[sidey79]

TARGETARCH and TARGETPLATFORM are standard docker variables, noting special to github actions:

https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope

TARGETARCH isn't used on this line, what is your approach?

[jpawlowski]

Be careful changing these to have a default value as on Travis CI, it wouldn't work the way it is now. May be different on Github Actions, to be verified.

[sidey79]

Before i assigned the default values, it was not possible to disable CPAN, PIP, NPM and so on, how did this work before?

[sidey79]

Just to get an idea, how the unset AGRG operated before:

buildx call failed with: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c if ( [ "${NPM_PKGS}" != "" ] \|\| [ "${IMAGE_LAYER_NODEJS}" != "0" ] \|\| [ "${IMAGE_LAYER_NODEJS_EXT}" != "0" ] ) && ( [ "${ARCH}" == "AMD64" ] \|\| [ "${ARCH}" == "ARMv7" ] \|\| [ "${ARCH}" == "ARM64v8" ] ] \|\| [ ${TARGETPLATFORM} == 'linux/amd64' ] \|\| ${TARGETPLATFORM} == 'linux/arm/v7' ] \|\| ${TARGETPLATFORM} == 'linux/arm64' ] \|\| ); then 

My opinion is, that this comes from the Dockerfile itself how it is preprocessed, but may i am wrong here.

[sidey79]

I set NPM_PKGS to "" becaue it it later tested against "" and not against 0 as the LAYERS.

[jpawlowski]

Please don't exclude any of the layers as it will otherwise break existing configurations.
All layers are essentially needed for maximum compatibility with existing FHEM modules.

[sidey79]

I excluded the layers, because espacally nodejs 14 isn't supported on some of the platforms anymore and this results in different behaviours of the images between platforms.
I am aware of, that this is may be a breaking change for some special cases. But for this cases it is still possible to install the needed packages via docker argument -E.

To save some time during testing an get some minmal working image i decided to disable all those packages to get a pure fhem image which can run fhem similar to the debian package which also has no dependencys to nodejs, python and so on. This redudces also some complexity during setup of the gighub workflow.
Currently the images aren't tested because they are nowhere published. In my oppinion it's best to first check if the base image works on the desired platforms and then move on to extend them again.

For nodejs there are also already available images which may can run the needed services for thos fhem modules who communitcate with a nodejs service . Also the health-check.sh isn't aware of the nodejs services or other laayer, so may it is better to move them to a separate service which is maintained seperate.

IMAGE_LAYER_DEV seems to bring in tools to compile software, which should normaly be not be included in a normal d docker image. Also all this stuff can be added via -E argument if really needed.

Maybe it's worth of providing some -minimal and some -full image variants, depending on the platform. But for now, my goal was to build something of a base image.

[jpawlowski]

The different layers for platforms are intentionally to provide maximum possibility on each platform. Providing even more variants is not beneficial in terms of increased maintenance and support effort.

Build tools in the image are required to support FHEM Installer and related components to install Perl modules and other stuff automatically.

[sidey79]

If the build tools are only needed to install the packages into the image, we can remove them after the installation process is done in a seperate layer.

There is also a job to create a image which has all that services included in the image, same as before. But i think that's not the docker way to have services in different containers.

[sidey79]

@jpawlowski

Did a lot of improvements. I think there is room left, but i think i now need some help to get correct push arguments for docker hub and later on update the readme

May we want to merge this into a feature branch instead of the dev branch to get the work done there

[sidey79]

@jpawlowski

Wie kommen wir hier weiter?
Willst Du die Repo Rechte ggf. weitegeben?

[jpawlowski]

@jpawlowski

Wie kommen wir hier weiter?
Willst Du die Repo Rechte ggf. weitegeben?

Ehrlich gesagt gefallen mir die Änderungen so überhaupt nicht. Die Abhängigkeiten zum FHEM Installer und den anderen FHEM Modulen wurden gar nicht berücksichtigt. Auch werden Änderungen vermischt, die mit dem Wechsel auf Github Actions nichts zu tun haben.

[jpawlowski]

Why would you mix this change into the Travis>GHActions migration path?

[jpawlowski]

Also, why is this method more reliable and what made you changing this command? pls explain so we have it documented as well.

[sidey79]

I've changed this, to avoid a additional package dependency in the image. The ip command comes via iproute2
It's mixed this in here, because this package isn't in the install list, so i changed to available commands.

[jpawlowski]

pls find a proper replacement for this variable as it is required to keep build information as part of the image (git commit id)

[sidey79]

You mean something like git rev-parse --short HEAD ?
I haven't included this, because in my oppinion the behaivior is the same as bevore, because TRAVIS_COMMIT isn't set somewhere inside this script, so if somebody called that script before, this was an empty variable.

[jpawlowski]

these ARGs are there so people can use them when building a local image themselves.

[jpawlowski]

you have to repeat BASE_IMAGE and BASE_IMAGE_TAG after FROM, otherwise they will not be available for other commands anymore (e.g. if used by somebody adding some customizations to the build process)

[jpawlowski]

The different layers for platforms are intentionally to provide maximum possibility on each platform. Providing even more variants is not beneficial in terms of increased maintenance and support effort.

Build tools in the image are required to support FHEM Installer and related components to install Perl modules and other stuff automatically.

[jpawlowski]

pinning was intentionally left out because there is no real value not to use the latest and greatest version...

[sidey79]

pinning is implemented, to invalidate the cache if a new version should be uses

[sidey79]

Here a reference what can bei the result If not aware of a compatible Version:

https://forum.fhem.de/index.php/topic,119201.0.html

[jpawlowski]

pinning was intentionally left out because there is no real value not to use the latest and greatest version...

[sidey79]

pinning is implemented, to invalidate the cache if a new version should be uses

[jpawlowski]

again, no pinning here. the intention is to rely on the latest version of the upstream environment.
FHEM Installer and Updater Modules would otherwise also complain about existing updates. Does not make sense to always be behind with updates even when pulling the latest image.

[sidey79]

The pinning is needed, to invalidate the cache if there is a new version. Otherwise if the layer doesn't change, the cache is never invalidated.

I'm aware, that there needs to be some update jobs done. Currently i'm hoping that this will get into dependabot soon, so every time a new package is releases, a new build can be triggered to be up to date.
With the old behaivour, i think, there was no information available what versions are installed, and if somebody builds it on his own, he will get different results for the same dockerfile and revision, if the build job is run on a different time.