Dbld image caching

[bazsi]

This branch annotates dbld images with the commit ID they were built with and do this both
for locally generated images (e.g. ./dbld/rules image-) and dockerhub generated ones.

It also adds a check-image- target that rebuilds the image conditionally, e.g. in case the dbld subdirectory has changed since the local copy. This can be used to significantly improve
CI performance if the previous image is still locally available.

[kira-syslogng]

Build SUCCESS

[bazsi]

This is a rebase issue, I will fix it up.

…

On Thu, Aug 8, 2019, 09:50 furiel ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In dbld/rules <#2858 (comment)>: > @@ -136,11 +136,6 @@ image: image-$(DEFAULT_IMAGE) image-%: cd $(DBLD_DIR)/images/ && $(DOCKER) build --network=host -t balabit/syslog-ng-$* -f $*.dockerfile . -pull-images: $(foreach image,$(BUILDER_IMAGES), pull-image-$(image)) -pull-image: pull-image-$(DEFAULT_IMAGE) These targets seemed useful. Is it necessary to remove them? Maybe this is a rebase problem. For example, I miss setting --build-arg=OS_PLATFORM in this patch, on the other hand this change is not necessary. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2858>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFOK5VJ3MABKFLTGDC45RTQDPF55ANCNFSM4IJDIB5A> .

[szemere]

Hello,

1. I have found the missing --build-arg in: 20e13cfcaf8c376cb261869f576f56941ba02958
2. Could you please provide some example of how to use this new target? (I get various errors - most probably because none of my images are labelled.)

[bazsi]

I have fixed the rebase issue, and will continue with the remaining open comments next.

[kira-syslogng]

Build FAILURE

[kira-syslogng]

Build SUCCESS

[kira-syslogng]

Build SUCCESS

[bazsi]

2. Could you please provide some example of how to use this new target? (I get various errors - most probably because none of my images are labelled.)

The point is to use this on jenkins slaves to conditionally rebuild the image, but only if needed. With those our distro packages build should be much faster.

This is what ./dbld/rules cache-image-ubuntu-xenial would do:

1. check if we have any build image for ubuntu-xenial locally
2. if there's nothing, it would perform a docker pull from dockerhub
3. check if the local version is "suitable" for building the current source tree (the "suitability" question is what we've discussed in the previous comments).
4. if it is suitable, do nothing, if it's not rebuild it.

With those you can be sure that a dbld image is available by the time the target finishes, and in ideal conditions this would only take less than a second.

Ideal conditions in the order of "ideality":

1. local image exists and is suitable (e.g. no dbld changes)
2. docker pull gets us a good enough version, thus no need to rebuild
3. we originally had a local image, but it is not suitable, so we need to rebuild
4. we didn't have an image, so we pulled one, but is not suitable and we also need to rebuild

So worst case is that we pull AND rebuild. This could be improved by first checking that the version on dockerhub is suitable before doing the pull, but I haven't implemented that.

[kira-syslogng]

Build SUCCESS

[bazsi]

On Thu, Aug 22, 2019, 11:27 László Szemere ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In dbld/rules <#2858 (comment)>: > @@ -134,12 +134,19 @@ shell-%: setup images: $(foreach image,$(IMAGES), image-$(image)) image: image-$(DEFAULT_IMAGE) image-%: - cd $(DBLD_DIR)/images/ && $(DOCKER) build --network=host -t balabit/syslog-ng-$* -f $*.dockerfile . + cd $(DBLD_DIR)/images/ && $(DOCKER) build --build-arg=OS_PLATFORM=$* --build-arg=COMMIT=$$(git rev-parse --short HEAD) --network=host -t balabit/syslog-ng-$* -f $*.dockerfile . + +check-image-%: + @image_commit=$$(docker inspect --format '{{ index .Config.Labels "COMMIT"}}' balabit/syslog-ng-$*); \ + dbld_changed=$$(git diff $${image_commit:-NO_COMMIT_LABEL_IN_DBLD_IMAGE} -- dbld | wc -l); \ First things first. I do not have a proposal yet, and I don't know a better solution. Sorry for that. My concerns/how do I see it: With image caching we introduced the plain old caching problem. We don't know what time we should refresh the cache. - Currently we do not follow / mark the versions of the upstream dependencies. - Using COMMIT instead of DBLD_COMMIT "just" increase the refreshing frequency, but still don't make any logical connection with the content of the image. (i.e.: a simple date check will do the same) Totally agree with Bazsi's 1. point. I think there's a lot of temporal state encapsulated in an image. Which means that one dbld revision is not the same as another one depending on the time the image was generated (because apt/pip/obs/etc changes) Only my conclusion differs. In theory: Two docker image should be identical if DBLD_COMMIT hasn't change, regardless of the master commit. And neither one of them is suitable to track the "temporal state" changes.

Our current set of requirements don't need precise tracking of dependencies, as our build products don't include those (well except the docker image that is). As our users install the deb/rpm on their systems, their specific dependencies may be different from what we built syslog-ng with. And this is not a problem as long as those dependencies keep binary compatibility, which they do in a specific distro version. So again, with dbld and with the packages it generates, we don't really care the exact dependency version as long as the runtime package of those dependencies are available and they keep binary compatibility. This may become different with production docker images and with installers that bundle dependencies. But I argue they should use a different logic for building and those are in an entirely different layer on top of what the syslog-ng OSE source tarball provides. —

…

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2858>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFOK5SAUMNEXGMDYT3K2V3QFZLWTANCNFSM4IJDIB5A> .

[bazsi]

I think I have answered all open concerns, so this should be ready to go in. I have two branches building on top of this one, so it'd be great if we could expedite this.

thanks.

[bazsi]

Any updates on when this could go in? I have works in progress in other branches but dont want to submit them until the dbld blockage clears up.

[bazsi]

Thank you!

…

On Wed, Sep 18, 2019, 13:14 László Várady ***@***.***> wrote: Merged #2858 <#2858> into master. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2858>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFOK5XSJAWM5MKI46FGI53QKIETFANCNFSM4IJDIB5A> .