Allow docker image creation timestamp to be configurable

[philipp-paland]

As mentioned in #20126 (comment) I'm creating an enhancement request.

For good reasons (reproducibility AFAIK) the docker image created by e.g. the maven plugins have their creation timestamp set to a fixed point in time (1970-01-01?). Other tools that use the same approach have a way to customize this, e.g with jib you can set creationTime to USE_CURRENT_TIMESTAMP. I would like to have this feature also in Spring Boot. I do not know if the build packs already support this - then it might just be a question of surfacing that through the maven plugin - or if the feature would also need to be implemented there.

My use-case is that our docker registry (Gitlab) has an expiration policy feature that uses the timestamp to delete old images. That feature is currently not usable for us.

[scottfrederick]

There is a Cloud Native Buildpacks RFC that proposed lifecycle support for customization of the dates in image metadata in a way that could be used by any platform (like pack CLI and Spring Boot build plugins), but the RFC was closed without action.

Without support for setting a current timestamp in the CNB lifecycle, the Spring Boot plugins should be able to set the createDate of the resulting image but may not have control over timestamps of all the layers in the image. I don't know how much this matter to users who don't care about build reproducibility and are willing to give that up in order to get a current createDate on the image.

[scottfrederick]

In addition to allowing configuration for using the current date/time, this enhancement could allow the user to provide their own fixed date/time as an alternative to the default epoch createDate (1980-01-01T00:00:01Z). Marking this as pending-design-work so we can discuss further before implementation.

[vgropp]

to specify an alternative would be very useful as we could use the git timestamp for a somewhat reproducible build.

As far as i can tell, pack does already supports this feature: --creation-time string Desired create time in the output image config. Accepted values are Unix timestamps (e.g., '1641013200'), or 'now'. Platform API version must be at least 0.9 to use this feature.

[quaff]

to specify an alternative would be very useful as we could use the git timestamp for a somewhat reproducible build.

As far as i can tell, pack does already supports this feature: --creation-time string Desired create time in the output image config. Accepted values are Unix timestamps (e.g., '1641013200'), or 'now'. Platform API version must be at least 0.9 to use this feature.

@scottfrederick Would spring boot follow up?

[plebcity]

This issue is more important now since many will be using bootBuildImage to build native images using the buildpacks.

[scottfrederick]

@quaff @plebcity This issue is still open, and marked as an enhancement for a future release. It just hasn't been a high priority compared to other things we've been working on. We can take a look at the pack --create-time implementation and revisit the prioritization.

[plebcity]

Created pull request to fix this issue: #34511

[wilkinsona]

Thanks, @plebcity. Unfortunately, we're not yet ready to accept contributions for this issue. The status: pending-design-work label indicates that we need to do some design work before an implementation can be written.

[AnakinPt]

I totally agree with @plebcity. This issue is making me totally crazy because my images are being old and being deleted by the repository in the cleanup procedures. Can we push this feature up as it causes a lot of problems?

[vgropp]

@wilkinsona @scottfrederick why is spring boot so hostile to community contributions? Is it about the exact implementation or the feature itself, like where to configure and how to call?

Of course you can criticize #34511 for the missing "old default" and aconfiguration, but it was at least a first step to solve this issue. Adding a configuraion would already lead to a "now" and "1980" solution. But closing a PR without any discussion is not really welcoming. Adding the exact date as propsed in #28798 (comment) might be a next step in that or another PR.

[plebcity]

I totally agree with @plebcity. This issue is making me totally crazy because my images are being old and being deleted by the repository in the cleanup procedures. Can we push this feature up as it causes a lot of problems?

This is exactly the issue we're having aswell. Gitlab deletes the oldest images by default.

[dirkgroot]

I think that always using the same timestamp for every Docker image is very counter-intuitive, because it's not how docker build works by default. Because of this, and because of the problems this default behaviour causes with expiration policies, IMHO, having a fixed creation timestamp should be an op-in feature instead of an op-out.

[wilkinsona]

@vgropp Sorry that my actions appeared hostile to community contributions. We really do welcome contributions as hopefully shown by the 1000s of PRs that we've merged from 100s of different contributors. Unfortunately, we can't accept everything and when we decline a proposal we try to respect the time of the potential contributor by explaining why.

In addition to labelling issues that are particularly good for a contributor with ideal-for-contribution we also try to help potential contributors by labelling issues that we're know can't yet be worked on with pending-design-work. The latter is the case with this issue.

#34511 could not be merged as it would be a breaking change for those relying on the current behavior. It would also bring us out of alignment with pack's default behavior that we try to match. We try to match it so that people can move from one to the other as smoothly as possible. Unfortunately, this means that if we had accepted the PR we would then have had to do additional work to address these two problems before the next milestone was released. Unfortunately, we have too many higher priority work items on our plates at the moment to make that time commitment.

Looking forward, we're pretty sure that we'll need a configuration option, but we don't yet know what we want it to be and how it will be exposed in the Maven and Gradle plugins. Once we've had the time to do that design work, we'll then know what work needs to be done and where. We'll then be more than happy to accept contributions in this area.

[plebcity]

@vgropp Sorry that my actions appeared hostile to community contributions. We really do welcome contributions as hopefully shown by the 1000s of PRs that we've merged from 100s of different contributors. Unfortunately, we can't accept everything and when we decline a proposal we try to respect the time of the potential contributor by explaining why.

In addition to labelling issues that are particularly good for a contributor with ideal-for-contribution we also try to help potential contributors by labelling issues that we're know can't yet be worked on with pending-design-work. The latter is the case with this issue.

#34511 could not be merged as it would be a breaking change for those relying on the current behavior. It would also bring us out of alignment with pack's default behavior that we try to match. We try to match it so that people can move from one to the other as smoothly as possible. Unfortunately, this means that if we had accepted the PR we would then have had to do additional work to address these two problems before the next milestone was released. Unfortunately, we have too many higher priority work items on our plates at the moment to make that time commitment.

Looking forward, we're pretty sure that we'll need a configuration option, but we don't yet know what we want it to be and how it will be exposed in the Maven and Gradle plugins. Once we've had the time to do that design work, we'll then know what work needs to be done and where. We'll then be more than happy to accept contributions in this area.

Until this is fixed we can't use the bootBuildImage step since images keep getting deleted from our registry. What would be the best workaround? Create our own DockerFile with a builder and run image or create a DockerFile which has the bootBuildImage output as a base image? Is there another way to get the image with the proper created date?

[ThomasVitale]

@plebcity For the time being, you can replace ./gradlew bootBuildImage with pack build --builder paketobuildpacks/builder:base and use the --creation-time argument (see pack docs) to specify your desired creation time. The optional arguments available in bootBuildImage can be similarly passed to pack.

One of the main features of Cloud Native Buildpacks is reproducibility, which is why the creation timestamp is set to a conventional date by default so to always achieve the same result if the input is unchanged.

In the case of images built in a reproducible way, it's common to define expiration policies on a container registry based on the "last used time" rather than "creation time". For example, an image could be deleted if it's not been used for more than 6 months rather than because it's been 6 months since its creation.

I hope that helps.

[quaff]

It's hopeful if bootBuildImage use last modified of most recently modified file in root project directory as creation time by default, user can override it by customizing bootBuildImage task passing their own creation time like latest git commit date.

[wilkinsona]

@plebcity, the suggestion from @ThomasVitale is exactly what I would have suggested. Thanks, Thomas.

[wilkinsona]

It's hopeful if bootBuildImage use last modified of most recently modified file in root project directory as creation time by default

@quaff We prefer to align our default behavior with that of the pack CLI.

[plebcity]

For anyone looking for a workaround, we've added a Dockerfile which has the output image from bootBuildImage as the base image (in FROM). Then we've added a label to the image to create a new layer. Before we publish our image to the registry we build it first (adding the new layer). The output of bootBuildImage is still reproducible like the Spring team wants and when we want to publish our image we then add a new layer which adds a proper created date.

[AnakinPt]

At the moment, until we have a good solution, I'm back to the old dockerfile and it should work nice.

[plebcity]

It's hopeful if bootBuildImage use last modified of most recently modified file in root project directory as creation time by default

@quaff We prefer to align our default behavior with that of the pack CLI.

pack's implementation seems like a workaround for the fact that they couldn't figure out how to make it reproducible with a "now" created date. Why would anyone expect that a "final" image that is ready to be published has a created date of windows epoch? I'm fine with intermediate/test images having windows epoch but not a final image which is meant to be published to a repository.

So I think the default behaviour should be what everyone expects to happen, so if we compare this with any other build tool we would get "now" as default, not windows epoch. Reproducible images for the only benefit of being able to build the exact same image with the same id's on someone else's machine seems like an edge case scenario that most people won't be interested in.

You can read the reasoning why pack chose windows epoch here: https://medium.com/buildpacks/time-travel-with-pack-e0efd8bf05db

So make "now" default and windows epoch configurable.

[snagiel]

@plebcity For the time being, you can replace ./gradlew bootBuildImage with pack build --builder paketobuildpacks/builder:base and use the --creation-time argument (see pack docs) to specify your desired creation time. The optional arguments available in bootBuildImage can be similarly passed to pack.

One of the main features of Cloud Native Buildpacks is reproducibility, which is why the creation timestamp is set to a conventional date by default so to always achieve the same result if the input is unchanged.

I appreciate the workaround suggestion, but IMHO having to install the pack-cli separately -- in local environments and/or CI/CD agents -- adds a layer of complexity that using Gradle and Spring Boot should shield most developers from.

Agreed 100% with @plebcity above that the default behavior when building images should be tagged with the current timestamp, since that's logical and expected. The "reproducibility" goal is understandable to some extent, but is not really worth debating; a simple mechanism should be exposed to override the created label for testing reproducible purity, but then correct/logical values assigned when building for actual deployment to image repositories.

---

This probably deserves its own ticket, but it occurred to me while investigating this that this plugin does not seem to allow exposing any custom LABELs on the resulting image, which is puzzling. Doing so, generically, would at least allow the user to set them if they wanted, e.g.

tasks.getByName<BootBuildImage>("bootBuildImage") {
  labels = mapOf(
        "created" to SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssXXX").format(Date())
    )
}

[scottfrederick]

We've added the ability to set the image Created metadata field to a date and time of the user's choosing, or to the current date and time.

To preserve backward compatibility for users who rely on the current behavior, the epoch date will still be used by default to enable reproducibility. This also maintains compatibility with the default behavior the CNB specification, the pack CLI and other CNB platforms including Skaffold, and other tools like Jib.

[scottfrederick]

this plugin does not seem to allow exposing any custom LABELs on the resulting image, which is puzzling. Doing so, generically, would at least allow the user to set them if they wanted

@snagiel The Created field is not technically a label as can be set with the LABEL instruction in a Dockerfile and viewed with docker inspect image-name --format='{{json .Config.Labels}}'. It is image metadata, which is controlled by CNB buildpacks.

If you are using the Paketo buidpacks you can set true labels on generated images by setting environment variables in the Spring Boot build plugins that configure the Paketo Buildpack for Image Labels.