New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add git commit hash to ENV #600

Open
trungpham opened this Issue Mar 3, 2016 · 22 comments

Comments

Projects
None yet
8 participants
@trungpham

trungpham commented Mar 3, 2016

Is it possible to add a new GitCommit environment variable? So the Dockerfile can read that environment variable at build time.

@naartjie

This comment has been minimized.

naartjie commented Jul 6, 2016

That would be awesome. Right now I have a bit of a nasty hack for this. I install git in the docker container first, and then:

# version info
RUN \
  VERSION=$(git rev-parse --short HEAD) && \
  DATE=$(date +%Y-%m-%dT%H:%M:%S) && \
  if ! [[ -z "`git status -s`" ]]; then VERSION="!! DIRTY ${VERSION}"; fi && \
  sed -i "s/@@__VERSION__@@/${VERSION}/g;s/@@__BUILT__@@/${DATE}/g" ./src/main/resources/VERSION.edn

Not amazing in terms of verbosity, but it works. GitCommit as an env variable would be a winner though.

@pchico83

This comment has been minimized.

pchico83 commented Jul 6, 2016

We inject the SOURCE_COMMIT env variable in the builder container.
In order to use it in your docker file, you will need to add it as a build arg and define a build hook to pass this variable as a build arg:

docker build -t $IMAGE_NAME --build-arg SOURCE_COMMIT=$SOURCE_COMMIT .

Hooks are documented here https://docs.docker.com/docker-cloud/builds/automated-build/#/use-custom-build-phase-hooks. Although the build hook is not documented, it replaces the build command that we execute by default.

@naartjie

This comment has been minimized.

naartjie commented Jul 6, 2016

Thanks @pchico83, so this is part of Docker Cloud only? Is there a way if I am using just a standard docker hub automated build from a github repo? Or should we just migrate to using Docker Cloud instead?

@pchico83

This comment has been minimized.

pchico83 commented Jul 7, 2016

This is also available in Docker Hub although it is not documented. I would recommend to switch to Docker Cloud, which has more features than Hub.
Some new Docker Cloud features include:

  • Run tests for pull requests.
  • Integration with github/bitbucket commit statutes.
  • Slack notifications
  • Ability to cancel and retry a running build.
@naartjie

This comment has been minimized.

naartjie commented Jul 7, 2016

Epic, thanks for confirming that @pchico83, will make a switch at some stage.

@ahaasler

This comment has been minimized.

ahaasler commented Sep 24, 2016

Thanks @pchico83, but I'm having problems trying to use build args on Docker Hub. I used some of the variables described on https://docs.docker.com/docker-cloud/builds/advanced/ (not docker hub, but you said it's also available there) and I can't get them working on my container.

You can see what I did on ahaasler/docker-jira@a6e8018. Am I using it wrong?

If I migrate these repos to Docker Cloud what happens to the page in the Hub? Does it still refresh the readme and all repo info on GitHub pushes? Does it still get marked as Automated build?

@pchico83

This comment has been minimized.

pchico83 commented Sep 26, 2016

@ahaasler You need to define a build hook to pass the variables into the Dockerfile arguments:

#!/bin/bash 
docker build -t $IMAGE_NAME --build-arg SOURCE_COMMIT=$SOURCE_COMMIT--build-arg DOCKER_TAG=$DOCKER_TAG .

The route of this file is hooks/build at the same path as your Dockerfile.

@ahaasler

This comment has been minimized.

ahaasler commented Sep 26, 2016

@pchico83 thanks, but $SOURCE_COMMIT doesn't contain anything. I'm echoing the value in the hook and it's empty.
Hook: https://github.com/ahaasler/docker-jira/blob/e9f599e81e89c92d323f09eb6f4d7cafad70414d/hooks/build
Log: https://hub.docker.com/r/ahaasler/jira/builds/bfwgkvgjy343pcnehxbkm6h/

$IMAGE_NAME and $DOCKER_TAG are working as expected.

@pchico83

This comment has been minimized.

pchico83 commented Sep 26, 2016

SOURCE_COMMIT is empty for manual builds. I have created an internal issue to fix it, in the meantime you can get it by executinbg:

git rev-parse HEAD

as part of your build hook.

@ahaasler

This comment has been minimized.

ahaasler commented Oct 2, 2016

@pchico83 thanks for the suggestion, now it's working as it should, but I'm getting the same error on random builds since I set up the build hook. You can see that some of the last builds (https://hub.docker.com/r/ahaasler/jira/builds/) are failing in the COPY launch.sh /launch instruction.

@pchico83

This comment has been minimized.

pchico83 commented Oct 7, 2016

@ahaasler We are working on a fix for and it will fixed in the next days. I will let you know when the fix is available.

@pchico83

This comment has been minimized.

pchico83 commented Oct 26, 2016

@ahaasler the fix took longer than expected, but the COPY instruction should be working now.

@ahaasler

This comment has been minimized.

ahaasler commented Oct 26, 2016

@pchico83 thanks!

@lilith

This comment has been minimized.

lilith commented Jan 6, 2017

SOURCE_COMMIT is still missing. Still gotta use git rev-parse HEAD

#!/bin/bash

echo "SOURCE_COMMIT: $SOURCE_COMMIT"

if [[ -z "$SOURCE_COMMIT" ]]; then
	export SOURCE_COMMIT="${SOURCE_COMMIT:-$(git rev-parse HEAD)}"
	echo "Updating SOURCE_COMMIT from git rev-parse HEAD"
	echo "SOURCE_COMMIT: $SOURCE_COMMIT"
fi


echo "DOCKER_TAG: $DOCKER_TAG"

docker build -t "$IMAGE_NAME" --build-arg "SOURCE_COMMIT=$SOURCE_COMMIT" --build-arg "DOCKER_TAG=$DOCKER_TAG" .

jdel added a commit to jdel/sspks that referenced this issue Mar 11, 2017

Update build hook
Partial roll back, SOURCE_COMMIT is not populated... docker/hub-feedback#600 (comment)

wvengen referenced this issue in foodcoops/foodsoft Aug 1, 2017

hemberger added a commit to hemberger/smr that referenced this issue Jan 19, 2018

hemberger added a commit to hemberger/smr that referenced this issue Feb 15, 2018

@andreif

This comment has been minimized.

andreif commented Mar 20, 2018

What is the status of this issue?

In my Dockerfile I have:

...
ARG SOURCE_COMMIT
ENV SOURCE_COMMIT $SOURCE_COMMIT
RUN echo "SOURCE_COMMIT = $SOURCE_COMMIT"
...

Docker Cloud build log is:

...
Step 21/25 : ARG SOURCE_COMMIT
---> Running in 68f212bbd977
---> 66b30dfb0318
Removing intermediate container 68f212bbd977
Step 22/25 : ENV SOURCE_COMMIT $SOURCE_COMMIT
---> Running in 1434006761a7
---> cfda94d58f00
Removing intermediate container 1434006761a7
Step 23/25 : RUN echo "SOURCE_COMMIT = $SOURCE_COMMIT"
---> Running in 5b6f00b2dc9b
SOURCE_COMMIT =
---> 1555addde746
Removing intermediate container 5b6f00b2dc9b
...

If I build locally, then it works:

$ docker-compose build --no-cache --force-rm --pull \
        --build-arg SOURCE_COMMIT=$(git rev-parse HEAD) api

...
Step 21/25 : ARG SOURCE_COMMIT
 ---> Running in ed1e3850db37
Removing intermediate container ed1e3850db37
 ---> 4f658cc5b3d0
Step 22/25 : ENV SOURCE_COMMIT $SOURCE_COMMIT
 ---> Running in c6909a2fda0e
Removing intermediate container c6909a2fda0e
 ---> 79071f63927d
Step 23/25 : RUN echo "SOURCE_COMMIT = $SOURCE_COMMIT"
 ---> Running in dc255dae1e5b
SOURCE_COMMIT = 0e0f20b5dc9a38a6bf867931e23d2c6d0dde0f85
Removing intermediate container dc255dae1e5b
 ---> ce2c51c2e93d
...
@andreif

This comment has been minimized.

andreif commented Mar 22, 2018

@pchico83 Could you confirm that the build args are still provided?

@andreif

This comment has been minimized.

andreif commented Mar 22, 2018

I have noticed that I missed to update docker-compose.test.yml according to the docs.
After updating it, I still don't see $SOURCE_COMMIT set before or after using arg.

version: "3.3"

services:
  sut:
    build: .
    entrypoint:
      - wait-for-services.sh
      - postgres
      - neo4j
      - make
      - --makefile
      - ../Makefile
    command: test
    depends_on:
      - postgres
      - redis
      - neo4j
    env_file:
      - ./docker/envs/common/api.env
      - ./docker/envs/test/api.env
    environment:
      - SOURCE_COMMIT
...
...
RUN echo "SOURCE_COMMIT = $SOURCE_COMMIT"
ARG SOURCE_COMMIT
ENV SOURCE_COMMIT $SOURCE_COMMIT
RUN echo "SOURCE_COMMIT = $SOURCE_COMMIT"
...
...
Step 21/26 : RUN echo "SOURCE_COMMIT = $SOURCE_COMMIT"
---> Running in 8378310f89d4
SOURCE_COMMIT =
---> 2b7986207f06
Removing intermediate container 8378310f89d4
Step 22/26 : ARG SOURCE_COMMIT
---> Running in 9d2b49189a23
---> f9145e1482e8
Removing intermediate container 9d2b49189a23
Step 23/26 : ENV SOURCE_COMMIT $SOURCE_COMMIT
---> Running in aa2b63b84261
---> 0265c713a82f
Removing intermediate container aa2b63b84261
Step 24/26 : RUN echo "SOURCE_COMMIT = $SOURCE_COMMIT"
---> Running in 747d7fb8e2cb
SOURCE_COMMIT =
---> 11603c2797e5
Removing intermediate container 747d7fb8e2cb
...
@andreif

This comment has been minimized.

andreif commented Mar 22, 2018

I added two variables to config:

...
    environment:
      - SOURCE_BRANCH
      - SOURCE_COMMIT
...

and while they are not available during build step (when it runs Dockerfile?), I can see them during testing now:

...
.PHONY: test
test:
	@echo "SOURCE_BRANCH = $(SOURCE_BRANCH)"
	@echo "SOURCE_COMMIT = $(SOURCE_COMMIT)"
	@python3 manage.py test \
...
...
SOURCE_BRANCH = cloud
SOURCE_COMMIT = 2d2764edf305453aa92ef6b994dda41c99b50bd8
...
@andreif

This comment has been minimized.

andreif commented Mar 22, 2018

Alright, so I have some progress now and found out how to set build arg for sut (testing) step, which is kinda useless since I already have $SOURCE_COMMIT variable available in the test container, but anyway it's a progress. I got it by specifying build arg in sut config for build:

services:
  sut:
    build:
      context: .
      args:
        - SOURCE_COMMIT=${SOURCE_COMMIT}
...
    environment:
      - SOURCE_COMMIT
...
Step 21/26 : RUN echo "SOURCE_COMMIT = $SOURCE_COMMIT"
---> Running in a003bbeaa6c5
SOURCE_COMMIT =
---> 0683ab252cba
Removing intermediate container a003bbeaa6c5
Step 22/26 : ARG SOURCE_COMMIT
---> Running in 889c50ba8fde
---> 319583da11a0
Removing intermediate container 889c50ba8fde
Step 23/26 : ENV SOURCE_COMMIT $SOURCE_COMMIT
---> Running in ee0e2a482983
---> d06693498783
Removing intermediate container ee0e2a482983
Step 24/26 : RUN echo "SOURCE_COMMIT = $SOURCE_COMMIT"
---> Running in fbb9627fce36
SOURCE_COMMIT = b85e6f81d3a1f567534080035ebcf245a2eef347
---> cc2e2356d0d4
Removing intermediate container fbb9627fce36
...

So there must be a similar way to provide it for build step which is executed before sut.

@andreif

This comment has been minimized.

andreif commented Mar 22, 2018

Good news, my eyes see now and I got it to work with the build hook as originally proposed by @pchico83. Will there be an easier way to provide build args similar to how one can do it for sut?

hemberger added a commit to hemberger/smr that referenced this issue Apr 5, 2018

Add git-commit label to docker image
See docker/hub-feedback#600 for details.

TODO: might want to do the following instead:

COPY ./.git/HEAD .
COPY ./.git/refs ./.git/refs
RUN cat .git/$(cat .git/HEAD | awk '{print $2}') >> git-commit

hemberger added a commit to hemberger/smr that referenced this issue Apr 5, 2018

Dockerfile: Add git commit data to the docker image
Add enough data from the `.git` directory to be able to store
the commit hash of HEAD. This allows us to tell which version
of the code is running in a docker image.

Ideally we would want this in the image metadata, but I think
Docker prevents this sort of dynamic logic at build time.

We use this method in favor of two other alternatives:

1. Docker Cloud build arg

Docker Cloud provides a `$SOURCE_COMMIT` build arg, but it can
only be passed into the build environment with a custom build
hook. Unfortunately, this causes the Docker Cloud build to not
utilize the layer cache correctly, even though diagnostic
messages indicate that it is pulling/pushing the layer index
for the repo. I don't understand why it does this, and Docker
Cloud support is very limited. I suspect it is because the
default build hook (which is not public) has special magic that
the recommended build command override is missing.

See docker/hub-feedback#600 for details.

2. Custom build script

If we want to forego Docker Cloud altogether, we can write our
own custom script to build images and push them to the Docker
registry. This would give us complete flexibility, but sacrifices
automation and uses personal resources instead of free commercial
services. We may have to do this anyway if Docker Cloud continues
its pattern of deactivating service components.
@tleyden

This comment has been minimized.

tleyden commented May 23, 2018

@pchico83 regarding #600 (comment), I tried adding the following to my Dockerfile that's being built via an automated build on dockerhub:

ARG SOURCE_COMMIT
ARG SOURCE_BRANCH

RUN echo "SOURCE_COMMIT: $SOURCE_COMMIT"
RUN echo "source branch: $SOURCE_BRANCH"

However the output is empty for both:

Removing intermediate container a5d6cecb74e5
Step 4/12 : ARG SOURCE_COMMIT
 ---> Running in b10eef64d7e2
 ---> dc81847407cd
Removing intermediate container b10eef64d7e2
Step 5/12 : ARG SOURCE_BRANCH
 ---> Running in 350d5e334cf4
 ---> 5ed34c63613e
Removing intermediate container 350d5e334cf4
Step 6/12 : RUN echo "SOURCE_COMMIT: $SOURCE_COMMIT"
 ---> Running in 93cb1c19f1eb
SOURCE_COMMIT: 
 ---> 2bf7a7227cb7
Removing intermediate container 93cb1c19f1eb
Step 7/12 : RUN echo "source branch: $SOURCE_BRANCH"
 ---> Running in 7f948ef79dae
source branch: 
 ---> 7dfc73ea5a8d
Removing intermediate container 7f948ef79dae
@jri-sp

This comment has been minimized.

jri-sp commented Jun 27, 2018

I can confirm that SOURCE_COMMIT varaible doesn't work in Docker Hub automated build.

elasticdog added a commit to elasticdog/tiddlywiki-docker that referenced this issue Jul 23, 2018

Inject Git source commit metadata into the image
These values are useful for looking up the state of code when examining
images at a particular tag. When setting LABEL values, we can easily
pull them out of an image through inspection:

    docker inspect --format '{{.Config.Labels}}' tiddlywiki

The default value of `unknown` is meant to catch the accidental case
where an image was pushed manually. The naming of the `SOURCE_COMMIT`
variable itself comes from the convention used by Docker Cloud [1] (but
might still be undocumented). Labels seemed more appropriate than using
ENV variables.

[1] docker/hub-feedback#600 (comment)

elasticdog added a commit to elasticdog/tiddlywiki-docker that referenced this issue Jul 23, 2018

Inject Git source commit metadata into the image
These values are useful for looking up the state of code when examining
images at a particular tag. When setting LABEL values, we can easily
pull them out of an image through inspection:

    docker inspect --format '{{.Config.Labels}}' tiddlywiki

The default value of `unknown` is meant to catch the accidental case
where an image was pushed manually. The naming of the `SOURCE_COMMIT`
variable itself comes from the convention used by Docker Cloud [1] (but
might still be undocumented). Labels seemed more appropriate than using
ENV variables.

[1] docker/hub-feedback#600 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment