Builder returns success after failing to push to the registry

[pguelpa]

We've experienced the builder returning success after failing to push to the registry when the registry is down. In our deploy process this ends up giving us false positives that code has been shipped. Ideally we would have this fail with a non-zero exit status.

error   09-Sep-2014 12:38:33    remote: �[1G-----> Go app detected�[K
error   09-Sep-2014 12:38:33    remote: �[1G-----> Using go1.2.2�[K
error   09-Sep-2014 12:38:33    remote: �[1G-----> Running: godep go install -tags heroku ./...�[K
error   09-Sep-2014 12:38:46    remote: �[1G-----> Discovering process types�[K
error   09-Sep-2014 12:38:46    remote: �[1G       Procfile declares types -> web�[K
error   09-Sep-2014 12:38:49    remote: �[1G-----> Compiled slug size is 4.1M�[K
error   09-Sep-2014 12:38:49    remote: -----> Building Docker image�[K
error   09-Sep-2014 12:38:49    remote: Sending build context to Docker daemon 524.8 kB�[K
error   09-Sep-2014 12:38:49    remote: Sending build context to Docker daemon 1.049 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 1.574 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 2.124 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 2.681 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 3.238 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 3.795 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 4.353 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon  4.91 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 5.434 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 5.959 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 6.499 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 7.056 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 7.613 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon  8.17 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 8.728 MB�[K
error   09-Sep-2014 12:38:50    remote: Sending build context to Docker daemon 9.285 MB�[K
error   09-Sep-2014 12:38:51    remote: Sending build context to Docker daemon 9.842 MB�[K
error   09-Sep-2014 12:38:51    remote: Sending build context to Docker daemon  10.4 MB�[K
error   09-Sep-2014 12:38:51    remote: Sending build context to Docker daemon 10.62 MB�[K
error   09-Sep-2014 12:38:51    remote: 
error   09-Sep-2014 12:38:51    remote: 
error   09-Sep-2014 12:38:51    remote: Sending build context to Docker daemon �[K
error   09-Sep-2014 12:38:51    remote: 
error   09-Sep-2014 12:38:51    remote: Step 0 : FROM deis/slugrunner�[K
error   09-Sep-2014 12:38:51    remote:  ---> b60688653de0�[K
error   09-Sep-2014 12:38:51    remote: Step 1 : RUN mkdir -p /app�[K
error   09-Sep-2014 12:38:51    remote:  ---> Using cache�[K
error   09-Sep-2014 12:38:51    remote:  ---> df69de4d0262�[K
error   09-Sep-2014 12:38:51    remote: Step 2 : ADD slug.tgz /app�[K
error   09-Sep-2014 12:39:12    remote:  ---> 61a102219d14�[K
error   09-Sep-2014 12:39:13    remote: Removing intermediate container f010ae485bd2�[K
error   09-Sep-2014 12:39:13    remote: Step 3 : ENTRYPOINT ["/runner/init"]�[K
error   09-Sep-2014 12:39:14    remote:  ---> Running in fa3e3e7898c2�[K
error   09-Sep-2014 12:39:32    remote:  ---> f2bd4c687783�[K
error   09-Sep-2014 12:39:32    remote: Removing intermediate container fa3e3e7898c2�[K
error   09-Sep-2014 12:39:32    remote: Successfully built f2bd4c687783�[K
error   09-Sep-2014 12:39:33    remote: /bin/sh: 1: cannot open no: No such file�[K
error   09-Sep-2014 12:39:33    remote: Could not tag Docker image�[K
error   09-Sep-2014 12:39:33    To ssh://git@deis.somedomain.io:2222/nyc.git
error   09-Sep-2014 12:39:33       de37a94..04b536c  master -> master
simple  09-Sep-2014 12:39:33    Finished task 'Push to Deis' with result: Success

[mboersma]

This is similar to #1622. We aren't handling celery task return codes well and need to fix this in general.

[Edit:] Or is this in a celery task context? I may be confused.

[bacongobbler]

Interesting. The exception that's being raised is right here: https://github.com/deis/deis/blob/master/builder/templates/builder#L108

I wonder why it's returning successfully when it should be failing. It must be something to do with how it's architected with git. The builder is a post-receive script, yet the git push finished successfully because the repository was successfully pushed. We'll have to investigate on how we can make git return a failure from the post-receive script.

[pguelpa]

@bacongobbler - Looks like you're right, the internet is telling me the only way to work-around it is by grepping for log output 😦

Maybe standardizing the deis client logs to have a log level as prefix could help in these cases? Then we could at least grep for an ^ERROR output

[evadne]

The post-receive hook, by design, can not reject pushes. I still think it’s better to move compilation to pre-receive. (Heroku does that.) Suggested workflow:

• Spin up the builder in pre-receive.
  • Create artifacts for the incoming commit.
  • Decline push if artifacts failed to build.
• At the end of the push you will always have a Docker image linked to your HEAD.
  • Send the release linked to HEAD to fleet if if differs from the current one

👍 ? 👎 ?