spawned slugbuilder pods finish properly, but aren't cleaned up

[bacongobbler]

non-critical, but still a bug that we should clean up after we're done with them. Tail of slugbuilder pod:

[...]
+ echo 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk 10.0.196.211:9000
8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk 10.0.196.211:9000
+ mc --quiet -C /app/mc_config config host add http://10.0.196.211:9000 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
Added host ‘http://10.0.196.211:9000’ successfully.
+ mc --quiet -C /app/mc_config cp /tmp/slug.tgz http://10.0.196.211:9000/git/home/go:git-5450cbcd/push/
‘/tmp/slug.tgz’ -> ‘http://10.0.196.211:9000/git/home/go:git-5450cbcd/push/slug.tgz’
Total: 1.62 MB, Transferred: 1.62 MB, Speed: 16.99 MB/s

><> kd describe po go-git-5450cbcd
Name:                           go-git-5450cbcd
Namespace:                      deis
Image(s):                       quay.io/deisci/slugbuilder:v2-alpha
Node:                           ip-172-20-0-29.us-west-2.compute.internal/172.20.0.29
Labels:                         heritage=deis,version=v2-alpha
Status:                         Succeeded
Reason:
Message:
IP:
Replication Controllers:        <none>
Containers:
  deis-slugbuilder:
    Image:              quay.io/deisci/slugbuilder:v2-alpha
    State:              Terminated
      Reason:           Error
      Exit Code:        0
      Started:          Thu, 17 Dec 2015 18:40:05 -0800
      Finished:         Thu, 17 Dec 2015 18:40:16 -0800
    Ready:              False
    Restart Count:      0
Conditions:
  Type          Status
  Ready         False 
No events.

[arschles]

I can reproduce this on every build, fwiw

[arschles]

The solution to this is to create slugbuilder and dockerbuilder as jobs, not pods. see https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/user-guide/jobs.md for docs on jobs.

Originally suggested in #63 (now closed, as it duplicated this issue)

[arschles]

example output from kubectl get pod -w --namespace=deis | grep slugbuilder:

slugbuild-gotest-b350616e-c55e671e   0/1       Pending   0          0s
slugbuild-gotest-b350616e-c55e671e   0/1       Pending   0         0s
slugbuild-gotest-b350616e-c55e671e   0/1       Pending   0         1s
slugbuild-gotest-b350616e-c55e671e   0/1       Running   0         2s
slugbuild-gotest-b350616e-c55e671e   1/1       Running   0         3s
slugbuild-gotest-b350616e-c55e671e   0/1       Error     0         13s

[technosophos]

I don't think this is blocking for Beta1. Can I remove it from the milestone?

[smothiki]

As each POD for each build is unique this is not a problem . Better solution would be to implement them as jobs . Removing this from milestones

[arschles]

Rel #298

[smothiki]

I think we can close this issue. As v1.2 garbage collects PODS which are successful immediately.

[kmala]

i don't think so..i still have day old slugbuilder and docker builder pods which are not Garbage collected.

[smothiki]

minimum-container-ttl-duration is 1 min default . According to docs http://kubernetes.io/docs/admin/garbage-collection/. Will research more on this

[arschles]

punting this again, this time to beta4

[jchauncey]

moved to rc1

[mboersma]

@smothiki do we know what the k8s GC behavior is here? This would be good to have fixed for rc1, or at least have the behavior known and documented.

[smothiki]

Pods are no longer showing up in error state unless there are errors in the new k8s version. I think we can close this ping @bacongobbler

[bacongobbler]

I'll defer to @kmala since he re-opened this issue.

[kmala]

Yeah...I just checked and GC seems working...hence closing