build post-commit container names are not unique

[bparees]

I ran a build, stopped my system, started it again, ran the build again, so now i get this error:

F0328 18:28:49.580909       1 builder.go:204] Error: build error: create container "openshift_s2i-build_ruby-sample-build-1_p1_post-commit": container already exists

users shouldn't have to clean up all their dead containers to be able to use this function. I think we need to append a unique suffix to the container name, if we're going to explicitly name them.

@mfojtik @rhcarvalho

[bparees]

(by stopped my system, i mean cleaned out etcd/etc. hence the reason my build sequence id went back to "1" and caused the conflict)

[bparees]

reducing to p3 because i think i hit this via a timing window (I killed my setup while the container was running, whereas normally the code would have removed the container after it finished). so it's less likely someone hits this under normal circumstances.

[rhcarvalho]

I suspect the "clean up" left some dirty state? It is possible that if you interrupt the server while a build hook is running, the hook container will not be cleaned up, leading to the situation described in the issue.

Perhaps now our clean up scripts should remove not only k8s_* prefixed containers, but also openshift_*...

And we should also fix openshift/source-to-image#433 to "namespace" containers created by S2I, and remove them in the clean up script.

Here's the original discussion where it was decided that we don't want the random suffix: #6715 (comment)

[bparees]

I suspect the "clean up" left some dirty state? It is possible that if you interrupt the server while a build hook is running, the hook container will not be cleaned up, leading to the situation described in the issue.

right, that's what i was implying in my follow up comment.

Perhaps now our clean up scripts should remove not only k8s_* prefixed containers, but also openshift_*...

perhaps, but i still think we should be using unique names. k8s/openshift do not have this problem for other containers they launch (you don't have to clean up dead containers to rerun something).

And we should also fix openshift/source-to-image#433 to "namespace" containers created by S2I, and remove them in the clean up script.

+1

Here's the original discussion where it was decided that we don't want the random suffix: #6715 (comment)

thanks, it's a fair point i suppose, but if you've got two concurrent builds running w/ the same name, running two instances of the post-commit image seems like the least of your worries (they are unlikely to intterfere with each other anyway). Not to mention it could happen if the two builds were running on distinct nodes anyway, so using the same name doesn't actually protect you from all that much. @smarterclayton ?

[smarterclayton]

I would expect the post-commit container names to be correlated to the
build and build namespace effectively so that they are unique across all
builds, for sure. I don't think we have to protect against two builds
running with the same name (they'd have to have the same pod, which means
something in Kube is horribly broken that we would fix first).

On Tue, Mar 29, 2016 at 8:50 AM, Ben Parees notifications@github.com
wrote:

I suspect the "clean up" left some dirty state? It is possible that if you
interrupt the server while a build hook is running, the hook container will
not be cleaned up, leading to the situation described in the issue.

right, that's what i was implying in my follow up comment.

Perhaps now our clean up scripts should remove not only k8s_* prefixed
containers, but also openshift_*...

perhaps, but i still think we should be using unique names. k8s/openshift
do not have this problem for other containers they launch (you don't have
to clean up dead containers to rerun something).

And we should also fix openshift/source-to-image#433
openshift/source-to-image#433 to "namespace"
containers created by S2I, and remove them in the clean up script.

+1

Here's the original discussion where it was decided that we don't want the
random suffix: #6715 #6715
(comment)

thanks, it's a fair point i suppose, but if you've got two concurrent
builds running w/ the same name, running two instances of the post-commit
image seems like the least of your worries (they are unlikely to intterfere
with each other anyway). Not to mention it could happen if the two builds
were running on distinct nodes anyway, so using the same name doesn't
actually protect you from all that much. @smarterclayton
https://github.com/smarterclayton ?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#8266 (comment)

[rhcarvalho]

Looks like we have at least two ways out here:

1. Add a random suffix, just like Kubernetes does, as per the original design.
2. Before starting the hook, force terminate any existing containers with a conflicting name, as per this comment from @smarterclayton.

[bparees]

personally i like the random suffix

1. aligns w/ k8s
2. avoids inadvertently terminating something important. (well, much less likely anyway)

[bparees]

(and we already have pruning to take care of old containers in case we accidentally leave one around...though i'm not sure if pruning applies to these containers?)

[rhcarvalho]

i'm not sure if pruning applies to these containers

Most likely it does not apply, since nothing outside of the builder knows about it...

[bparees]

Most likely it does not apply, since nothing outside of the builder knows about it...

right, i'm not sure how pruning identifies containers for pruning. I actually think it may prune any stopped containers not associated w/ a pod, in which case these would qualify. If not, our assemble containers would also not be subject to pruning, which is a bigger problem.

@ncdc can you comment on how containers are identified for pruning?