notifies :redeploy fails, but manually notifying :stop, :remove and :run works

[rdsubhas]

Reference: https://snap-ci.com/rapidftr/RapidFTR/branch/master/logs/defaultPipeline/74/DEV?back_to=build_history

notifies :redeploy, ..., :immediately

Fails randomly. Instead we have switched to:

notifies :stop, ..., :immediately
notifies :remove, ..., :immediately
notifies :run, ..., :immediately

And that works consistently well better. Any ideas what could be the issue?

EDIT: That doesn't work consistently well, it just works "better", and does fail occasionally.

[jschneiderhan]

I think it may be due to the mixing of platform resources and in-line shellouts. When I change the cookbook to use platform resources only (#197) the :redepoly action works as described. I'm not sure if switching to using platform resources is possible for all of the shellouts, but I think that may be a possible solution.

[rdsubhas]

@jschneiderhan By "in-line shellouts" you mean execute commands? (because we have only one execute and that is for cleaning up stale docker images, and I'm not sure if that could affect anything, excuse me if I misunderstood)

[jschneiderhan]

Hi @rdsubhas,

By inline shellouts I mean calls to docker_cmd! which end up here and are executed immediately. I have a hunch that mixing those calls with platform resources, which are added to the resource collection and then processed, is causing issues. I don't think the platform resources are executed immediately, which in #194 leads to the docker rm command executing before the service resource's :stop command. I think this may be related.

What got me headed down this train of thought is a warning on the Chef Custom Resources docs:

It is recommended to not mix platform resources with custom resources in the same lightweight provider. This helps ensure the proper processing of platform resources that have been embedded into the resource collection during the chef-client run.

I'll admit though, this warning isn't completely clear to me. It isn't clearly stated what is meant by "custom resources". I think they mean using the approach shown in the "Lightweight Providers w/Custom Ruby" section which does a shellout. If that is the case, the warning reads to me that mixing shellouts (an example of a "custom resource") and platform resources is not recommended. That is just my interpretation though ... I could be way off base. Changing the shellouts to execute resources (except the call in the docker_containers method, which is called in load_current_resource method) seems to make everything execute in the order that the code reads, at least it does for the :redeploy action.

[rdsubhas]

@jschneiderhan thanks for the explanation! let's see what the core committers have to say...

[djdefi]

We are experiencing something similar to this. Seems like the stop step just never happens on a redeploy

[rdsubhas]

In case anybody is using Jenkins, we used the Naginator plugin to automatically re-trigger failed builds. Works like a charm when run the second time...

[djdefi]

+1

[caleb]

This is my experience as well. When using :redeploy I get an error about not being able to remove an running container. As a fix I'm using action [:stop, :remove, :run].

[someara]

FIxed in master