One service is conflicting, but not running

[AndrewVos]

I have three services, portiere.{1,2,3}.service, and three ec2 instances running. Each service X-Conflicts with portiere*.

I have two of these running at the moment, but the third never finds a place to run because for some reason fleet thinks that portiere is being run in all three places.

This conflicts with what list-units thinks:

UNIT                    LOAD    ACTIVE  SUB     DESC            MACHINE
portiere.1.service      loaded  active  running portiere        de2eacc3.../10.185.208.27
portiere.2.service      -       -       -       portiere        -
portiere.3.service      loaded  active  running portiere        b6bfeed6.../10.165.32.45

This is all I get in systemctl status fleet:
Mar 06 03:00:05 ip-10-185-208-27 fleet[469]: I0306 03:00:05.759903 00469 engine.go:78] Published JobOffer(portiere.3.service)

Any idea how I can debug this?

[AndrewVos]

Also useful information, if I stop portiere.3.service then portiere.2.service will launch. From then on portiere.3.service will not launch with any amount of destroy/submit/start. Pretty sure this means the job offer is just not able to be accepted.

[AndrewVos]

curl 'http://127.0.0.1:4001/v2/keys/_coreos.com/fleet/?recursive=true'

https://gist.github.com/AndrewVos/42b984b11c63b524d4c9

[bcwaldon]

I bet one of your fleet agents thinks it has an outstanding bid for portiere.1.service and will not submit conflicting bids. Restarting fleet on the EC2 instance that refuses to run anything should fix it. I've added a feature to fleet that allows you to send that SIGUSR1 signal to a fleet daemon and cause it to dump the current AgentState to the logs. That will help us debug this kind of stuff in the future. Is there anything else we can do wit this specific issue.