shut down an idle pilot

[dsschult]

Currently STARTD_NOCLAIM_SHUTDOWN is fixed at 30 minutes, and supervisord restarts condor repeatedly, never exiting the container.

This PR allows the idle time to be configured, and shuts down the container if condor_master exits normally.

[brianhlin]

@dsschult could you rebase this on master? We've got some basic CI tests now so I'd like to run this change through them.

[dsschult]

Rebased. And now I see tests running.

[brianhlin]

@matyasselmeci @rynge what do you think about this? This would change the behavior of the containers quite a bit for our other users so we'd have to communicate it to them

[dsschult]

The main behavior change is autorestart=unexpected. Not sure if that could be made an option, so default behavior is unchanged. Might be able to do that in the entrypoint script, before supervisord starts up.

[brianhlin]

At this point users just start up the container when they want work from the access points and whether or not they get it is our problem. With this change, sites will have to monitor whether or not their container is running to account for transient lulls in usage.

[matyasselmeci]

This seems similar to https://opensciencegrid.atlassian.net/browse/SOFTWARE-4608.

[brianhlin]

That's in a similar vein but I think it means we only want autorestart on successful exit: the container should exit if the startd isn't working so that the admin can just check if their containers are running or not to determine if they're offering their resources to the OSPool.

[matyasselmeci]

Wouldn't that delete the log files of that pilot, making it harder to troubleshoot?

[dsschult]

I added a change to make it configurable. Then you can decide which policy you like better at runtime.

It now takes SUPERVISORD_RESTART_POLICY=unexpected as an env variable (and for some reason that didn't make it into the commit message).

Could potentially name it better if you can think of something.

[brianhlin]

I added a change to make it configurable. Then you can decide which policy you like better at runtime.

All disagreements can be satisfied by more configuration :). This works for me!

Wouldn't that delete the log files of that pilot, making it harder to troubleshoot?

We do tell folks to mount something to /pilot so the logs should still be around. I think it's probably better than the alternative where you need to be an expert in condor and this container to realize that you're running the container but it's not advertising back to the OSPool.