dokku ps

[slaskis]

dokku-ps

A process management plugin for dokku.

Modeled after the process/dyno management of heroku.

Written to solve these problems:

1. Restarting crashed containers.

   Because sometimes the process exits because of something like a memory leak and, until we have it fixed, we just want it to keep going.

   And the idea is to use upstart to handle the containers, as described in dockers host integration docs.

2. Being able to start up non web processes.

Bonus problems to solve:

1. Monitor & notify.

   I'm guessing since upstart is monitoring the respawning of containers it should be able to notify me by email, irc or something even cooler using some custom events.

2. Process scaling.

   If we are able to add each process as a backend to the nginx config we should easily be able to load balance the requests over multiple processes. This PR seems like it would help a lot as we would only have to generate $APP_ROOT/nginx.conf with the processes defined in $APP_ROOT/PS. Something like:

    CONF="$APP_ROOT/nginx.conf"
    echo "upstream $APP {" > $CONF
    grep "^web=" $APP_ROOT/PS | while read line; do
      NUM=${line#*=}
      for ((i=1; i<=$NUM; i++)); do
        PORT=$(docker port $APP.web.$i 5000 | sed 's/0.0.0.0://')
        echo "server 127.0.0.1:$PORT;" >> $CONF
      done
    done
    echo "}" >> $CONF

Status

• Install plugin
• Deploy app
• Restart a process dokku ps:restart APP web
• Restart app dokku ps:restart APP
• Scale processes dokku ps:scale APP web=4 worker=1
• Stop a process (such as "web") dokku ps:stop APP web
• Load balancing (of "web" processes only?)
• Logs (is it possible to cat multiple streams into one?)
• Cleanup and format the output/noise
• Tests

Thoughts?

[progrium]

Is it necessary to add a mandatory 5 seconds to all deploys?

[slaskis]

Woops, that was left there from when I thought docker needed some time to give me a port.

[progrium]

This is great! It may take some time to review. Maybe we can even simplify it a bit.

[progrium]

We should figure out a way to make it so dokku doesn't depend on a plugin.

[slaskis]

I know it's a bit of a specific hack but would it suffice to wrap the current code in an if?

if [[ ! -f "$APP_PATH/PS" ]]; then 
  # current kill / start code here
fi

[progrium]

I really don't think I'd +1 this without a real solution to this.

On Thu, Nov 7, 2013 at 12:37 PM, Robert Sköld notifications@github.comwrote:

In dokku:

@@ -47,18 +47,17 @@ case "$1" in
APP="$2"; IMAGE="$3"
pluginhook pre-deploy $APP $IMAGE

• kill the app when running

• if [[ -f "$DOKKU_ROOT/$APP/PORT" ]]; then

•  oldid=$(< "$DOKKU_ROOT/$APP/CONTAINER")
  

•  docker kill $oldid > /dev/null
  

• fi
• dokku ps:deploy $APP

I know it's a bit of a specific hack but would it suffice to wrap the
current code in an if?

if [[ ! -f "$APP_PATH/PS" ]]; then

current kill / start code here

fi

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/298/files#r7509242
.

Jeff Lindsay
http://progrium.com

[slaskis]

What do you think of this other PR: #296, maybe it could provide a bit more flexible solution?

But, if I understood the current implementation of pluginhook correctly, it would still do multiple deploys which seems a bit wasteful but maybe a flag to pluginhook which makes it only execute the last hook? Or is there a case when multiple deploy scripts would be desirable?

[shalakhin]

It would be awesome to have this in dokku. Thank you @slaskis for the contribution!

[AlJohri]

@slaskis Does your syslog plugin https://github.com/slaskis/dokku/tree/wip/logger-0.2/plugins/syslog already account for Logs (is it possible to cat multiple streams into one?).

Thanks for making this! It would be awesome to have this in dokku.

[slaskis]

@AlJohri haha, yeah it's a pleasant side effect of using syslog. Log rotation is another one. And the possibility to send logs to a central place (like papertrail) is yet another one...

[AlJohri]

That's awesome! @slaskis

What's left to be done then in terms of merging this much awaited PR? :)

[slaskis]

@AlJohri I wrote about some issues I have still in this comment: #386 (comment).

(copied it to here, since it's relevant to this issue)

Zero downtime is an issue with ps that I didn't get around to solve.

Mainly because the way I used named docker containers, which was really nice and convenient, but it wouldn't allow it to have two containers with the same name (and renaming them was not possible). So to get it to work I'm thinking that it would have to move from named docker containers and store the container id in a file in something like $APP/ps/$name which simply reads the container name.

But I have been running ps in production for a few months now and I have found some other kinks that needs to be fixed as well. And I'm curious to try it with newer docker too, might solve a few instabilities.

[slaskis]

But the main issue with getting this merged is that I haven't spent any time on it since like december. I've been busy with unrelated projects which take up all my time.

[josegonzalez]

@slaskis any chance you could write a note on what other "kinks" you found with this plugin? I can get it merged if I know what issues you've found :)

[slaskis]

@josegonzalez unfortunately I can't remember what they were anymore :(

[josegonzalez]

Are you still using dokku in prod? Any chance you have any other changes?

[slaskis]

@josegonzalez actually they are still alive and kicking but I haven't touched them for almost exactly a year (the campaign for which the servers were created for was just until christmas).

There are some changes implemented for a syslog plugin in a logger branch which is being used in those too. But that's based on whatever docker version that was released a year ago (definitely pre-1.0). So I'm sure the logging and other stuff could be handled better now.

[josegonzalez]

I think we're not going to implement this as is, and instead provide a pluginhook. For instance, I can foresee a plugin that scales AWS instances with dokku containers, or one that modifies the number of procs running inside of a supervisor controlled container.

If anyone has any comments on that, feel free to pile them on, but doing anything more inside of dokku feels a bit weird (because there are so many corner cases).

[josegonzalez]

We have an official ps plugin as of #897, and the items not implemented here are I think out of scope. Feel free to open a new PR to implement some of these in the new plugin as you see fit.