New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hooks for more customization #6982

Open
tiborvass opened this Issue Jul 11, 2014 · 27 comments

Comments

Projects
None yet
@tiborvass
Copy link
Collaborator

tiborvass commented Jul 11, 2014

There are a few issues that ask for specific hooks, but none of them have apparently been fully addressed. The ones that are closed were promised a "generic" hooks mechanism for all commands/events.

A few usecases from reading the issues:

  • shutdown hook (EXITPOINT ?) for gracefulness (#2100)
  • start hook (#252 is a bit old, and #3317)

From kubernetes/kubernetes#140 (comment):

[...] we felt there were two cases we wanted to handle:

  • hooks that either need the container context (and as such executing outside the process namespace would be pointless), or if interrupted by container shutdown would not be internally inconsistent. Pre-termination is a good example
  • hooks that should be outside of a container, because they need to continue to run even if a container fails. Deploy across multiple containers is a good example, or post-termination.

Please feel free to comment with more.

It makes sense to offer hooks for greater customization, however, this might affect the current way of doing things (see the exec discussion in #3317 (comment))

For solutions, there was a mention of "predefined paths" for executables.

Some hooks (start and shutdown) would be more useful if they were executed synchronously (unlike how docker events fires currently).

We need to define what hooks should be implemented in docker and what should be done externally via docker events.

@tiborvass

This comment has been minimized.

Copy link
Collaborator

tiborvass commented Jul 16, 2014

For reference: #4550

@foliea

This comment has been minimized.

Copy link
Contributor

foliea commented Jul 18, 2014

Perhaps that some docker options could also use the hook system ? I am thinking of docker run --rm. It would solve a few issues with that one (eg. conflict with -d) if we could use a on_exit hook.

@mrunalp

This comment has been minimized.

Copy link
Contributor

mrunalp commented Aug 7, 2014

Also, a hook to execute arbitrary code before we exec user command would be nice to allow customization in the namespaces.

@phemmer

This comment has been minimized.

Copy link
Contributor

phemmer commented Aug 9, 2014

I keep running into things where I would find a pre-start hook very useful, so I'll throw in some of my thoughts on this.

I can see the possibility for several hooks

  • pre-start - namespace is completely configured but the process has not been launched
  • post-start - process has been launched
  • pre-stop - just before sending a SIGTERM
  • pre-kill - just before sending a SIGKILL
  • post-stop - all processes have exited, but the namespaces & mounts still exist
  • pre-pause
  • post-pause
  • pre-unpause
  • post-upause

Now whether all these actually have a purpose or not is another matter. Personally I would find the pre-start and post-stop to be the most useful. My frequent use case is setting firewall rules for the container, and then cleaning them up.

I think each hook should not execute in the context of the container, but in the host context. If the hook needs container context, it can nsenter or nsinit.
The hook would be a binary or script, and it would receive a single argument of the container ID. If it needs any additional information, it should use the CLI and/or remote API to query the information it needs.

The hooks should be executed synchronously.
I also think the pre-* hooks should be able to abort an operation. For example, if the pre-start hook returns a non-0 exit status, the container should be marked as failed-to-start.

I see 2 possibilities for what level the hooks should be configured at, daemon level, so the hooks trigger on every container, and at the run level, where the hooks are passed when starting a container. I think both scenarios have valid use cases, and so both should probably be supported. Run level hooks would trigger after the daemon level hooks.

As for how to pass the hooks in. For the daemon, it would be passed as an argument with the path to the binary/script.
For run hooks, I see 2 possibilities:

  1. Use multipart/related, and send each hook binary/script as additional parts after the JSON body.
  2. Send the binary/script as a JSON attribute. This option has a few issues though as sending binary through JSON is rather cumbersome. And large JSON attributes just don't feel right.

 

I think hooks have a lot of potential to prevent feature bloat. By providing the capability for users to easily implement whatever they want, we can keep docker from getting a plethora of features which few use, and are difficult to maintain.

@grepory

This comment has been minimized.

Copy link

grepory commented Sep 3, 2014

I detailed my use cases in #7847 -- but I think a generalized startup (that is to say post-startup) hook and a shutdown hook (that executes regardless of container exit status) would be nice. Use cases I can come up with quickly:

Service registration/discovery
Orchestration
Easier management of boot2docker volumes

@bgrant0607

This comment has been minimized.

Copy link

bgrant0607 commented Oct 8, 2014

Hooks are critical to both implementation of higher-level management/orchestration systems, and (perhaps counterintuitively) portability of images across different environments, since they can provide whatever glue is necessary to adapt the container to its environment.

The hooks proposed by @phemmer SGTM. IMO, pre-start, pre-stop, and post-stop would be the most useful now (pre-pause and post-unpause would be useful eventually, too, once migration is fully supported).

In order to implement pre-start and post-stop, we need to decouple the namespace lifetime from the lifetime of the main process. That would be extremely useful to higher-level management systems, in general.

I also agree with @phemmer that the hooks should execute in the host context.

Users will often want their hooks to execute within the container context. The case for pre-start executing within the container context is to initialize the filesystem, and the case for post-stop is extracting data from the filesystem.

However, what's attractive about Docker executing the hooks in the host context is that would be provide a hook for the management layer -- Kubelet in our case. We need this, for example, in order to use Docker container restarts and still provide Kubernetes-specific functionality upon each restart.

/cc @vishh @erictune

@erictune

This comment has been minimized.

Copy link

erictune commented Oct 8, 2014

As @bgrant0607 says above, both a management layer and a user (person who wants to run a container) both may want to run a hook point. How do two different sources provide hooks, and what is their order or precedence? Can there be mulitple instances of a hook or does the management layer wrap the user-provided hook command in its own command?

Also, a management layer may want to treat a user-specified hook as untrusted, and force it to run in a container (for resource isolation, protecting host filesystem). How should a management layer do this? Suppose hooks always run in host context, and the hook can optionally nsenter the container context (as suggested above). How should a management layer ensure that an arbitrary hook runs in container context? Parsing the user-provided hook is an unattractive option. Running nsenter unilaterally seems like it won't work.

Possible solution:

  • Define separate host-context and container-context versions of each type of hook.
  • the management layer provides the host-context hooks
  • the user (person who wants to run this container) provides the container-context hooks.

Advantages:

  • user provided hooks always run in container context, protecting the host filesystem and maintaining resource isolation.
  • no conflict between two systems trying to define hooks.
  • user provided hooks could be possibly be defined in the dockerfile, if they are something that applies to all uses of the container.
  • Management hooks can be provided via the docker command line.

Can come up with a concrete example if that helps.

@phemmer

This comment has been minimized.

Copy link
Contributor

phemmer commented Oct 8, 2014

@erictune I'm totally not following what you're getting at.

There's only 2 concievable ways you could set hooks in docker. At the daemon level (docker -d), and at the runtime level (docker run, docker start, docker build). I'm assuming by "management layer" you mean daemon level, and by "user specified hook" you mean runtime level. I don't get why the 2 would interact with each other at all.
Every hook system I've ever seen operates in layers. Layer 1 gets data, can manipulate it however it wants, and then the data gets passed to layer 2. Layer 1 has no clue that layer 2 even exists and vice versa. Each layer assumes that the next thing to receive the data is the core application itself.

@erictune

This comment has been minimized.

Copy link

erictune commented Oct 8, 2014

I meant something different, I think. By "management layer", I mean an orchestration system that supports docker (such as Kubernetes or Mesos or a number of others). By "user", I mean a person that wants to run a container using an orchestration system.

The orchestration system might want to do some extra setup before running any container. It could use a "pre-start" hook.

The user may also want to do some setup before running her particular container. She could also use a "pre-start" hook.

But, the user should not need to see or understand the orchestration system's use of the hook, and the orchestration system should not need to grok the user's use of the hook.

@progrium

This comment has been minimized.

Copy link
Contributor

progrium commented Oct 17, 2014

This is weird because this is what the event stream is for. You can (and I'm about to) expose that event stream via a nice hook system as a container that doesn't need to be in docker core...

@jessfraz

This comment has been minimized.

Copy link
Contributor

jessfraz commented Oct 17, 2014

@progrium are you bind mounting the socket in the container? I would be interested in seeing it :)

@robhaswell

This comment has been minimized.

Copy link

robhaswell commented Oct 17, 2014

@progrium the author has acknowledged docker events:

Some hooks (start and shutdown) would be more useful if they were executed synchronously (unlike how docker events fires currently).

There is also discussion here about the requirement for pre-start and pre-stop hooks.

@phemmer

This comment has been minimized.

Copy link
Contributor

phemmer commented Oct 17, 2014

@progrium An event stream isn't a hook. A hook sits in between things. You can use it to capture data yes, but you can also use it to manipulate the environment. If you want to prevent docker from starting a container, your hook would return an error code. If you want to change a network setting before any processes are launched, your hook would make the change before continuing execution of the normal flow.

I've already coded up daemon level hooks. It was fairly simple. I have to add run-level (container) hooks. I know how I'm going to do it, but just havent yet. Will probably get it done this weekend.

@progrium

This comment has been minimized.

Copy link
Contributor

progrium commented Oct 17, 2014

Yes, you're both right. I missed that mention of events. If I know how Solomon thinks (and how I'd want it either way), this would not be directly implemented as shell hooks, but using some form of libchan/rpc accessible from containers that could then optionally expose them as hook scripts. Much like the container I'm working on.

And @jfrazelle, yes. I'll show you soon. Solomon wanted me to share something(s) for Docker Hack Day and I was thinking this.

@bgrant0607

This comment has been minimized.

Copy link

bgrant0607 commented Oct 17, 2014

@progrium Requiring participation of the container, especially in a Docker-specific fashion, largely defeats the purpose of this feature.

@progrium

This comment has been minimized.

Copy link
Contributor

progrium commented Oct 17, 2014

Hmm, alright. I think one of us isn't following something. Can you
elaborate?

On Fri, Oct 17, 2014 at 5:15 PM, bgrant0607 notifications@github.com
wrote:

@progrium https://github.com/progrium Requiring participation of the
container, especially in a Docker-specific fashion, largely defeats the
purpose of this feature.


Reply to this email directly or view it on GitHub
#6982 (comment).

Jeff Lindsay
http://progrium.com

@tcurdt

This comment has been minimized.

Copy link

tcurdt commented Nov 19, 2014

So I take it from this issue still being open, hooks/notifications of container's started or stopped are at this stage just merely wishful thinking? Right now I would have to listen to docker events manually (with e.g. https://github.com/deoxxa/docker-events)

Is that correct?

@progrium

This comment has been minimized.

Copy link
Contributor

progrium commented Nov 19, 2014

You have the events, post-event hooks are quite possible. In fact, here they are wrapped up nicely so you can create individual plugins:
https://vimeo.com/110835013
https://github.com/progrium/docker-plugins

This was made with a generic event-to-hook tool below, but I think if you're doing more than a one-off hook, docker-plugins might be a better way to go.
https://github.com/progrium/dockerhook

@tcurdt

This comment has been minimized.

Copy link

tcurdt commented Nov 19, 2014

awesome! thanks, @progrium

@analytically

This comment has been minimized.

Copy link

analytically commented Dec 11, 2014

👍

@tcurdt

This comment has been minimized.

Copy link

tcurdt commented Dec 11, 2014

btw: while dockerhook and friends are great I ended up rolling my own slightly more specialised version. using it for managing nginx upstreams and reloads.

https://github.com/tcurdt/docker-upstream
https://registry.hub.docker.com/u/tcurdt/upstream/

seems to work fine - but still needs documentation.

@lukemarsden

This comment has been minimized.

Copy link
Contributor

lukemarsden commented Jan 29, 2015

Just wanted to mention that we've released https://github.com/clusterhq/powerstrip to help solve the problem of adding blocking pre- and post- hooks to Docker.

The Powerstrip project is explicitly intended to be a way of prototyping Docker extensions. Please try it out and send feedback as github issues or PRs!

FYI @tcurdt @analytically @progrium @bgrant0607 @robhaswell @phemmer @jfrazelle @erictune @grepory @mrunalp @folieadrien @tiborvass

@peter-leonov

This comment has been minimized.

Copy link

peter-leonov commented Jun 17, 2015

👍

@luebken

This comment has been minimized.

Copy link

luebken commented Feb 22, 2016

Any news on this? I find this topic pretty interesting.

@kirkstrobeck

This comment has been minimized.

Copy link

kirkstrobeck commented Apr 20, 2016

WANT

@jizhilong

This comment has been minimized.

Copy link
Contributor

jizhilong commented May 4, 2017

I also found a need for pre-start hook mechanism when trying to attach openstack-neutron ports to docker containers and do some customized network settings which is not modeld in libnetwork.
My final solution is to build such a 'prestart' hook based on docker's primitive mechanism, with no modification to docker-engine.
For those interested, this solution is now open sourced at docker-wait.

@pablo-ruth

This comment has been minimized.

Copy link

pablo-ruth commented Nov 3, 2017

If you need pre-start and post-stop hooks, you can use go-init. it's a very lightweight init system for Docker (< 500kb) with support for lifecycle hooks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment