phusion baseimage

[moltar]

Have you seen this? Why not use that as a base image for this setup?

[brianclements]

I have actually seen that image before, and there is a lot to like about the things they've put in there. The short answer I guess as to why I don't use it is because I don't want to include so many things in a default image until I know I need them. I'd rather have many small images that are layered that add only the features they need instead of one big fat one that makes many assumptions about things that are needed. And it's not even a size thing, it only weighs about 80mb more than mine, I guess it's just principal.

This image strikes me as really trying to get a container to be a full blown virtual environment for long running and multi-purpose containers. I could definitely see this image as a host for a "docker-in-docker" situation if I were to use it as my virtual "host" to make sure things are as close to normal as possible. I can even see this as a base for situations where docker is used to run remote X servers and other user/gui tools that expect such things as what they added in there. But for now, I'll leave it out until I know i need it or run up against problems in my images. And I may even just add pieces of their image as I need them into my images. To their credit too, I now know what to look out for as possible blind spots as I create my images.

Did you have a specific need that their images offered?

[moltar]

I just wanted to know if you were aware of that, before building your own base images. I like that it makes syslog and cron work. It has runit built in, supervisor is nice too, but I think runit is more stable and less memory.

Also their start up script has command flags that can disable certain behaviours.

IMO it'd be nice combine the work they did, with your ideas. Unless your container addresses all those issues that they describe already here and any other that came in via tickets and Google Group discussion. I don't remember exactly the problems they were running into, but there were some.

Anyway, I am not married to it, and I haven't really explored what you are offering. I was just throwing it out there to see if you had any thoughts.

[brianclements]

I appreciate the suggestion. I'll keep this open for further discussion and I'll think on it too. Right now I have more ideas and strategy than actual examples and testing regarding my images, so I'll let myself run into these issues as Radial matures and keep their solutions in mind. My hunch is that it's easier from a "drop-in" configuration perspective to use supervisor for most of these tasks, but I'll definitely be revisiting this at a later date.

[damm]

My view on the phusion base image is:

1. It's heavy (and every image you FROM it will be it's size and larger)
   It's no fun shipping large images :(
2. It has a /lot/ of components out of the box that not everyone will use
3. It makes every container a server
   Which goes against the notion that Docker containers should be a single process. People do this every day but it doesn't mean it's right.

As far as best practices go; it could be lighter and provide some of the components only if they are requested. There is Ubuntu Upstart which is a lot leaner and you can add the components as you need them.

Personally my containers ship all my logs to a central store; I also ship all my docker logs to the same endpoint. If your going to use a Container as a full server you really need to log what's going on because it's basically Russian roulette.

[moltar]

@damn:

1. It's only heavy the first time you have to download it, as @brianclements himself said, the difference is only 80mb. Since the baseimage is just a layer, then anything you build on top will be counted as additional size.
2. I think the only extra is SSH daemon. Which is disabled by default.
3. I don't think there is a solid defined use case for single process. I don't think the creators ever said that, unless I just missed it. I've heard this notion being thrown around, but I haven't really seen the evidence. I generally agree that that the design ultimately should be very modular, but not having syslog or cron (or even SSH sometimes), might not very useful.

[damm]

@moltar also we should keep in mind that anything from Phusion is a product; it's goal it to make users who use those products happier. That's not a bad thing; and I do think both answers are completely valid.

I wish I used single process more.

Refactoring is a pain to adopt better standards.

[brianclements]

@moltar @damm
While I've never seen any of the Docker devs specifically oppose using Docker containers as full servers, whenever I've seen explanations/powerpoints/discussions they've always been coming from the "process container" standpoint with separation of concerns as a general rule. So I'm pretty certain that was always their mindset. Now I know that eventually they plan on allowing multiple processes per container, probably by way of multiple CMD|ENTRYPOINT entries, and it remains to be seen how they do that init-wise. I think that is a problem for their Upstart image to solve.

As for Phusion's selling points, I did some homework on it and am riffing off that list:

A correct init process

Supervisor can manage the processes just fine. There shouldn't be rogue processes going on anyway that can go "zombie", so supervisor scripts for start/stop and other admin processes are the way to go. Just SSH into the container, and everything you would ever want to do to your process should be pre-scripted and then run with supervisorctl {some-task}. I've also seen mentions of future docker features that can send custom signals into your container to help with graceful shutdowns and the like. But until then, it shouldn't be too cumbersome to use SSH for delicate shutdowns.

Fixes APT incompatibilities with Docker

This fix was baked into the official Ubuntu image eventually anyway. So I'm not sure why they are taking credit for it on their page.

sudo docker run ubuntu:14.04 ls -hal /sbin/initctl should show a symlink to /bin/true

syslog-ng

syslog-ng looks great. I don't have any experience with it however, but it seems they're only using it for basic features, which of course, Supervisor is already doing by being the init system. They also seem to forget that Docker containers share the kernel with the host, so outside of your specific process's log (Supervisor), the output of syslog and dmesg aren't relevant or unique to your container (dmesg in your container is the same output as running it on the host).

logrotate

Supervisor

ssh server

got one of those too by default in my spoke container. Same hardened settings. Key-only access. disable via Supervisor .ini if you like.

cron

I can't stand using cron on real systems, and I hope I never have to use it in my containers. I think what they need it for is log-rotate tasks. To that I say "Supervisor".

runit

I thought we already handled init? Not sure if I'm comfortable using runit, I've never really heard of it before or know anything about. It In any case....Supervisor!

setuser

This is kinda cool. But again...Supervisor can run processes as any user desired.

So it really comes down to whether we like doing everything in one program, and specifying all of init/logs/start|stop/processes/user/etc.. in one configuration file (Supervisor) or having to make our own init scripts and try to catch everything in our own nonstandard scripts for our own non-standard init system... no, thank you. 1 program doing everything is better then >1 programs in this case, even at the cost of a couple megabytes of ram (Supervisor is ~13mb I think, but my image is smaller 😋 ) Besides, one of my main goals with Radial was to keep configuration simple and very manageable so that I can splice different pieces together with git and not add very many files when building apps.

Thanks anyway @moltar for the suggestion, I saw that image before and how popular it was, but didn't really dig deeper into their methods. Glad you gave me a reason to do that!

[damm]

@moltar @damm
While I've never seen any of the Docker devs specifically oppose using Docker containers as full servers, whenever I've seen explanations/powerpoints/discussions they've always been coming from the "process container" standpoint with separation of concerns as a general rule. So I'm pretty certain that was always their mindset. Now I know that eventually they plan on allowing multiple processes per container, probably by way of multiple CMD|ENTRYPOINT entries, and it remains to be seen how they do that init-wise. I think that is a problem for their Upstart image to solve.

Basically they tried nicely but there were enough (sometimes rude) users who did not take the suggestion to not use a container as a full server well. You still see people mentioning how you should use single process containers but you can't force people to get it. So they don't beat a dead horse.

Upstart

We should talk about some of the problems about Upstart before we start selling it; There was someone in #docker the other day having problems getting mysql-server to start using the upstart by default; tianon was going to look into this I believe.

I believe when init starts it assumes certain things are going; and does not kick off everything.

• I have to use a upstart file called 'start.conf' to finish starting the services that do not start properly with init by default.

start on (local-filesystems)

task

pre-script script
  /sbin/start runsvdir
  /sbin/start cron

Logging

Really we don't want any logging collection done in the container; /dev/log can drop if there is enough messages coming through so forwarding your logs to a central point (via logstash or whatever) is the way to go.

• It maybe acceptable to configure remote logging; but syslog in a container is a knee jerk reaction because you enabled other services and you want to know what they are doing.

SSH

There's people for or against this. I'm against this because unless you do this with SSH Host Certificates you cannot validate the public RSA key. Basically if your throwing up SSH you should give it a certificate; if you can't you should not /enable it/ because it's a door. If you need access to your container use nsinit or nsenter.

Cron

There are many alternatives to cron out there; truthfully vixie cron is horrible and is known to loosing the crontabs from time to time. You should just replace cron instead of using it.

Runit

Huh?

Security

You mention setuser; but really we shouldn't be giving the user root at all. These are containers; you could treat them as a hands off approach. But really allowing SSH connections to sudo is just asking for you to get owned.

[moltar]

If we were to head single process per container, then why do we even need supervisor or upstart or any of that? Just run it via CMD / ENTRYPOINT and log to stdout. That would also prevent SSH, and log collection from inside the container.

[moltar]

@brianclements: setuser especially shines when you actually build the container and need a bunch of stuff done from other user's permissions, this saves from chown'ing everything at the end.

[moltar]

@damm: what the good alternatives to cron?

P.S. Sorry for multiple messages - early - mind is all scattered.

[damm]

@moltar here are 3 alternatives; there are more out there

1. ResqueScheduler
   Often abused as cron
2. Chronos
   I doubt this will be picked often due to it's dependencies; but I could be wrong.
3. Celery
   If you use Python they support periodic tasks.

Additionally if you use a single container; there maybe a time when you need to say deploy or run tasks? Either have a image that you would --volume-from to get your content and run your deploy script and exit.

Re-using your images and containers to get to your content also provides a more secure mechanism than SSH'ing into a public port.

I avoid using SSH because I don't like people making changes to a container when it's live; often those changes don't go back to the Dockerfile in error (if your in a hurry!)

and really my point about setuser is don't. Just set your USER in your Dockerfile as a non privileged user; don't allow them to su/sudo and win.

[moltar]

@damm: these are not cron replacements!

1. Assumes one is using ruby, how about regular shell scripts? Also would have a huge dep chain to even set that up.
2. Also too big for simply cron tasks
3. Assumes you are using Python.

Sometimes you just need to run once-in-a-while maintenance tasks that cron handles best. It is not meant to be reliable like MQs. Stuff like, deleting temporary files, clearing expired sessions from the database, fetching some data from a remote location, that is not mission critical (e.g. maxmind updates), and so on.

Alternative, I guess, it would be possible to run new docker container from the host via host's cron, attaching needed data via --volume-from and then doing said maintenance.

[damm]

@moltar
Let's start with 1. A few scenarios:

• Let's say you use Rails:
  • Traditionally most of your cron jobs will be calling rails console or some other gem you have installed. Sometimes irb but basically it's some form of Rails
    • So you can likely use Resque Scheduler to schedule periodic jobs; and if you need to run them sooner you can login to resque/admin and run them.
• Let's say Python
  • Pretty much the same thing except it might be other pythonic things. One contract I worked was primarily python and did adopt Chronos. Sure you can start with crontab but you can move it out.
    • You can also use Celery if you are in a environment where that fits.

So to respond to 2. I kinda felt the same way until I watched a few guys setup everything by hand just to have Chronos. Must be something in the water.

So to quash a few assumptions; not all rails apps use redis nor resque; but a lot do. There are alternatives but overall due to some failures in regular vixie crontab you can still see people running a script that will sleep 60 and continue forever.

Lastly it's not really security that I say SSH is bad; it's more of proper mechanics. If we use volume-from and a data container our options open up. You can run a new container to get your data path to look at the logs or review data; or to change code and then restart the worker. It just feels more intentioned and proper tooling than just sshing in and running capistrano.

[moltar]

@damm: I am sensing some "Not invented here" anti pattern smell ;)

I am not using either Ruby or Python. I am using Perl. There are cron-like "solutions" in Perl as well, but I choose to avoid them in favour of old & tested cron. I have never experienced any problems aside from having to deal with the environment variables.

[damm]

@moltar the old tested and true of cron is no longer true. That was one of my original points; it's not as reliable as people assume. There are alternatives; albeit not always great but cron should not be a service in your containers as there are many services not fully setup to alert you when crons go bad.

• It's helpful to have some kind of local mail delivery available so you know when a cron job fails; or what it outputs. Without postfix or exim (or sendmail) you won't know if there's a problem and what it is.

A bad pun.

if a cron job falls(fails) in the container does anyone know?

[brianclements]

Wow, so much here, let me begin:

@damm

1. What do you think of my strategy of dumping all logs to a volume container for dumb storage? Then I can use whatever tool I like later (or even multiple tools) with --volumes-from to analyze/view/archive etc. I would think this is more flexible then baking-in a forwarder into my container no?

2. Help me understand what you mean by SSH Host Certificates? Would this be acquiring the SSH public key via a verified third party like I do in my Spoke container using Github? In light of your comments, I'm tempted to make two flavors of my spoke-base container: one for development and one for production and have SSH and no SSH accordingly.

3. Thanks for reminding me about nsinit I remember reading that post. It seems to me that this method of accessing containers can only be done on the host machine itself. Doesn't that severely limit your ability to use it across multiple hosts and even make it impossible to use in data-centers? I would think it would be even more risky to SSH into a full virtualhost that contains my docker daemon and all my containers just so I can modify only one of them instead of SSH directly into the container I need to. Am I missing something there? Besides, Jérôme suggested that these tools operate with heightened privileges anyway on the container, so it seems that it would be overkill just to get into a container and run some small maintenance task remotely. Thoughts?

4. Your point about sudo and SSH. I understand the threat SSH poses, but are you also saying that having ANY tool installed in my container who's job is to grant more privileges to users (sudo, setuser, etc) is just bad news? That makes sense of course. So ideally you would have root user, and the other system users (no sudo capability, no shells, no remote login etc) that the process itself uses, and nothing installed that can grant anyone anything correct?

re-using your images and containers to get to your content also provides a more secure mechanism than SSH'ing into a public port.

5. That's how it would occur to me to get content in as well. Lots of overlapping images and volume containers. But what about performing "actions" in those containers? What alternates to SSH would you recommend? (given what I already said about nsinit and the like in point 3).

Lastly it's not really security that I say SSH is bad; it's more of proper mechanics. If we use volume-from and a data container our options open up. You can run a new container to get your data path to look at the logs or review data; or to change code and then restart the worker.

6. I like this for logs, obviously, that is my strategy for Radial. Regarding changing code and restarting the worker however, you can't --volumes-from a container and then act upon processes in that container. I guess you could stop|start the container, which re-inits the process with "supposedly" new code or modification, but I think I'd rather rebuild the container altogether so that every change is represented as a new image/build/Dockerfile etc. Using Hub containers (volume containers) for configuration/process src and then a Spoke container (full os + app) for the process helps with that hack/test cycle I think. I want at any time to be able to delete all my images and containers and not loose anything src/config wise. That is, all changes happen outside containers in source control. Docker is part of my "test" step, not my "hacking" step. At least for now. Could be cool to make a "hacking" container with all my tmux/vim/zsh that I can docker pull from anywhere and be all ready to go!

---

@moltar

If we were to head single process per container, then why do we even need supervisor or upstart or any of that? Just run it via CMD / ENTRYPOINT and log to stdout. That would also prevent SSH, and log collection from inside the container.

In my experience, extracting logs via the docker logs output is a mess. Especially with long running processes and verbose logs. There are also memory caps and dropping of logs bugs that people have mentioned. If the logs are of any importance, then you would try a different method of extraction.

So that alone warrants Supervisor's use or at least some other method of log processing. But also I would imagine that not all containers are simply "hands-off" once run. Databases need maintenance and sometimes processes have multiple steps in order to start|stop|reload in subtle ways that a simple docker start|stop|restart|stop just don't be able to handle. For those situations, a script or again, Supervisor could perform such a task. And getting into the container to do such a thing would need SSH or some other method that I'm not aware of (that is still considered "safe").

setuser especially shines when you actually build the container and need a bunch of stuff done from other user's permissions, this saves from chown'ing everything at the end.

I'm not sure what setuser offers that I can't do using normal bash and Dockerfile features. In the docker file, I can actually switch users mid-file with a USER step. So that can handle some things, but also, you are root as default in the Dockerfile, I can do everything I need as root and then run my final process using the USER declaration or via Supervisor which has a per-process "user" option.

Sometimes you just need to run once-in-a-while maintenance tasks that cron handles best. It is not meant to be reliable like MQs. Stuff like, deleting temporary files, clearing expired sessions from the database, fetching some data from a remote location, that is not mission critical (e.g. maxmind updates), and so on.
Alternative, I guess, it would be possible to run new docker container from the host via host's cron, attaching needed data via --volume-from and then doing said maintenance.

I'm currently asking this question on the docker-user group because I'm not sure about how to go about that. I don't have much experience with databases so I don't know which of those maintenance tasks come up frequently and how to go about them (for now). I really like the idea of having my database data in a shared volume upon which a specialized "maintenance container" can be run whenever with --volumes-from and do all those tasks as needed, then delete itself. But I also can easily see those tasks being triggered manually, or timed automatically in a loop script, or caught with log stream events within Supervisor. Which does so many things for me in my containers it seems; one-stop shopping.

[damm]

@moltar

1. I've been debating LoggerFS.
   Currently I solve this by using beaver in the container to ship any logs generated by the application into Logstash. I have beaver running on the host to consume the docker logs and ship those up into Logstash as well.

2. Similar to SSL you can generate a ca and then have that ca sign the host key generated upon bootup and configure sshd to use it. OpenSSH certificates has more on it. Then you update ~/.ssh/known_hosts and specify the IP range (hostnames) with the public signature and connect. You never get asked again to verify the SSH key.

   This is very useful as whenever you reinstall openssh it typically generates brand new Host keys. Easily you could have 40 containers sharing the same host key (not good) or 40 different hostkeys and you don't know what to trust; so you always trust it. (This is Wrong but 99% of the world does this)

3. That depends how you script it; if you use fabric you can inspect your container get the pid; enter the namespace and run your commands. Basically I view SSH as a door into your system; if you keep credentials around and other goodies you will make an attacker very happy someday. Which is what we're hoping to avoid.

4. Right; if you don't let them have sudo; don't let them obtain root over SSH the only way you can obtain root is via nsinit then. Basically you have obscured the way to own it providing yourself a buffer.

5. See answer Bash/ash Auditing #3 and Broken links to viewdocs.io in README.md files #4; Using something like Fabric (for example, capistrano could fit here too) to SSH into the Host; grab the details you want and run the tasks you need. You can have it nsenter and then run arbitrary commands to deploy; restart or scripts.

6. Yeah it basically makes everything in Docker which means your hacking and usage should be straight forward. I bet it could get hard to manage but there are tools now like etcd and likely newer tools as time progresses to help us at least describe where everything is. Having 40 apache containers sounds like a pain; but having 1 apache container link to 40 containers sounds worse.

[brianclements]

@damm I really appreciate your input to this conversation. I am really excited to learn more about the OpenSSH Certificates, that absolutely feels like THE solution to the docker ssh/key problem in containers; I wasn't aware of that before and want to put that in my images (the pending "dev" flavors of course, not production in light of your recommendation). I really like your logging suggestions too. But all I can find is artifacts of LoggerFS on the web, which is the most recent source that you use?

[damm]

@brianclements LoggerFS was announced the other week and based on the HN comments and the lack of being able to find actual code; might not be out yet. I had bookmarked the page and was going to evaluate it next week, good thing I looked now.

SSH Host keys normally don't change on a system; if you use EC2 you likely accept the default unique host key per instance; it just makes it difficult to trust without a way to display (in a way you trust) what the public key should be.

Having an SSH Certificate is fairly easy you would just modify ~/.ssh/known_hosts

@cert-authority iprange1,hostname2,iprange3 ssh-rsa cert

Yes you can specify multiple IP ranges or Hostnames this way.