Urgent: add a FAQ section + disclaimer about docker-related issues

[knz]

Right now the top 1 question that I see in our Gitter channel is "I started my cockroach node in docker and I can't access the admin UI" (about once a day).

Meanwhile it is also true that we at Cockroach Labs both avoid using docker for production deployments (we barely tolerate it for testing), despise it as being poorly designed and implemented and generally have little in-house knowledge about how to troubleshoot these issues.

Therefore we absolutely need a big warning sign in our docs that we provide docker documentation for convenience only, and that we discourage users from using Docker if they expect support for us. What I would like to see happen is this:

1. add the above on the various "how to do this or that with docker"

2. add a new FAQ page with the following:

   "Question: I have set up something with Docker, but now it doesn't work (can't access admin UI, etc.) What to do?"

   "Answer: It is likely that the Cockroach node works just fine, and that a network configuration issue on your side is preventing proper function. We acknowledge that Docker makes it unduly complicated to set up a network service successfully, that subtle differences exist between Docker versions which makes it complicated to replicate successful configurations reliably, and we deplore that the tool provides so little means to troubleshoot these issues. We do not recommend using Docker directly for any serious deployment of CockroachDB. Instead we advise to use technology X or Y. Unfortunately at this time Cockroach Labs is unable to provide technical support to troubleshoot Docker-related issues."

(then pick X and Y, probably Kubernetes will appear there.)

3. link the FAQ page from the disclaimers from point 1.

[bdarnell]

I wouldn't go as far as saying that you can't get support from us if you use docker (if we won't support it we shouldn't publish or promote the images at all), but we definitely need to make it clear that running a stateful distributed system in docker is harder than most uses of docker and that unless you're already an expert in docker you should probably start with other deployment options. We should also discourage raw use of docker in favor of higher-level orchestration schemes like kubernetes/mesos.

I also read a good blog post recently (called something like "running docker containers without the nonsense", but I can't find it now) which recommended a set of command line flags that make containers behave more like regular processes. In particular the --network=host flag gives you less surprising network behavior. We may want to start recommending some of these options.

[jseldess]

@bdarnell, it be great if you could track down that blog post.

@bdarnell, @knz: I'm a little confused about how to proceed here. We have two Docker-specific pages, one for running a local cluster for testing only (straight-up Docker) and the other for running a real cluster across multiple hosts (Docker Swarm).

I would like to take a look back through the Gitter history to get a concrete sense of how often Docker issues are coming up and what the patterns are. I suspect that, in at least some cases, users just aren't following the instructions in the docs, which would be a problem in itself. In any case, I think it'd help to review all questions around Docker. @sploiselle, can we query your Gitter monitoring app to pull out all questions around Docker, or is that not yet possible?

I'm not arguing that we shouldn't include a disclaimer, though. If we do suggest starting with other options, does that mean: in the case of local testing, run from a binary, and in the case of real deployment, run from binary or using Kubernetes (but not Docker Swarm)?

[sploiselle]

Emailing @jseldess the results of the Gitter query.

[knz]

So the issue is simple to understand once you step in the users' shoes. Basically people, albeit willing to learn, always start as total beginners. They know they need something something database and something something container. What happens is the following:

1. they install their system's idea of what the basic Docker package looks like. They do not know at this point that the container technology is only 20% of what they need for their container needs.
2. they briefly glance at the Docker web page / README file which says "Whatever you need to do, just run docker run!"
3. they download the cockroachdb binary, and see on our web page "CockroachDB is easy! nothing to set up! Binary is standalone!" So they just skip reading the docs entirely (it's easy after all)
4. they run docker run cockroach. After all, docker and cockroachdb are easy right? easy+easy = easy?
5. It doesn't work!
6. Our web page says "come ask us questions on Gitter!" Then they do so.

The crux of the problem here is that we're not doing a good job at explaining or underlining to people what are their unknown unknowns. Things they should care about but they do not know they do not know.

In particular:

• A CockroachDB cluster, if not explicitly configured with either --insecure or the full certificate shebang, only listens on localhost, which will make it deaf and mute inside Docker.
• Docker networking is hard. Its default configuration is useless and harmful.
• Docker is not made for stateful services like databases. To make it good needs extra effort, and explanations on how to do so are nowhere to be found in beginner docs.
• They don't usually know what coordination entails. Yet they do need coordination technology next to the basic container stuff.

So what's in there for us to do?

1. Despite whatever marketing/PR reasons we may have, I believe we should not explain how to set up a basic Docker deployment, even for testing, at all. Our "basic doc" should explain K8s or perhaps docker swarm right away. If a user has a legitimate use for a testing cluster in docker, they are not beginners and can be redirected to "advanced", "dev-only" docs which we can provide a bit more hidden in our docs site.

2. We should present how to configure a secure cluster first. And just under this explain to people that any network service can listen on either one, all or some network addresses, and that our defaults are not necessarily what they want/expect. Usually people who are reading beginner docs do not know this and they absolutely must be taught ASAP lest they will do something stupid with their configuration, like configuring an insecure cluster that listens on a public internet address.

3. If we believe that a turn-key solution is key to adoption by casual/drive-by visitors, we should provide either an online sandbox (my preference) or a downloadable VM they can run with Virtual Box or similar. Setting up a properly working, no-UX-surprise, no-self-shooting-in-the-foot distributed network service is really out of practical reach by real beginners.

[bdarnell]

Found it: https://jvns.ca/blog/2016/11/16/ideas-about-how-to-use-aws/ (I vaguely remembered it was on Julia Evans' blog but skipped right past this post several times because the title said AWS instead of docker/containers) and https://blog.skyliner.io/the-happy-genius-of-my-household-2f76efba535a "Use containers. Not too much. Mostly for packaging"

On re-reading it, while there are some good philosophical points about the use of containers, I think the only command-line flag we may want to consider recommending from these posts is --net=host. That would simplify things a bit, although since we already recommend all the docker network create steps it may not really help - the problem is users getting stuck when they try to go their own way without the docs, so we may not be able to document our way out of the problem.

[knz]

Another instance of this problem: cockroachdb/cockroach#12593

[jseldess]

Thanks, @knz. I'll get on this first thing once I'm back from vacation.

[BramGruneir]

Ok, so I found what I hope are only errors with our windows docker setup:

1. On https://www.cockroachlabs.com/docs/install-cockroachdb.html - Windows
   We should tell people to open up powershell and the prompt should look like
   PS C:\Users\Username> not $

2. On the same page, we need to specify that they will need to enable sharing of their drives with the docker containers. To do that, follow the directions here:
   https://docs.docker.com/docker-for-windows/#/shared-drives

3. On https://www.cockroachlabs.com/docs/start-a-local-cluster-in-docker.html we are going to require some specific examples for windows instead of linux/mac since PWD returns the wrong format for docker.

-v "${PWD}/cockroach-data/roach1:/cockroach/cockroach-data"

on linux/mac becomes
-v "/Users/username/cockroach-data/roach1:/cockroach/cockroach-data"

but on windows becomes
-v "C:\Users\username/cockroach-data/roach1:/cockroach/cockroach-data"

So instead, for windows, the absolute path must be specified like so
-v "//c/Users/username/cockroach-data/roach1:/cockroach/cockroach-data"

See https://docs.docker.com/engine/tutorials/dockervolumes/#/mount-a-host-directory-as-a-data-volume for more info

All three nodes commands will require this update.