Remote SSH into containers

[markfinger]

Docker cloud has a suggested approach for enabling remote SSH access to docker containers: https://docs.docker.com/docker-cloud/infrastructure/ssh-into-a-node/

SSH support for containers would help to alleviate a lot of pain points.

Understandably, the nature of containers will still preclude this from being a silver bullet. For example, I'd imagine that we'd need to use docker cloud's UI to trigger a db dump, write it to the container's FS, then access the SSH available container and pull the dump from that.

Given that the suggested conf relies on a white-listed set of SSH keys, which will prove ungainly for any project with multiple devs, I wonder if it's possible to tunnel through another IC server?

cc: @mrmachine @jmurty

[markfinger]

My above example relies on the assumption that we wont have pg_dump binaries available on the SSH'ble container, but we could have a shared file system with one that does.

[mrmachine]

@markfinger @jmurty that is SSH access to the node, not the container, as root. From there you need to Docker exec into a running container. I'm hesitant to encourage this in any but the most important scenarios because I like the fact that a node is an ephemeral black box that we deploy and scale and destroy as needed and otherwise don't need to think of, and I think we should only rarely need shell access to a container (and these needs should reduce over time as well). For obtaining backups I'd rather automate the creation of regular backups to a more accessible location like s3 (not done) or allow direct access to the database from a whitelist of IP addresses (done). For remote access from a dynamic IP you can tunnel through one of our servers with a static IP (not done).

[markfinger]

@mrmachine

node is an ephemeral black box that we deploy and scale and destroy as needed and otherwise don't need to think of

Is definitely what we should be aiming for, but as 95% of our sites will never have scalability concerns, this feels like optimising for the edge-case and introducing road blocks to every-day tasks (db dumps, introspecting file systems, immediate patches for deployment issues, etc).

From a philosophical perspective, this seems like similar contentions to many of my original concerns about docker's trade-offs: opaque, abstracted systems and difficulty accomplishing what previously took seconds. While I appreciate and value the drive for clean systems, improved shell access will massively improve the developer experience for the usual maintenance tasks.

---

Ultimately, you're the gatekeeper on this, Tai. So, I'm happy to work out a compromise, as the current situation is not great.

Out of your suggestions and/or others, what is the low-hanging fruit, and what can be actioned by someone other than yourself?

[mrmachine]

@markfinger You're right, many sites don't need to scale. But the nature of EC2 instances in general, and especially for nodes managed by Docker Cloud, is that they are still ephemeral and might be destroyed not only by scaling operations.

For example, we redeploy to a new differently sized node cluster. Or Docker Cloud releases a new version of the node image and we need to redeploy.

What I'd like to avoid is:

1. Ever having any manual configuration applied to a node, that will be lost when it is destroyed. If we don't generally have access, there's no temptation or possibility for this.
2. The danger of having anyone or everyone access a node as root and interacting directly with Docker Engine. I can imagine the wrong command might have an impact on every container running on that host.

Tasks that we can action to reduce the need for terminal access are:

• Document how to take a DB dump from a whitelisted IP address.
• Document how to take a DB dump from an unlisted dynamic IP address through an SSH tunnel.
• Create a Docker image that accepts multiple DB credentials and will frequently (at least daily) backup all DBs on our RDS instances and store them on S3 or sync them to a local server or developer machines via BitTorrent Sync, so we have a convenient way to always access a recent backup.
• Send logs for all containers to a logging server, so we can easily search this data and have it persist indefinitely.
• Document/verify that copy/paste in the web terminal only works in Chrome (I think?)

[markfinger]

Closing in favour of the tickets broken out from Tai's list