Add a non-scratch-based Docker image?

[JoelJaeschke]

TLDR: Current Docker image is based on scratch and cannot be used directly to proxy connections to another container. Solution could be to create a second set of container images that ship the tsnsrv binary along with a minimal OS.

Hey 👋,
first of all, really cool project and I am trying to integrate it into my own lab now! However, I came across a small issue (which may totally be due to me not knowing how to use the tool properly). For context, I am using podman and setup a pod for each service I intend to use along with a tsnsrv instance.

Issue

Since the base Docker image created in CI is based on scratch, running the Docker image directly fails in multiple ways. Running directly using

podman run \
    --name test \
    -e TS_AUTHKEY=<mykey> \
    tsnsrv:main tsnsrv -name test http://127.0.0.1:80

results in tsnet not being able to connect with the following error:

2023/09/30 19:52:01 could not connect to tailnet: tsnet.Up: tsnet: neither $XDG_CONFIG_HOME nor $HOME are defined

This error can be remedied by adding a "fake" $HOME such as -e HOME=/root for example, which allows tsnet to establish a connection and show up in the Tailscale UI, but when hitting the actual endpoint in the tailnet, results in an error such as

2023/09/30 20:05:56 http: TLS handshake error from 100.88.179.118:37224: 500 Internal Server Error: acme.GetReg: Get "https://acme-v02.api.letsencrypt.org/directory": tls: failed to verify certificate: x509: certificate signed by unknown authority

This bug is already documented (see tailscale/tailscale/issues/9437) and its solution also applies here.

Solution

What I ended up doing was to just create a new image based on Ubuntu (could be Alpine or anything just as well I guess) which contains ca-certificates and properly sets $HOME which then allows me to use the image as I want in my setup.

I was wondering whether you would be open to adding a second set of container images that make tsnsrv usable directly in this way? I understand the benefit of having a scratch-based Docker image of just the binary, but I am personally not a fan of having to create images that combine multiple services. I would be happy to tackle this myself but wanted to hear your opinion before.

[antifuchs]

Oh man, I'd not even thought about ca-certificates: I've been threatening to replace the docker build as it is with something more sensible (e.g. using nix oci tools to build container images), and this issue might drive me to finally do this.

For the $XDG_CONFIG_HOME error message: It is a red herring; the issue is that tsnsrv needs to know where to put the state file, and if not passed -stateDir it'll attempt to place the state file in the user's XDG config directory... which of course does not exist if $HOME isn't set.

In the meantime, I'd suggest mounting the host's CA certificates as well as a backed-up directory for the tailscale state into the container, and running tsnsrv with -stateDir. I'll deal with the CA certificate issue in the container (:

[JoelJaeschke]

Thanks for getting back so quickly @antifuchs! The part regarding $XDG_CONFIG_HOME definitely makes sense now. And your idea of mounting ca-certificates into the container is actually very elegant. Using this call works now:

podman run --name proxy --pod tsnsrv-demo -d \
    -e TS_AUTHKEY=<mykey> \
    -e TS_STATE_DIR=/var/lib/tailscale \
    -v /home/deploy/tailscale:/var/lib/tailscale \
    -v /etc/ssl/certs:/etc/ssl/certs:ro \
    tsnsrv:main tsnsrv -name test http://127.0.0.1:80

I am leaving this here so maybe someone can benefit from this at a later point. Thanks again and feel free to close this issue

[antifuchs]

Very glad to hear you got it to work, @JoelJaeschke! I've updated the docker build process now, which generates images containing a current ca-certificates package, and is faster at building too. The resulting image should allow you to drop the -v /etc/ssl/certs:/etc/ssl/certs:ro line, and otherwise just continue to work (: