Implement privilege de-escalation

[mholt]

Go can't really perform de-escalation safely within the same process because of the runtime, which has always led us to recommending using setcap.

But this comment gave me an idea -- that simply starting a new, unprivileged process and passing it the listeners that were created from the privileged process -- could work as an effective way to drop privileges. Caddy already does this for graceful reloads, except for the part about dropping privileges.

Even though Caddy can kind of already do this, plenty of refactoring will need to happen so that this could work smoothly. (Of course, this won't work on Windows.)

I have not done this before so I need to learn more how this works. For example, it seems like I will need to set the user ID and/or group ID for the new process (for the "Credential" field)... and I am not sure how to get those without asking the user, if that's even possible.

EDIT: A Google employee made this PoC for it: http://play.golang.org/p/dXBizm4xl3 -- user 65534 is "nobody" I think, and I suppose the user and group IDs could be configured via command line flag.

Feedback is welcome.

[mholt]

At start, Caddy would start the listeners then run a new unprivileged process. At this point two things could happen:

1. The original process stays alive and blocks until the unprivileged process stops. This leaves two caddy processes running, but the root one is idle and harmless. Preserves current experience, except having to discern which caddy process is the 'real' one.
2. Or, the privileged process exits and the unprivileged process continues to run in the background. This unblocks the shell, but it leaves only one caddy process running.

Remember, this only happens if run as root, and the new process only happens once.

I'm leaning toward the first option, where the original process stays alive and blocks, in the interest of preserving the current experience.

Thoughts/preferences?

[jungle-boogie]

I don't know if this is possible with go/caddy, but a program I used (fossil-scm) puts itself in a chroot when it starts:
http://fossil-scm.org/index.html/doc/trunk/www/server.wiki

In both cases notice that Fossil was launched as root. This is not required, but if it is done, then
Fossil will automatically put itself into a chroot jail for the user who owns the fossil repository 
before reading any information off of the wire.

The context above is referencing inetd/xinetd, but doesn't always mean that. If I launch ./fossil, it will still create a chroot.

Is this something caddy can do?

Here's the C code that handles what's described above:
https://www.fossil-scm.org/index.html/artifact/e75796be5338a81c?ln=1463,1523

[mholt]

I doubt chroot would be useful, since: golang/go#1435 (comment)

[dstapp]

@mholt I'm not really into Go but from a operating systems' point of view I think option 1 is the way to go e.g. regarding PID tracking, especially when it comes to init scripts for the various platforms. That would also include having all child processes killed once the main process is stopped.

Also, if I'm not misinterpreting that, Apache does it that way, too:

 |-+= 06109 root /usr/local/sbin/httpd -DNOHTTPACCEPT
 | |--- 07390 www /usr/local/sbin/httpd -DNOHTTPACCEPT
 | |--- 07398 www /usr/local/sbin/httpd -DNOHTTPACCEPT
 | |--- 07421 www /usr/local/sbin/httpd -DNOHTTPACCEPT
 | |--- 07440 www /usr/local/sbin/httpd -DNOHTTPACCEPT

As far as I'm concerned, privilege dropping is the one most important topic that keeps me from using Caddy for larger production usage though I'm using it for smaller projects and really love it. As Caddy is so portable and self-contained i'd be sad to see setcap being the final solution to this problem as this requires additional system configuration efforts.

About the Windows thing... I'd suggest to keep the current behavior on Windows as I think, Caddy on Windows might primarily be used for web development rather than production deployment.

EDIT: Just comprehended the priv-dropping PoC you linked. Seems to be exactly what'd be neccessary but I'm not sure if this will work when combining it with chroot because as you're leaving the process context you might also leave the chrooted context (based on the implementation), but I'm not sure about that.

[mholt]

@dprandzioch Thanks for the feedback. This is good to know.

So if Caddy had two processes going, which one would process the signals? Both? Or just one? Which one? How do we make it clear which process to signal?

[mark-kubacki]

In distrustful decomposition with the pattern privilege separation the parent is in charge of handling all OS-level signals (and hence reaping childs, too).

Sometimes the parent attaches to the child and monitors its activity with the intent of killing it on trespassing. (We have now Seccomp to do this.)

With Caddy, the parent would load and (ideally) hold all certificates, or delegate handling them to another set of child processes.

[martijnve]

@dprandzioch
I agree, killing the parent messes with init scripts and container engines (rkt / docker). Which is s shame since caddy and containers are currently such a nice fit.

[DenBeke]

Any updates on this?

[mholt]

Nope.

[mark-kubacki]

Caddy does not need to run as root, and does not need the setcap hack – which has the dangerous side-effect of any other unprivileged user being able to hog ports at will.

1. An init daemon can start caddy as unprivileged user, but keeping its capability to open a low port (like 80 and 443). Here is how you can do that with systemd (it's my version before any of the later dubious edits).
2. You can use capsh (see also: capsh --print), standalone or in a less elaborate init daemon. See fig 1.
3. Start caddy using Docker or rkt, which configure port forwarding for you.
4. Start caddy in its own namespace. (There are about four other files needed in its chroot to comprise a valid Linux system.)
5. Use socket-activation with a patched caddy.

1–4 already work without any additional work.

fig 1:

capsh --keep=1 --uid=1000 --gid=1000 \
  --caps="cap_net_bind_service,cap_lease+eip" \
  -- -c "exec /usr/bin/caddy …"

[cmehay]

In fact, we need privilege de-escalation to protect TLS certificates. Caddy cannot read TLS certificate owned by root, and the rights on certificate must be changed to be readable by another user than root. It's a security issue.

[mholt]

@cmehay

In fact, we need privilege de-escalation to protect TLS certificates. Caddy cannot read TLS certificate owned by root, and the rights on certificate must be changed to be readable by another user than root. It's a security issue.

Why are your certificates only readable by root? You can still use users/groups to wall off access to the certificates from others. Add your web server to some group that's allowed to read the certificates maybe?

[cmehay]

It's a common security practice, if you have any security issue in your web server or in you web app, if the cert is readable by the same user that run the web server, it will be readable by the attacker.

[mholt]

@cmehay I assume when you say "cert" here you mean private key. Caddy loads these into memory, which, even if the keys are read-protected, are still available to an attacker who gains privileges.

Also, keys are not necessarily loaded at server startup. They may be loaded later on.

[cmehay]

Yes, I'm talking about the private key.

Reading memory of another process is not the same deal that reading file on file system afaik.

Also, keys are not necessarily loaded at server startup. They may be loaded later on.

This is the behavior of Apache and nginx, they read cert and private key as root and fork to another process with another uid and gid.

[mholt]

This is the behavior of Apache and nginx, they read cert and private key as root and fork to another process with another uid and gid.

But they read the key before forking; Caddy might not know it needs the key until after the fork.

[cmehay]

But the keys are specified into Caddyfile right ? For sure this won't apply for generated cert through let's encrypt, but for the certificates in Caddyfile, caddy could read them before forking.

[mholt]

But the keys are specified into Caddyfile right ?

Not usually anymore, no.

For sure this won't apply for generated cert through let's encrypt, but for the certificates in Caddyfile, caddy could read them before forking.

This is not the majority use case. Besides, what security problem are we solving if the Let's Encrypt certs remain unprotected?

[mark-kubacki]

An attacker can read any private key from the process' memory. The mitigation is using a different architecture (which I suggested, and which has been rejected understandably due to the scope of caddy), one where the same process cannot r/w to any device or disk, and the other which cannot contain or work with any secrets. (A consequence of applying here the design pattern → distrustful decomposition.)

It's in fact easier for an attacker to read process memory than an arbitrary file! The latter needs interaction with the kernel (syscalls), which could be detected by a monitor {3.42 there}.

Furthermore, Linux' DAC is more versatile than you might think:

chown root:root secret.key

setfacl -m u:caddy:r secret.key

But, as said above, the contents will be loaded into memory anyway and (even in nginx) made available to any childs. (Compare: keyless SSL)

If you're concerned with this and don't want to switch to a daemon with a different architecture, then sponsoring work to interface caddy or Golang with HSM modules might be the way to go here. (→ e.g. miekg/pkcs11)

[cmehay]

Thanks for your reply, it's very instructive.

[coolaj86]

FYI: launchd on macOS does not support the set cap type of stuff, unfortunately:

See https://superuser.com/a/726922/73857

There are ways to get around it, but they aren't things that should be installed alongside caddy nor changes that should be made without the user doing it themselves (like fudging the ipfw port forwarding)

Seems like using the setuid and setgid support that's available and A) not letting insecure modules into caddy and B) filing bugs as they pop up might be a good forward movement?

[tlrobinson]

Related question: is there any way for a non-root user to send the USR1 signal (reload config) to Caddy (started as root)?

(I'm specifically interested in a solution for macOS)

EDIT: or something similar to setcap (https://github.com/mholt/caddy/tree/master/dist/init/linux-systemd#instructions) but for macOS.

[sftim]

sponsoring work to interface caddy or Golang with HSM modules

Picking this approach also lets admins deploy a software HSM, which gives some improvement in information security without needing any new hardware. Eg SoftHSM

[lassik]

Very interesting server with the Let's Encrypt automation! Thank you for making it.

May I ask about the status of this issue?

If the goal is to bind sockets as root and then drop privileges, would you accept a trivial C program to do it? It's like 50-100 lines of code. Very similar to the http://play.golang.org/p/dXBizm4xl3 example linked in the first post.

The caddy executable written in Go could take an optional command line flag, -socket-fds 3,4,5. This flag would give a comma-separated list of file descriptor numbers to look for already-bound server sockets. For each of these numbers, it would try getsockname() to get the address to which that socket is bound. If that syscall returns errno ENOTSOCK then it's not a socket. Otherwise the resulting struct sockaddr can be used to find out the address, port number, and whether it's IPv4/IPv6/something else. I don't know what the Go equivalents to these Unix syscalls are, but surely they exist.

So the trivial launcher program in C would bind a port-80 server on fd 3 and a port-443 server on fd 4 and then drop privileges to another UID/GID and execute caddy -socket-fds 3,4. It could use Unix execvp() so caddy retains the same PID as the launcher program.

Are there problems with this approach? It's Unix only but seems simple to implement and would be a big help there. Also there would be no Go code running as root, so no worries about goroutines and threads. Trivial to audit all the code that runs as root.

[lassik]

Another thing that would tie into this is the daemontools family of process supervisors. I got the impression that Caddy needs to restart every now and then due to renewal of certificates, and is already using exit codes to say what caused the exit. The daemontools family operates on the principle that a service (such as Caddy) is constantly running. A "run script" starts it and leaves it running in the foreground, using execvp() to keep the same PID as the run script. daemontools monitors this PID and when it exits, it re-runs the "run script" to start it again. In this scenario, Caddy could simply exit whenever it needs to be restarted. The run script would run the above-discussed 50-100 line C program as root, which would cause Caddy to start in the foreground and keep running. The result seems very reliable and natural to me.

I don't know if you're planning to have boutique "remote control" facilities for Caddy but with 15 years of Unix experience I can't speak highly enough of the simplicity and robustness of the daemontools approach. Even if you don't want to adopt it as Caddy's main approach, it would be nice to have command line flags to support it. I'll gladly discuss details and answer any questions about it.

[sftim]

the trivial launcher program in C would bind a port-80 server on fd 3 and a port-443 server on fd 4 and then drop privileges to another UID/GID and execute caddy

@lassik: one thing that then becomes difficult is protecting private key material (so that it's usable for TLS but not readable by the main Caddy worker process)

[mark-kubacki]

I will be implementing such a webserver shortly. Wouldn't be my first iteration, as I've done so in the past for clients for certification. Follow me on Github or Twitter for updates. :-)

[jdebp]

lassik, you have just reinvented

• the LISTEN_PID/LISTEN_FDS mechanism;
• programs like tcp-socket-listen, local-stream-socket-listen, fifo-listen, s6-tcpserver6-socketbinder, and s6-tcpserver4-socketbinder; and
• programs like setuidgid, setuidgid, and s6-setuidgid.

[mholt]

Given the dynamic nature of Caddy 2's configuration, and the growing proliferation of kernel features like setcap, this doesn't seem to make as much sense as it did 4-5 years ago. (For example, using the API you could POST a config that requires a new binding of a low port, but if Caddy has already de-escalated, that is impossible.)

Currently, the best way to drop privileges in a Go program seems to be to not run it with those privileges in the first place, which is very doable on the most common production environments (semi-modern Linux) using setcap or by just using higher ports. (And on MacOS, low ports no longer require root privileges.) While you can run Caddy as root -- and probably even more safely than any other C-based web server -- you don't have to 99% of the time.

This is the oldest open issue and no progress has been made on it, nor is there any known or pressing exploit that necessitates my time right now in lieu of other development priorities, so I'll close it.

Feel free to continue discussion, but I doubt anything will be done here unless the following is presented:

1. a convincing argument that this needs to be a high development priority to benefit the majority of users going into the future -- including a concrete, nonnegotiable reason why Caddy must be run as root in the first place,
2. a working PoC exploit that would affect most/all users which demonstrates why Caddy without privilege de-escelation is a vulnerability and cannot be solved by not running Caddy as root (and ANY Go program for that matter),
3. and a feasible, technically sound fix for the demonstrated vulnerability.