Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0059]: Systemd Service Secrets #59

Closed
wants to merge 11 commits into from
273 changes: 273 additions & 0 deletions rfcs/0058-secrets-for-services.md
@@ -0,0 +1,273 @@
---
feature: secrets_for_services
start-date: 2019-10-29
author: @d-goldin
co-authors: (find a buddy later to help our with the RFC)

shepherd-team: (names, to be nominated and accepted by RFC steering committee)
shepherd-leader: (name to be appointed by RFC steering committee)
related-issues: (will contain links to implementation PRs)
---

# Summary
[summary]: #summary

This RFC introduces some interfaces, terminology and library functions to help managing
secrets for NixOS systemd services modules.

The general idea is to provide some basic infrastructure within nixos modules to
handle secrets more consistently while being able to integrate pre-existing solutions
like NixOps, or a simple secrets folder.

# Motivation
[motivation]: #motivation

There is currently a lack of consistent and safe mechanisms to make secrets
available to systemd services in NixOS. Various modules implement it in various
ways across the ecosystem. There have also been ideas like adjustments to the
Nix Store (like [issue #8](https://github.com/NixOS/nix/issues/8)), which
would allow for non-world-readable files, but this issue has made no progress
in several years.

With the introduction of Systemd's `DynamicUser`, the more traditional
approaches of manually managing permissions of some out-of-store files could
become cumbersome or slow down the adoption of DynamicUser and other sandboxing
features throughout the nixpkgs modules.

The approach outlined in this document aims to solve only a part of the secrets
management problem, namely: How to make secrets that are already accessible on the
system (be it through a secrets folder only readable by root, or a system like
vault or nixops) available to non-interactive services in a safe way.

It assumes that shipping secrets is already solved sufficiently by krops, nixops,
git-crypt, simple rsync etc, and if not, that this can be addressed as a separate
concern without needing to change the approach proposed here. Further, it is outside
of the scope of this proposal to ensure other properties of the secret store, such as
encryption at rest.

The main idea here is to allow for flexibility in the way secrets are delivered to the
system, while at the same time providing a consistent and unobtrusive mechanism that can
be applied widely across service modules without requiring large code-changes while allowing
for a gradual transition of nixos services.

# Detailed design
[design]: #detailed-design

To summarize, necessary preconditions:

* Delivery of secrets to target systems is a solved problem
* It's sufficiently secure to store the secrets or access tokens in a location
only accessible by root on the system
* The secrets store locations is secure at rest, such as full-disk-encryption.
* Interactive unlocking scenarios should be treated separately
* Linux namespaces are sufficiently secure
* The service can be run using `PrivateTmp`

Design goals:
* A set of secrets are made available to a set of services only for the duration of their execution
* Retrieved secrets are only accessible to the service processes and root
* Retrieved secrets are reliably cleaned up when the services stop, crash,
receive sigkill or the system is restarted

Core concepts and terminology:

* *Secrets store*: a secure file-system based location, in this document
`/etc/secrets`, only accessible to root
* A *fetcher* function: a function whose task it is to resolve the secret
identifier, retrieve the secret and place it in the service process' private
namespace within `/tmp` name
* Simple helper functions to *enrich* expressions defining systemd services
with secrets
* "Side-car" service: A privileged systemd service running the fetcher
function to retrieve the secret, and initially create the service
namespace
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not clear to me why the helper unit is needed. Can't the keys be fetched using an ExecStartPre=+... command?

More generally, instead of creating our own mechanism for passing keys to services, maybe the kernel keyring mechanism can be used for this? Units would call keyctl request/search/read/... to fetch keys. These keys would either be preloaded into the keyring or produced on demand using the request-key program.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I briefly tried ExecStartPre, but unfortunately it does not seem to run within the mount namespace, so it doesn't have a access to the PrivateTmp we want. I am not sure if this is intentional or not.

Regarding kernel keyring mechanism - I agree, it might be a good default backend. The directory based thing was just the dumbest proof-of-concept case I came up with (given that similar approaches are used in nixops and krops/stockholm). Part of the intention is to have a somewhat agnostic interface.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kernel keyring mechanism sounds overkill to me. Its usecase is not to communicate files between userland processes, but between userland and kernel drivers. Files are a perfectly sufficient abstraction for passing secrets around in userland.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have mixed feelings about this.

Using files usually implies having to worry about ACLs and who's allowed to access them. We can cheat by mounting them in a private namespace, but it's still a bit cumbersome.
Sometimes you want to have "use once" properties and provide a new key on every read / issue tokens on access etc. We could cheat again by providing these key files by a fuse filesystem, but then it just gets more complicated.

The kernel keyring might be a good abstraction over all this, it's just not widely adapted currently and lacking real-world usage. Reading keyrings (7) looks promising. In addition to thread/process/session, there's also an upcall feature, bouncing back to userspace to request secrets which could be a request to whatever credentials provider is used.

I'd love to experiment with that a bit, or see some real-world examples. Anybody aware of these?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kernel keyring is very much intended for userspace keys too. See the kerberos stuff that has been ported to use it for that, or systemd's cryptsetup.

I think the kernel keyring has deficiencies (upcalls, yuck! also no namespacing for containers, …), but it probably is the right approach in the long run.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I briefly tried ExecStartPre, but unfortunately it does not seem to run within the mount namespace, so it doesn't have a access to the PrivateTmp we want. I am not sure if this is intentional or not.

I think that's what the ! prefix is for:

"!" - Similar to the "+" character discussed above this permits invoking command lines with elevated privileges. However, unlike "+" the "!" character exclusively alters the effect of User=, Group= and SupplementaryGroups=, i.e. only the stanzas that affect user and group credentials. Note that this setting may be combined with DynamicUser=, in which case a dynamic user/group pair is allocated before the command is invoked, but credential changing is left to the executed process itself.

* Secrets scope: provides a context in which secrets are accessible as
attributes resolving to path names within the private namespace

The general idea is centered around this simple process:

A privileged side-car service is launched first, creates a namespace, executes
the fetcher function which retrieves the secrets and copies them into the private
tmpfs. The side-car service binds to the target service to ensure that it's shut
down and the namespace is destroyed when the target service disappears. The side-car
uses `RemainAfterExit` to keep the namespace open for other services.

The target service launches once the side-car service has been launched,
the target service then joins its namespace with the side-car namespace
and is able to access the secrets provided in the shared tmpfs in `/tmp`.
The service is now free to access the file in whichever way it wants -
for instance just passing the path to the software to be launched as
an argument, or load it up into an environment variable.

Example of user-facing API:

```
let
secretsScope = mkSecretsScope {
loadSecrets = [ "secret1" "secret2" ];
type = "folder";
};
in
systemd.services = secretsScope ({ secret1, ... }: {
foo = {
description = "Simple test service using a secret";
serviceConfig = {
ExecStart = "${pkgs.coreutils}/bin/cat ${secret1}";
DynamicUser = true;
};
};
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems unnecessarily verbose. I would do something like this:

systemd.services.foo = {
  needsSecrets = [ "secret1" ];
  ...
};

and then our systemd module can generate whatever helper units are necessary.

It also avoids imperative-sounding function names like mkSecretsScope which suggest that they allocate a unique new scope, but that's not the case, e.g. in

  secretsScope1 = mkSecretsScope {
     loadSecrets = [ "secret1" "secret2" ];
     type = "folder";
   };
  secretsScope2 = mkSecretsScope {
     loadSecrets = [ "secret1" "secret2" ];
     type = "folder";
   };

secretsScope1 and secretsScope2 are actually the same scope.

Copy link
Author

@d-goldin d-goldin Nov 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems unnecessarily verbose. I would do something like this: [...]

I agree, it's more compact. Why I initially decided against it and for something that modifies the resulting structure separately was to reduce changes to the existing systemd modules. At the same time I thought it would be nice to be able to get the secrets as arguments, which I thought made it nicer to deal with in code. In the needsSecrets case, if I understand it correctly, the user would be requesting a secret by its name, like in the scope creation, but then would have to possibly deal with paths (as strings) to pass the secret as an argument to the service, or load it into an env-var, because there would not be an automatic mapping mechanism anymore. Unless we pack it up into a shell variable or so.

It also avoids imperative-sounding function names like mkSecretsScope which suggest that they allocate a unique new scope, but that's not the case, e.g. ...

I do not necessarily perceive mk* as an imperative terminology, but that depends on the reader. We have a lot of pure mk* functions that construct some structure. But I'm not at all attached to the naming here, so we can change it to whatever seems more suitable if they're still around further down the road.

Edit: In fact, I'm not attached to most of the terms in the RFC, such as "sidecart" and similar. I merely picked them from what I thought would make it easily enough understood. So if there are suggestions to rename things, I'm up for it.

Copy link
Author

@d-goldin d-goldin Dec 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@edolstra: Did this sufficiently address your remarks? I'm not super familiar with the inner workings of the process, so for now I just left those as discussion comments here, but I'm willing to incorporate some of the things pointed out into "alternatives" or the core section, if there is some consensus around that.

Copy link
Author

@d-goldin d-goldin Apr 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, getting back to this suggestion - if somebody could point me to the simplest way of implementing it with such an additional field on any systemd service without needing to change too many things in the guts of core modules I'm willing to try and adjust the poc to see how that works. One thing I do kinda like though is the ability to have some at least very basic checking of secrets used, which wouldn't work the same way with just strings.

```

This is a minimal example of a service depending on a secret called `secret1`.

More specifically, in this example a secrets scope is created - to allow for
extensibility and differentiation a store has a type. In this case "folder"
denotes a secrets store in the form of a root-only accessible locked down
directory on the local filesystem. Here we want to acquire access to
2 secrets, and 2 secrets only, which are specified in `loadSecrets`, by
their id. How a secrets identifier is resolved, should be up to the fetcher
function and here it's just trivially the file-name (this of course does not
allow for file extensions).

These secrets are then made accessible to the target service's unit definitions as
arguments passed into a lambda within the scope. These arguments then point to
some private location within the namespace - in our case `secret1 ->
/tmp/secret1`.

The resolution and location of the secrets is decided by the implementation and
should be of little concern to the user as it could potentially change if other
private locations besides `/tmp` become available. It is still possible
to point to the file locations, but is less convenient and
would not result in build time errors when wrong paths are specified - thus the
arguments add a little bit of convenience and safety, aside from the indirection
they offer.

For every service defined this way in a scope, a side-car container is generated
_per service_ and wired up with the target service. This means that the ability
to create a scope does not break isolation between multiple target services
but can add a little bit of developer convenience.



A working POC example can be found in https://github.com/d-goldin/nix-svc-secrets/blob/master/secrets-test.nix.
In this example the target service is forced/asserted to utilize `PrivateTmp=true`.

For the above simple case, the generated service definitions looks like the following:

Side-car service:

```
[Unit]
Before=foo.service
BindsTo=foo.service
Description=side-car for foo

[Service]
Environment="[...]"

ExecStart=/nix/store/v1bm9bnmbxbq9740yj0a64b3vz3y7ryz-secrets-copier secret1 secret2
PrivateTmp=true
RemainAfterExit=true
Type=oneshot
```

Target service:

```
[Unit]
Description=Simple test service using a secret
JoinsNamespaceOf=foo-secrets.service

[Service]
Environment="[...]"

DynamicUser=true
ExecStart=/nix/store/3kqc2wmvf1jkqb2jmcm7rvd9lf4345ra-coreutils-8.31/bin/cat /tmp/secret1
PrivateTmp=true
```

## NixOS modules integration

To implement an interface as outlined above, a little bit of supporting functionality
needs to be added somewhere in the nixos library functionality.

An example of some needed functions, of which some could be user exposed configuration,
is shown in https://github.com/d-goldin/nix-svc-secrets/blob/master/secretslib.nix.

This is mostly functionality containing a _registry_ of existing fetchers, which
might need to be configured by the user via their system configuration, the
fetcher logic itself and functionality to generate side-car services and
expose the secrets scope.

## Rotating secrets

Right now, secrets rotation is not done automatically. When new secrets are
pushed, it is the responsibility of the user to restart the services affected.
d-goldin marked this conversation as resolved.
Show resolved Hide resolved

It is assumed that once secrets are rotated, old secrets will become invalid and
no further harm is done aside from failing to access the resources (and possibly
restart on its own).

It would be possible to allow for automatic restarts using systemd path monitors.
Also see _Future work_.

# Drawbacks
[drawbacks]: #drawbacks

I can't really think of a serious drawback right now, but hopefully the
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does the secret store deal with system/main config upgrades? When I uninstall an old service and install a new one that asks for a secret of the same name, does the new service get access to the old secret?
When I uninstall a lot of services, are old secrets garbage collected? Do downgrades always work seamlessly?

Can you outline the possible pitfalls regarding that topic in an extra section?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the current shape, no such management would happen. The secrets would just remain in the secrets store and if a config author decides to re-use them, then so be it. I can include that. I was initially thinking of the drawbacks of "sth that would be worse when adding this", but I guess it depends on the reference point. Definitely good points to add, even if they would remain unhandled.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it's maybe a strawman, but if you put a secret in /nix/store, it's going to vanish upon garbage collection when I uninstall the package. If I don't care whether it's readable by root or by all users (e.g. inside a container), this is a drawback now because no such garbage collection would take place.

As in "something that would be worse when adding this", you could also say that people possibly expect that NixOS doesn't do any state management by default and that downgrading will bring their machine back into the exact state, but that's not the case anymore.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could also say that people possibly expect that NixOS doesn't do any state management by default

The difference being that before this RFC, I'd manually handle the secret state and remember to handle it. After all, I'm deleting the line saying service.foo.secretsFile = "/foo/bar";, and when I delete the line, I'll delete the file. This is now not the case anymore. Instead, I need to remember how this semiautomatic secret handling works.

Copy link
Member

@globin globin Nov 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you put a secret in /nix/store

It is exactly the purpose of this RFC not to have secrets in the nix store. This is possible without any problems and if they are referenced in the nixos config they would not be garbage collected as currently is the case. And this can still be achieved with this proposal as before.

If I don't care whether it's readable by root or by all users (e.g. inside a container)

Actually currently the nix store of (nixos) containers is shared with the host and all other containers, so that might be an issue nonetheless.

With regard to secret state handling, a lot of services have the possibility do use a passwordFile, secretFile, etc. but there is no abstraction for handing out permissions to secrets correctly, which is something like the proposal in this RFC is needed. Currently you have to not only drop the files on the machine yourself in the correct destination (same with this RFC, as long as you are using a fetcher that requires that), but also set the correct permissions for the respective service, which might be a chicken-and-egg problem of adding users for service by enabling it and the service requiring the secret.

RFC process can surface some.

One aspect is of course the additional number of services generated, but this
does not seem to be a big issue when using NixOps.

# Alternatives
[alternatives]: #alternatives

* One approach that has been proposed in the past is a non-world readable store,
in issue #8 (support private files in the nix store, from 2012). While this would
be pretty great, it's rather complex and has not made progress in a while.

* "Classical" approach of just storing secrets readable only to a service user and
utilizing string-paths to reference them. This does not work well anymore with
DynamicUser.

* A somewhat similar approach exists in
https://cgit.krebsco.de/stockholm/tree/krebs/3modules/secret.nix
(loosely related to krops).

* NixOps implements a similar approach, providing a service to expose secrets
via a systemd service after it has taken care of deployment.

Impact of not doing this:

Continued proliferation of various, individual solutions, per module and
depending on the users environment.

Persistent confusion by new-comers and veterans alike, about what the a
recommended way looks like and a variety of different approaches within
nixos service modules.

# Unresolved questions
[unresolved]: #unresolved-questions

* Is it sufficient to put responsibility on restarting services after key changes
onto the user or would an automated mechanism be better?
d-goldin marked this conversation as resolved.
Show resolved Hide resolved

* Would it be better to create a side-cart per secret instead of per secret-scope+service?

# Future work
[future]: #future-work

* When using a scope with multiple services, ideally only the secrets
referenced in the services definition should be made available to each
service. Right now all the secrets of the scope are blindly copied.
* Transition of most critical services to use proposed approach
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to transition some non-critical services first, to gain some experience without breaking anything. If this works for 10+ regularly used non-critical e.g. webapps, then transition everything to this approach.

Copy link
Author

@d-goldin d-goldin Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix the wording on that one. Parts of this section are maybe still a bit too "note to self"-like. What I intended was not for "inclusion into upstream" but more to hypothetically verify this is covering all the most important aspects needed for the most critical services. Your version is definitely better for an actual slow inclusion into upstream step.

* Implementation of more supported secret stores, such as nixops and vault
* Optional restarting for services affected by rolled secrets
* Merging some attributes better than in the POC - like JoinsNamespaceOf
* Provide simple shell functions for features like loading a file into an environment
variable and possibly some wrappers to make injecting secrets into config file templates
easier.
* Decouple secret id name from secret file name, for convenience and to make more complex
file names work.