PKI Directory should follow principle of least privilege

[rcythr]

FEATURE REQUEST

Problem
Currently kube-apiserver, kube-controller-manager, and etcd all mount in more of the files from /etc/kubernetes/pki than they need to perform their job. This is a violation of the principle of least privilege.

Proposed Fix
The certificates generation process should be adjusted to create separate directories for each combination of services which require files. An analysis of command line arguments shows that this can be done with 5 directories. The following lists files under the category based on what process is configured to use them.

1. None
   /etc/kubernetes/pki/front-proxy-ca.key
   /etc/kubernetes/pki/etcd/ca.key

2. Etcd only
   /etc/kubernetes/pki/etcd/peer.crt
   /etc/kubernetes/pki/etcd/peer.key
   /etc/kubernetes/pki/etcd/server.crt
   /etc/kubernetes/pki/etcd/server.key
   /etc/kubernetes/pki/etcd/healthcheck-client.crt
   /etc/kubernetes/pki/etcd/healthcheck-client.key
   /etcd/kubernetes/pki/etcd/ca.crt

3. Apiserver only
   /etc/kubernetes/pki/apiserver-etcd-client.crt
   /etc/kubernetes/pki/apiserver-etcd-client.key
   /etc/kubernetes/pki/apiserver-kubelet-client.crt
   /etc/kubernetes/pki/apiserver-kubelet-client.key
   /etc/kubernetes/pki/apiserver.crt
   /etc/kubernetes/pki/apiserver.key
   /etc/kubernetes/pki/ca.crt
   /etc/kubernetes/pki/etcd/ca.crt
   /etc/kubernetes/pki/front-proxy-ca.crt
   /etc/kubernetes/pki/front-proxy-client.key
   /etc/kubernetes/pki/sa.pub

4. Controller-Manager only
   /etc/kubernetes/pki/ca.key
   /etc/kubernetes/pki/sa.key

5. Api-server + Controller Manager
   /etc/kubernetes/pki/ca.crt
   /etc/kubernetes/pki/front-proxy-client.crt

Notes
#1350 proposes that /etc/kubernetes/pki/ca.crt be split into a separate file for controller manager.

If it's agreed that this is a problem which we'd like to resolve, I'm happy to work on the code change to resolve it.

[neolit123]

thanks for logging @rcythr

Currently kube-apiserver, kube-controller-manager, and etcd all mount in more of the files from /etc/kubernetes/pki than they need to perform their job. This is a violation of the principle of least privilege.

i agree with the problem statement.

The certificates generation process should be adjusted to create separate directories for each combination of services which require files.

can symlinks be used instead of duplicating files for the separate directories?

cc @fabriziopandini @randomvariable

[rcythr]

If we symlink we'd have to have the directory with the actual file mounted in as well or it would be a bad link. Mounting that other directory would include other files we don't need.

I was thinking of making subdirectories called :

/etc/kubernetes/pki/etcd/
/etc/kubernetes/pki/apiserver/
/etc/kubernetes/pki/controller-manager/
/etc/kubernetes/pki/apiserver_controller-manager/

That way no files need to be duplicated, they just exist in a directory mounted by both pods.

[rcythr]

/assign
/lifecycle active

[neolit123]

I was thinking of making subdirectories called :

if these directories are on the host, wouldn't there will be duplication of /etc/kubernetes/pki/ca.crt between /etc/kubernetes/pki/apiserver/ and /etc/kubernetes/pki/apiserver_controller-manager/?

i think it might be better to simply mount only the required files in the Pods and keep the existing structure that we have.

xref for a list of mounts the Pods are doing right now:
#1367 (comment)

[rcythr]

if these directories are on the host, wouldn't there will be duplication of /etc/kubernetes/pki/ca.crt between /etc/kubernetes/pki/apiserver/ and /etc/kubernetes/pki/apiserver_controller-manager/?
i think it might be better to simply mount only the required files in the Pods and keep the existing structure that we have.

I was thinking of modifying the structure so ca.crt would be moved into apiserver_controller-manager/. This keeps the number of mounts at a minimum. If keeping the structure the same is required it can be done, but the number of mounts will be larger.

I’m happy to try it the the way which involves no file reorganization first. Once we see the code and results we can revector a bit if we want/need to.

[neolit123]

If keeping the structure the same is required it can be done, but the number of mounts will be larger.

i have concerns of introducing directory structure changes on the host, which might break a lot of setups.

I’m happy to try it the the way which involves no file reorganization first. Once we see the code and results we can revector a bit if we want/need to.

thanks. i'd prefer to see feedback from @fabriziopandini @randomvariable too, before proceeding with an implementation.

[rcythr]

i have concerns of introducing directory structure changes on the host, which might break a lot of setups.

Agreed, it definitely creates a breaking change or a very complicated upgrade procedure.

thanks. i'd prefer to see feedback from @fabriziopandini @randomvariable too, before proceeding with an implementation.

Understood, I won't be able to work on it for a day or two anyway. I'll standby until we reach a consensus.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/lifecycle frozen

[fabriziopandini]

@rcythr @neolit123 I apologize for being super late on this thread
The topic is interesting, but as commented above, manging upgrades is far from being trivial.
It is also requires a good chunk of work on the documentation, E2E tests, and a massive communication plan to inform/coordinate all the high-level tool maintainers before the change goes lives.

Considering ^^^, I think this topic definitely requires a KEP, and that also we should explore a few options like e.g. "automatic upgrade" vs "dual-mode (support for both directory layouts) with manual opt-in for existing clusters"