kubernetes: 1.19.5 -> 1.20.4 (dockerd -> containerd)

[johanot]

Motivation for this change

This is an alternative to #109275 and #114722

Since docker 20.X has removed dockershim and Kubernetes as well has deprecated docker as container-runtime, it seemed sensible to switch to containerd as default runtime for the NixOS module.

The standard RBAC and DNS tests pass on my machine: nix-build nixos/release.nix -A tests.kubernetes.rbac.singlenode -A tests.kubernetes.rbac.multinode -A tests.kubernetes.dns.singlenode -A tests.kubernetes.dns.multinode

The PR is WIP because it still lacks documentation. Notably rel-notes, but probably also updates to sec-kubernetes in the manual.

Also, I'd like reviewers input to the question of how production ready the container runtime should be configured? Currently, I've gone with a minimal containerd.toml and minimal systemd parameters for the service. For production workloads, one likely want to configure things like ulimiits, oom-scores, cgroup params etc. Some of these settings are already set explicitly for virtualisation.docker, so users might expect this to be taken care of upstream.

Let me hear your thoughts.

cc @saschagrunert @srhb

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[blaggacao]

I believe the kube-addons manager can be provided with a deprecation message since upstream did deprecate the concept (not meant to block anything, jut a chore idea).

The CI tests fail at the same point as temhe other two PRs for good reason, the attribute is effectively not devlared in top.

Thank you a lot for pushing this!

[zowoq]

I think creating a containerd module would be better rather than adding it here, would also allow it to be used independent of k8s.

[johanot]

Will look into separating this now.

[johanot]

First version separate module here (7fc7d420b32f8dae291990a8ef5f6670a98f43d4)

[johanot]

Heads up for reviewers: 964ee25fed6772b955f6d5d603bbf9a4a8385e2a adds limits and oomscore for the containerd systemd-service. Personally, I feel infinity is quite aggressive here; that is nonetheless what upstream "recommends" (https://github.com/containerd/containerd/blob/master/containerd.service#L29-L37). Feedback would be much appreciated here.

[johanot]

Added release note.

IMHO manual changes are not needed for now, since sec-kubernetes doesn't mention docker specifically. With regards to the deprecated addon-manager, I plan to create an issue on removing this, but that's likely gonna be a next-release (21.11) thing.

(kind of expected we now have rel-note merge conflicts, will rebase nixos-unstable shortly)

[zowoq]

LGTM, thanks @johanot.

[saschagrunert]

LGTM

[zowoq]

Rebased on master to resolve a merge conflict in rl-2105.xml.

I'm running the nixos tests , I'll merge after they pass.

[zowoq]

@ofborg build kubernetes

[blaggacao]

For informative purposes only: kustomize is going to land soon as part of kubectl with kubernetes/kubernetes#98946

We should try to get this change into packages before next NixOS release.