Allow setting kernel parameters (sysctl) from instance user data

[hendrikhalkow]

What I'd like:

I'd like to to run buildah in non-privileged containers on Bottlerocket. For that, I have to set /proc/sys/user/max_user_namespaces to a value greater than zero. It would be great if I can do that from the instance user data.

Any alternatives you've considered:

Currently, I do that with a privileged DaemonSet. That DaemonSet has a privileged init container that just does echo 16384 > /proc/sys/user/max_user_namespaces and a regular container that does nothing, but consume memory.

[samuelkarp]

Hi @hendrikhalkow, thanks for opening this issue!

Enabling a mechanism to customize kernel parameters at boot is something we've started to talk about a little bit, but I don't think we've come to any specific conclusions on what that should look like yet. One of the challenges is trying to find the balance of what should be directly modeled in the Bottlerocket API versus what shouldn't. procfs and sysfs are reasonably large interfaces already, and there's a ton of existing software that expects to use those interfaces; trying to hide them in the Bottlerocket API can make it harder for those programs to use Bottlerocket while also creating a risk for the settings exposed in the Bottlerocket API to become out of sync with the kernel!

I think the work-around you have today is reasonable; are you having any issues with it?

Do you mind if I change the title of this issue to better reflect the general discussion of customizing kernel parameters through user-data?

[hendrikhalkow]

Hi @samuelkarp, sure – go ahead and change the title to something more general that covers the cases you mentioned. The only issue I have is that it's there, does nothing but using resources, but besides that, it works perfectly. Here is the DaemonSet that I use. Feed free to adopt it according to your needs.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: buns
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: buns
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
      labels:
        app.kubernetes.io/name: buns
    spec:
      nodeSelector:
        h5k.io/node-role: buildah
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - key: h5k.io/node-role
        operator: Equal
        value: buildah
        effect: NoSchedule
      initContainers:
      - name: buns
        image: registry.access.redhat.com/ubi8/ubi-minimal:8.2-349
        imagePullPolicy: Always
        securityContext:
          privileged: true
        command:
        - sh
        - -c
        - echo 16384 > /proc/sys/user/max_user_namespaces
        resources:
          requests:
            memory: 16Mi
            cpu: 50m
          limits:
            memory: 32Mi
            cpu: 50m
      containers:
      - name: sleep
        image: registry.access.redhat.com/ubi8/ubi-minimal:8.2-349
        imagePullPolicy: Always
        command: ["/bin/sleep", "infinity"]
        resources:
          requests:
            memory: 16Mi
            cpu: 50m
          limits:
            memory: 32Mi
            cpu: 50m