Using karpenter with bottlerocket

[jcstanaway]

Description

Observed Behavior:

When using "d" instance types in AWS and karpenter 1.0.8 with Bottlerocket, we followed the instructions for using ephemeral storage as described at NodeClasses.

Specifically, this section simply says to "set instanceStorePolicy to RAID0" and further goes on to state that "The disks must be formatted & mounted in a RAID0 and be the underlying filesystem for the Kubelet & Containerd." Instructions for doing that was provided for AL2 and AL2023 and "For all other AMI families, you must configure the disks yourself."

That led us to Configure ephemeral disks where we were instructed to execute apiclient commands during bootstrap, so we updated the userData section in our ec2nodeclass to do that.

[settings.bootstrap-commands.k8s-ephemeral-storage]
commands = [
    ["apiclient", "ephemeral-storage", "init"],
    ["apiclient", "ephemeral-storage" ,"bind", "--dirs", "/var/lib/containerd", "/var/lib/kubelet", "/var/log/pods"]
]
essential = true
mode = "always"

This worked fine until we upgraded from karpenter 1.0.8 to 1.4.0. Once we upgraded to 1.4.0, any "d" instance types that were launched by karpenter failed to bootstrap and therefore failed to join the cluster:

[    4.148596] bootstrap-commands[1223]: Bootstrap command 'k8s-ephemeral-storage' failed.
[FAILED] Failed to start Bootstrap Commands.

After much troubleshooting, trial & error, searching through GitHub issues, raising an AWS support ticket and finally going through the Karpenter code, I found that PR aws/karpenter-provider-aws#7044 was added in 1.1.0 that automatically adds the above bootstrapping commands for Bottlerocket (specifically at aws/karpenter-provider-aws@55d3322#diff-399b318bb5eac0868d2ec74a40c4ee5c3bd55215accedbe385362ed9dabceb98R82). Thus when our userData also attempted to execute those commands - which were already executed - it failed and resulted in the bootstrap failure.

So, we simply needed to update our ec2nodeclass objects to remove the changes we had previously added to the userData section. However, the NodeClasses documentation is incomplete/misleading/wrong and needs to be updated.

Expected Behavior:

The NodeClasses user documentation needs to be updated to include a section for Bottlerocket AMIs and explicitly state that for Bottlerocket AMIs that "On Bottlerocket, Karpenter automatically configures the disks." and provides any additional details that may be pertinent and explicitly states that configuring ephemeral disks as described by Configure ephemeral disks is handled by Karpenter.

Reproduction Steps (Please include YAML):

1. Deploy karpenter 1.4.0.
2. Create an ec2nodeclass with the following:

    userData: |
      [settings.bootstrap-commands.k8s-ephemeral-storage]
      commands = [
        ["apiclient", "ephemeral-storage", "init"],
        ["apiclient", "ephemeral-storage" ,"bind", "--dirs", "/var/lib/kubelet"]
      ]
      essential = true
      mode = "always" 

3. Attempt to create a pod that uses ephemeral storage:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ephemeral-storage-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ephemeral-deploy-affinity
  template:
    metadata:
      labels:
        app: ephemeral-deploy-affinity
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: "karpenter.k8s.aws/instance-local-nvme"
                  operator: "Exists"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - ephemeral-deploy-affinity
            topologyKey: topology.kubernetes.io/zone
      containers:
      - name: ephemeral-storage-container
        image: busybox:1.34.1
        command: ["/bin/sh", "-c", "fallocate -l 85G /ephemeral/85G && sleep 3600"]
        volumeMounts:
        - mountPath: /ephemeral
          name: ephemeral-storage
        resources:
          requests:
            cpu: 10m
            memory: 100Mi
            # this needs to be greater than containerd_storage_size for the test to be valid.
            ephemeral-storage: 90Gi
          limits:
            cpu: 10m
            memory: 100Mi
            ephemeral-storage: 90Gi
      volumes:
      - name: ephemeral-storage
        emptyDir: {}

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[k8s-ci-robot]

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dborysenko]

it was in upgrade guide in here: https://karpenter.sh/v1.5/upgrading/upgrade-guide/#upgrading-to-110

[jcstanaway]

Fine. What about someone that is deploying karpenter for the first time using latest? Would you really expect someone to refer to the upgrade guide when they aren't upgrading? No, they would refer to the above documentation and that documentation implies that you need to explictly setup the userData section to configure the ephemeral storage.

[engedaam]

@jcstanaway Please feel free to add what you think would be helpful the documentation for the https://github.com/aws/karpenter-provider-aws. Also, can you move this issue to that repo since this is specific to the AWS provider?