Relax soft constraints if pods cannot be scheduled

[ellistarn]

1. Issue, if available:
#417

2. Description of changes:
Useful for:

• AZ Preferences (and regional fallback)
• Spot Preferences (and OD fallback)

3. Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: v1alpha4 Docs Mega Issue #704
• [] No

Testing

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - preference:
              matchExpressions:
              - key: node.k8s.aws/capacity-type
                operator: In
                values:
                - spot
            weight: 1
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - m5.large
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - us-west-2a
      containers:
      - image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
        imagePullPolicy: Always
        name: inflate
        resources:
          requests:
            cpu: "1"
      nodeSelector:
        karpenter.sh/provisioner-name: default

provisioner.yaml

apiVersion: karpenter.sh/v1alpha4
kind: Provisioner
metadata:
  name: default
spec:
  architectures:
  - amd64
  provider:
    apiVersion: extensions.karpenter.sh/v1alpha1
    capacityTypes:
    - spot
    - on-demand
    cluster:
      endpoint: https://BE5C79AC824BB1EF67DB8CC14D6A263F.gr7.us-west-2.eks.amazonaws.com
      name: etarn
    instanceProfile: KarpenterNodeInstanceProfile-etarn
    kind: AWS
  ttlSecondsAfterEmpty: 30

Starting provisioning loop
Waiting to batch additional pods
Found 1 provisionable pods
Computed packing for 1 pod(s) with instance type option(s) [c5a.24xlarge]
Discovered subnets: [subnet-0f3fa09b6ca9de74a subnet-05ad119d43b8c07b3]
Discovered kubernetes version 1.20
Discovered ami ami-0494b922d4a70c8d6 for query /aws/service/eks/optimized-ami/1.20/amazon-linux-2/recommended/image_id
Discovered security groups: [sg-000d68e67c3fe53d8 sg-0c8c73058f4cbaaaa]
Discovered caBundle, length 1025
Discovered launch template Karpenter-etarn-10670878007216278433
error	{"commit": "027bab6", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "name": "default", "namespace": "", "error": "launching instance, with fleet error(s), UnfulfillableCapacity: Unable to fulfill capacity due to your request configuration. Please adjust your request and try again."}
Starting provisioning loop
Waiting to batch additional pods
Found 1 provisionable pods
Pod inflate-7c8c66dbcd-b586k/default previously failed to schedule, removing soft constraint: spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution[0]={"weight":1,"preference":{"matchExpressions":[{"key":"node.k8s.aws/capacity-type","operator":"In","values":["spot"]}]}}
Computed packing for 1 pod(s) with instance type option(s) [c5a.24xlarge]
Discovered caBundle, length 1025
Launched instance: i-0c262704459ede77b, type: c5a.24xlarge, zone: us-west-2a, hostname: ip-192-168-75-12.us-west-2.compute.internal
Bound 1 pod(s) to node ip-192-168-75-12.us-west-2.compute.internal

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✔️ Deploy Preview for karpenter-docs-prod canceled.

🔨 Explore the source changes: 855c58a

🔍 Inspect the deploy log: https://app.netlify.com/sites/karpenter-docs-prod/deploys/6153a5f3aec65900076de5de

[JacobGabrielson]

Looks good to me so far

[JacobGabrielson]

Just checking - it should not be relaxed because it's the last (only) one?

[ellistarn]

Correct. You can't remove all the hard requirements otherwise they wouldn't be very hard =/

[JacobGabrielson]

Nice