Feature: instance requirement for nested virtualization support

[a7i]

Description

What problem are you trying to solve?

Workloads that run nested VMs (e.g. agentic-ai / runtime sandboxing) need to schedule only onto EC2 instance types that support nested virtualization, and eventually launch those instances with nested virtualization enabled.

#8966 covers enabling CpuOptions.NestedVirtualization at launch time. This issue is complementary: a scheduling requirement / instance attribute so NodePools can express "only instance types that support nested virtualization", similar to karpenter.k8s.aws/instance-capability-flex.

Today there is no first-class Karpenter requirement key for this. Consumers must maintain explicit instance-family lists or use out-of-band tooling.

Proposed Karpenter API

Add a requirement label derived from instance type capabilities, e.g.:

requirements:
  - key: karpenter.k8s.aws/instance-capability-nested-virtualization
    operator: In
    values: ["true"]

Karpenter would set the label on each instance type from AWS capability data (true when the type supports nested virtualization, false otherwise), matching the instance-capability-flex pattern.

AWS API mapping (querying supporting instance types)

DescribeInstanceTypes already exposes this via ProcessorInfo.SupportedFeatures:

aws ec2 describe-instance-types \
  --filters Name=processor-info.supported-features,Values=nested-virtualization \
  --query 'InstanceTypes[].InstanceType' \
  --region us-west-2

Reference: Amazon EC2 nested virtualization on virtual instances

GetInstanceTypesFromInstanceRequirements / Fleet

GetInstanceTypesFromInstanceRequirements does not currently expose processor-info.supported-features as an InstanceRequirements field. Options:

1. Karpenter-side (short term): resolve instance types via GetInstanceTypesFromInstanceRequirements, then filter with DescribeInstanceTypes using processor-info.supported-features=nested-virtualization (same approach as hypervisor filters today).
2. AWS API enhancement (long term): add nested virtualization support to InstanceRequirements so Fleet / capacity queries can filter in one call, e.g. a SupportedProcessorFeatures (or similar) parameter on GetInstanceTypesFromInstanceRequirements and EC2 Fleet InstanceRequirements.

Launch-time enablement remains separate and belongs with #8966:

aws ec2 run-instances \
  --instance-type m7i.4xlarge \
  --cpu-options NestedVirtualization=enabled

How important is this feature to you?

Important for EKS/Karpenter users running nested VM workloads. Without a standard requirement key, every platform team reimplements the same DescribeInstanceTypes filter logic.

Related: #8966

• Please vote on this issue by adding a 👍 reaction to help prioritize
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[a7i]

/assign

[ryan-mist]

Consumers must maintain explicit instance-family lists or use out-of-band tooling.

This should be handled today without requiring instance family lists - its done by a compatibility check in offering generation (https://github.com/aws/karpenter-provider-aws/pull/9043/changes#diff-4bea6c64e0f3e9d388b6fdf62e35e6e14ea69bbf8be65ecd561e3bbf8652a300). Are you seeing errors logs from CreateFleet related to this?

[a7i]

@ryan-mist the linked PR adds the support to include it in the EC2NodeClass. This issue is to request a NodePool to only filter instances that support nested virtualization.

For example, if I have a NodePool that has multiple Intel node types (m5, m6i, m8i), when I have the requirement, I want it to only provision m8i (a node that supports nested virtualization)

[ryan-mist]

Sorry for the confusion from my earlier statement - I was trying to state that if a EC2NodeClass enables nested virtualization, then the linked NodePools will only provision instances that support nested virtualization.

The example

NodePool that has multiple Intel node types (m5, m6i, m8i), when I have the requirement, I want it to only provision m8i (a node that supports nested virtualization)

should already work today without this new proposed requirement on the NodePool

[github-actions]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.