Allow allocating storage capacity based on instance type

[tonylsw]

Tell us about your request
When using custom launch template with Karpenter, a single node storage capacity (specified in the launch template) is used by all types of instances launched by Karpenter. We'd like to able to allocate storage capacity based on instance type.

For example:
m5.xlarge: /dev/sdf=100
m5.2xlarge /dev/sdf=200

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Currently Karpenter using a single volume capacity for all types of instances, it's not possible to allocate volume based on instance type.

Using multiple provisioners, one for each instance type, can support this, but the instance launched in such case is undeterministic
and not optimal.

Are you currently working around this issue?

Have to specify the maximum volume size in the launch template in order to avoid running out of disk.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[dewjam]

Hey @tonylsw,
Thanks for creating this issue and sorry for the delayed response. Would you be willing to share more info regarding your use-case? Are you looking to scale storage based on instance types to accommodate container images and filesystems? Or are you allocating local ephemeral storage to pods/containers?

[carlesjavierre]

Hello,

I believe he means the following (which is my case):

Either some pods generate a lot of noise (logs/stuff that doesn't need to be saved in a separate disk), or that some containers use a lot of disk space. For example, I have a client that has ONE docker image over 5GB compressed (they have the whole environment, GPU drivers and a lot of stuff) and they just can't be deployed because it instantly taints the host machine as not having enough disk. Having the ability to specify the default "root space" for the machines instead of the default 20G would be appreciated.

(In my case, the pod gets evicted as it tries to run and the node gets tainted with DiskPressure)

Now I noticed as I re-read the original post that we might have different ideas, but they have a common solution. Let's see where we get to!

[tonylsw]

Hi @dewjam ideally, we'd like to have a solution to scale up/down node storage on-demand, much like Karpenter does for pod :-) , short of that, we'd like to allocate storage based on instance type for container images.

[carlesjavierre]

@dewjam Do we have any news regarding this/these (maybe they can be considered different) features?

[dewjam]

Hey @carlesjavierre ,
My apologies for the delayed response. For your use-case, you could expand a nodes storage from the default (20GB) by defining a Block Device Mapping in the Karpenter Provisioner. Take a look at the docs and let me know if that helps: https://karpenter.sh/v0.14.0/aws/provisioning/#block-device-mappings

[dewjam]

ideally, we'd like to have a solution to scale up/down node storage on-demand, much like Karpenter does for pod :-)

@tonylsw, one feature we've considered is dynamically scaling node storage based on the ephemeral-storage requested by incoming pods. So for example, if you were to spawn 5 pods each with a request for 20GB of ephemeral-storage, then Karpenter will provision a node with 100GB + overhead to accomodate those pods.

I'm curious what your thoughts are on this potential feature.

[dewjam]

Closing this in favor of #2394