Where can I find sagemaker ML instance limits?

[jackdh]

I can't see them in the EC2 -> Limits

and I keep running into the issue of having to wait 12-24h for the service to be increased.

Is there anyway them all increased?

[fandotli]

hi, you can find default limits of AWS services including those of Amazon SageMaker on this page: https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html. SageMaker limits are independent of EC2 instance limits.

We recommend you to submit limit increase requests and we'll do our best to get them approved quickly. Please note the goal of the limits is to prevent customers from mistakenly launching too many instances which leads to surprising bills. Because of that, we require customers to provide reasons for the limit raise to make sure we are raising the right limits for their needs. Thanks.

[jackdh]

That does not show mu current limits jus the default ones.

[fandotli]

unfortunately SageMaker does not support showing remaining account quota at the moment. We have taken your request and will try to prioritize it into our roadmap.

[aurelien-roy]

For some reason, my newly created account has every instance type (other than t2 medium) in SageMaker limited to 0, instead of default limit documented on the above link.

[twbraam]

For some reason, my newly created account has every instance type (other than t2 medium) in SageMaker limited to 0, instead of default limit documented on the above link.

I have the same issue. Currently I have no idea what my limits are. I do have m4.xlarge available for training at least. However, this does not apply to endpoint hosting where the limit of m4.xlarge is set to 0.
A crucial oversight since the alternative, t2.medium, is not part of the Free Tier. I just requested a limit increase, but I read it can take a month for it to finally be put in place, which is half of the time available of the 2 months that are given for the Sagemaker Free Tier.

[dshahrokhian]

+1

Any news on this regard?

[jackdh]

We got feedback from AWS support that there is a hard cap of 100 instances per batch job.

However you can increase your total limit of running them in parallel, for instances 5 * 100 batch jobs.

[adiell]

API for the remaining quota could be very useful.

[keerath]

@fandotli Any idea about when the service quotas for sagemaker will be available through an API ?

[setu4993]

Our account didn't have the limits as specified here but different, much lower limits. How can we find out current limits without trying to spin up instances and getting an error message?

Also, which limits apply to SageMaker Studio notebooks? Studio doesn't appear in the dropdown for service quota increases on AWS Support, and the instance sizes possible for those notebooks (ml.m5.*, ml.g4dn.*, etc.)? I don't see them in the dropdown for Notebook Instances or SageMaker Training.

[ydennisy]

Has anyone found where to increase these limits?

[setu4993]

@ydennisy : Contacting AWS Support has worked for me. There's a specific type of request that applies to increasing limits.

[Leggerla]

Is this issue really resolved? I can't find a list of my current Sagemaker limits anywhere.

[jmahlik]

This probably should be re-opened. Landed here after searching around for the quotas.

[benbogart]

After getting a quota increased (19 days after requesting) I asked AWS support where I could verify the new limits. AWS support could not answer, they could only send me to the default limits page .

[jmahlik]

Related #1119

[jackhamburger]

Agree this should be re-opened. There also should be a lever for requesting increases that doesn't involve a support ticket. I believe that this also changed recently as old runs that were passing are now failing for not having instance access.

[jmahlik]

There's quite a bit of overhead with the current process. Requesting an adjustment for multiple instance types, job types and/or regions can require multiple service tickets. Ended up having to submit around 8 service tickets manually last time. Then attempt to manually validate each of them after submitting another two service tickets to get the new limits after the increase.

I did open a support ticket referencing this issue requesting the service team look in to adding the functionality. So, hopefully, it will be addressed in the future.

[gdanthi]

SageMaker has now integrated with Service Quotas. You can find your current SageMaker quotas per region in the Service Quotas console. You can also use the AWS SDK/CLI to access your quota values.

[jmahlik]

SageMaker has now integrated with Service Quotas. You can find your current SageMaker quotas per region in the Service Quotas console. You can also use the AWS SDK/CLI to access your quota values.

Amazing! Just gave it a try and this is great. Thanks!

Will there be a blog post about this on the ml blog or somewhere else? So that this issue isn't the first thing that pops in search results.