Log more information when updating deployments

[ospillinger]

Description

Add more status messages in cx logs.

Motivation

Improve user experience when waiting for long deployments to update.

[wise-east]

Has this been updated? I've modified some code for the text-generator example in the repo and currently waiting for it to deploy, but it's been saying updating for over an hour now. I have also deployed the given text-generator example but it is also taking a long time. If I try to do cortex logs <api that is taking a long time>, I just get fetching logs.... I am wondering what the usual wait time is for deployment of a GPT-2 model. FYI, I was able to get the iris-classifier example deployed and working in less than 10 seconds.

[deliahu]

@wise-east There are a few things that could be causing this, do you mind checking your autoscaling group activity history by following these instructions, and seeing if there is any useful information there?

Also, were your previously-deployed APIs still running in your cluster when you deployed the one that got stuck?

One guess is that when you deployed the 2nd API, it didn't fit on your existing instance(s) (e.g. if the other API was still running), and when your cluster requested an additional instance from AWS, for some reason AWS was not able to fulfill that request. One possible reason is that you have reached your max_instances you configured when you spun up the cluster, or you have reached your AWS limits for your instance type (there is a separate limit for spot and for on-demand). The AWS limit issues would show up in the auto scaling group activity history, and you can check your cluster max_instances by running cortex cluster info.

[wise-east]

This is what I found according to the instruction you provided:

Status: Failed / Description: Launching a new EC2 instance

Description:
Launching a new EC2 instance. Status Reason: You have requested more vCPU capacity than your current vCPU limit of 4 allows for the instance bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit. Launching EC2 instance failed.

Cause:
At 2020-04-02T04:04:10Z a user request explicitly set group desired capacity changing the desired capacity from 3 to 2. At 2020-04-02T04:04:28Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 1 to 2.

My max_instances is set to 5, so I don't think that was the issue and as the message says above, it has to do with reaching the AWS limit for my instance type. I'm using a p2.xlarge instance and I can't quite get my head around why 4 CPUs were not enough for accommodating my API. Does this message mean that I should be using an instance with more CPUs, such as p2.8xlarge?

As a side note, it would be nice if there was a way to see this AWS error message through the cortex CLI or have this mentioned in the docs as something to check when deploying a model takes too long.

Also, were your previously-deployed APIs still running in your cluster when you deployed the one that got stuck? --> Yes, I had a iris-classifier API running when I added a new API, but the new API worked initially when I first deployed it while the iris-classifier was already running, but as I changed the code and tried to update the additional API, it has been held with the updating status since.

[deliahu]

@wise-east that is helpful, thank you for including that output.

it has to do with reaching the AWS limit for my instance type

Yes that is correct. The way AWS enforces EC2 limits is purely based on vCPU within an "instance family", no matter the instance type or the GPU/memory on that instance. Your current limit in the "P" family for your region seems to be 4 vCPU. This means that you can only run one p2.xlarge in your AWS account (in that region), unrelated to Cortex, since a p2.xlarge has 4 vCPU. Once your cluster already had one p2.xlarge instance running, your were at your limit of 4 vCPU, and AWS rejected Cortex's request for another instance. You can request a limit increase from the EC2 dashboard; search "P instances", select "Running On-Demand All P instances", and click "Request limit increase" in the top right. I know this can be a bit confusing... let me know if that wasn't clear and I can try to explain differently, or feel free to email me at david@cortex.dev if you'd like to set up a call to discuss further. Also here is an AWS blog post which explains the vCPU limits.

Does this message mean that I should be using an instance with more CPUs, such as p2.8xlarge?

As mentioned above, currently AWS will not allow you to run a p2.8xlarge instance, since it has 32 vCPU but your limit is 4.

As a side note, it would be nice if there was a way to see this AWS error message through the cortex CLI or have this mentioned in the docs as something to check when deploying a model takes too long.

Yes I agree 100%. We have this GitHub issue #936 to address this, and I think adding something in the docs now would go a long way, so I just added it to our list of docs changes to consider for our next release (#914). Thanks for the suggestion!

as I changed the code and tried to update the additional API, it has been held with the updating status since

This is another scenario which we'll add to the same docs you suggested above. Here is what happened: By default, cortex performs rolling updates on all APIs. Your API requested 1 GPU and was running on an instance with 1 GPU. When you update your API, cortex deploys the updated version, and only takes down the old version once the new one has deployed without error (this is to ensure that traffic can continue to be served during updates, and that there is no downtime if there's an error in the new version). In your case, since there was no GPU available on your instance, the new version of your API requested another instance to run on. Normally this will be ok (it might just take a few minutes since a new instance has to spin up): the new instance will become live, the new API replica will run on it, once it starts up successfully the old replica will be terminated, and eventually the old instance should spin down. In your case, it got stuck because you had reached your AWS account limit, and the second instance could not be created. We'll definitely add this to the "troubleshooting" section you suggested above.

It is possible to disable rolling updates in cortex: in your API configuration (e.g. in cortex.yaml), set max_surge to 0 (in the update_strategy configuration). Here are the docs. E.g.:

- name: iris-classifier
  predictor:
    type: python
    ...
  update_strategy:
    max_surge: 0

[wise-east]

Thank you so much for the detailed response, now everything makes a lot more sense and I hope I can get around these issue to make my particular use-case working. I really admire this effort in making scalable deployments easier 🥇

[deliahu]

We're happy to help any time; thanks for you suggestions, and feel free to keep the questions coming!

[deliahu]

@wise-east Also I thought of something last night but forgot to mention it in my previous response: unless there is a specific reason your model requires the p2 instances, many of our users have found that the g4dn.xlarge have better price/performance for inference. Just wanted to let you know in case it's relevant!

[wise-east]

@deliahu That is a great tip, I'll try using the g4dn.xlarge instance and compare performance. I blindly assumed the p2 instance is better given the description here: https://www.cortex.dev/cluster-management/ec2-instances. (The model I'm using is a variant of GPT-2 for text generation)

[deliahu]

@wise-east thanks for pointing that out, I see how those docs can be confusing. I've updated them on master: https://www.cortex.dev/v/master/cluster-management/ec2-instances