Cannot clean up model registry, after undeploy (and gratitious sleep)

[cjmcgraw]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

• Search the issues already opened: https://github.com/googleapis/python-aiplatform/issues
• Search StackOverflow: https://stackoverflow.com/questions/tagged/google-cloud-platform+python

If you are still having issues, please be sure to include as much information as possible:

Environment details

• OS type and version:
• Python version: python --version
• pip version: pip --version
• google-cloud-aiplatform version: pip show google-cloud-aiplatform

Steps to reproduce

1. Deploy model to endpoint with SDK
2. Undeploy model from endpoint and then delete

Code example

old_model_id = ...
print("undeploying model")
endpoint.undeploy(
    deployed_model_id=deployed_model.id,
    sync=True
)
print("successfully undeployed model!")
print("waiting for undeployed model to release resources")
time.sleep(60)
print("deleting model version", old_model_id)
model = aiplatform.Model(old_model_id)
model.wait()
model.delete(sync=True)
print("finished deleting model", old_model_id)

Stack trace

google.api_core.exceptions.FailedPrecondition: 400 The model "<...>" can't be deleted because it's deployed or being deployed at the following endpoint(s): <...> Undeploy the model from all endpoints first and then delete it

Unsure if this is because of some caching layer or something?

[sasha-gitg]

Please confirm the endpoint referenced in the exception, <...> Undeploy the model from all endpoints first and then delete it, is the same resource as the referenced Endpoint when undeploying earlier in the code block.

If it is, check that the Model hasn't been deployed multiple times to the same Endpoint. Endpoint.list_models()