Figure out how to delete stale containers created during deployment

[derat]

Right now, I have a "Delete object / 7+ days since object was created" lifecycle rule on the us.artifacts.<project-id>.appspot.com bucket. This seems to keep storage costs down, but I saw confusing file-not-found Cloud Build errors during deployment yesterday. Deploying worked again after I went to Container Registry in the console and manually deleted all the app-engine-tmp containers. I'm guessing that the containers contained stale references to GCS objects that had been deleted by the lifecycle rule.

I have no idea what the right way is to clean up this junk. http://www.theappliedarchitect.com/what-the-heck-are-these-cloud-storage-buckets/ describes various buckets that get created, but it claims that you can just clean them up with lifecycle rules like I'm already doing (backed up by a stale link to a GCP documentation page).

I was thinking I could maybe write a script to delete old containers via gcloud container images, but I haven't figured out how to do it. I gave my dev user the "Storage Object Admin" permission on the GCS bucket, which prevents a ERROR: (gcloud.container.images.list) Access denied error, but I don't know how to list the individual images. (Note also that you need to pass --repository=us.gcr.io/<project-id> since otherwise it'll try to query gcr.io.)

For example, I see two images listed in the console, but I don't know how to list them using the CLI:

% gcloud --project=... container images list --repository=us.gcr.io/.../app-engine-tmp/app/default
NAME
us.gcr.io/.../app-engine-tmp/app/default/ttl-18h
% gcloud --project=... container images list --repository=us.gcr.io/.../app-engine-tmp/app/default/ttl-18h
Listed 0 items.

There's a script linked from https://stackoverflow.com/a/67402286, but it seems like it's just doing the same thing that I tried.

[derat]

More links that don't really help:

• https://stackoverflow.com/questions/52176949/cannot-deploy-app-engine-python-3-after-cleanup-cloud-storage-containers-images
• https://stackoverflow.com/questions/61292914/can-i-delete-default-buckets-on-google-app-engine-storage
• https://stackoverflow.com/questions/61082741/my-gcp-project-is-automatically-creating-storage-buckets