Package and Registry Resources not Deleted upon app deletion

[drpdishant]

Deleting an app should delete all the related data and routes:

Data includes Packages stored in minio blob storage, container images in Docker Registry

I have been using cf-for-k8s to deploy development workloads. There are times developers will create delete and recreate app to troubleshoot, or to just have a fresh start.

The packages uploaded for an app are not accessible once its deleted and recreated, cf starts a whole new upload for packaged zip.

This certainly is not an issue but the package data from previous deployment of the same app isn't deleted from blob store, this is leading to serious storage issues, especially when using private object storage and registry
Even my dockerhub account is populated with so many images from the app guids that don't exist anymore.
The resources being stored and named with long app guid are difficult to identify and delete manually.

Application Lifecycle Management Must destroy all the resources, with deletion of app.

Or a cleanup task should be there to cleanup dangling resources for apps(guids) that don't exist anymore

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/178514708

The labels on this github issue will be updated when the story is started.

[drpdishant]

Update:
Following are logs from the cf-api-worker.

Request failed: 500: {"error_code"=>"UnknownError", "description"=>"An unknown error occurred.", "code"=>10001, "test_mode_info"=>{"description"=>"Server error, status: 500", "error_code"=>"CF-Error", "backtrace"=>["/workspace/lib/registry_buddy/client.rb:56:in `with_request_error_handling'", "/workspace/lib/registry_buddy/client.rb:23:in `delete_image'", "/workspace/app/jobs/kubernetes/registry_delete.rb:13:in `perform'", "/workspace/app/jobs/wrapping_job.rb:13:in `perform'", "/workspace/app/jobs/timeout_job.rb:13:in `block in perform'", "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'", "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb:33:in `block in catch'", "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb:33:in `catch'", "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb:33:in `catch'", "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb:110:in `timeout'", "/workspace/app/jobs/timeout_job.rb:12:in `perform'", "/workspace/app/jobs/wrapping_job.rb:13:in `perform'", "/workspace/app/jobs/logging_context_job.rb:17:in `block in perform'", "/workspace/app/jobs/logging_context_job.rb:67:in `with_request_id_set'", "/workspace/app/jobs/logging_context_job.rb:15:in `perform'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb:81:in `block in invoke_job'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb:78:in `invoke_job'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:230:in `block (2 levels) in run'", "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'", "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb:105:in `timeout'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:230:in `block in run'", "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:229:in `run'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:312:in `block in reserve_and_run_one_job'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:312:in `reserve_and_run_one_job'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:213:in `block in work_off'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:212:in `times'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:212:in `work_off'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:175:in `block (4 levels) in start'", "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:174:in `block (3 levels) in start'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:173:in `block (2 levels) in start'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:172:in `loop'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:172:in `block in start'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:79:in `block in add'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:171:in `start'", "/workspace/lib/tasks/jobs.rake:76:in `start_working'", "/workspace/lib/tasks/jobs.rake:49:in `block (2 levels) in <top (required)>'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/honeycomb-beeline-2.7.0/lib/honeycomb/integrations/rake.rb:14:in `execute'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/bin/rake:23:in `load'", "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/bin/rake:23:in `<top (required)>'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli/exec.rb:58:in `load'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli/exec.rb:58:in `kernel_load'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli/exec.rb:23:in `run'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli.rb:477:in `exec'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli.rb:31:in `dispatch'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli.rb:25:in `start'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/exe/bundle:49:in `block in <top (required)>'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/friendly_errors.rb:128:in `with_friendly_errors'", "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/exe/bundle:37:in `<top (required)>'", "/layers/paketo-buildpacks_bundler/bundler/bin/_bundle:23:in `load'", "/layers/paketo-buildpacks_bundler/bundler/bin/_bundle:23:in `<main>'"]}}
2021-12-10T09:52:34+0000: [Worker(cf-api-worker-6df4987d65-ddmtl)] Job VCAP::CloudController::Jobs::Kubernetes::RegistryDelete (id=9358) (queue=cc-generic) FAILED (1 prior attempts) with RegistryBuddy::Error: Server error, status: 500
2021-12-10T09:52:34+0000: [Worker(cf-api-worker-6df4987d65-ddmtl)] 1 jobs processed at 4.2840 j/s, 1 failed
2021-12-10T09:52:56+0000: [Worker(cf-api-worker-6df4987d65-v9zs4)] Job VCAP::CloudController::Jobs::Kubernetes::RegistryDelete (id=9358) (queue=cc-generic) RUNNING
about to run job VCAP::CloudController::Jobs::Kubernetes::RegistryDelete
RegistryBuddy returned: 500 with unable to delete image registry.openxcell.dev/cloudfoundry/5dca345e-36b5-42b3-904a-2eabdd4851fe:latest