Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: Unable to delete image on node with containerd runtime #89

Closed
senthilrch opened this issue Jul 30, 2021 · 0 comments
Closed

Bug: Unable to delete image on node with containerd runtime #89

senthilrch opened this issue Jul 30, 2021 · 0 comments
Assignees
@senthilrch
Labels
bug Something isn't working
Milestone
v0.8.1

Comments

@senthilrch
Copy link
Owner

Pod event:-

Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    108s                default-scheduler  Successfully assigned kube-fledged/imagecache1-26ff5-m85lb to node-10-120-127-21
  Warning  FailedMount  45s (x8 over 108s)  kubelet            MountVolume.SetUp failed for volume "runtime-sock" : hostPath type check failed: /run/containerd/containerd.sock is not a file

Controller log:-

I0730 05:37:37.866026 1 image_manager.go:415] Job imagecache1-7ksk8 created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-25, runtime: containerd://1.4.4)
I0730 05:37:37.874927 1 image_manager.go:415] Job imagecache1-hxjnv created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-26, runtime: containerd://1.4.4)
I0730 05:37:37.887076 1 image_manager.go:415] Job imagecache1-pkq69 created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-26, runtime: containerd://1.4.4)
I0730 05:37:37.895511 1 image_manager.go:415] Job imagecache1-69qxk created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0730 05:37:37.899097 1 image_manager.go:415] Job imagecache1-sctsk created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0730 05:37:38.030825 1 image_manager.go:415] Job imagecache1-ss78m created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-20, runtime: containerd://1.4.4)
I0730 05:37:38.231420 1 image_manager.go:415] Job imagecache1-k7w8p created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-20, runtime: containerd://1.4.4)
I0730 05:37:38.431199 1 image_manager.go:415] Job imagecache1-6q8j8 created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-21, runtime: containerd://1.4.4)
I0730 05:37:38.630803 1 image_manager.go:415] Job imagecache1-26ff5 created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-21, runtime: containerd://1.4.4)
I0730 05:37:38.830593 1 image_manager.go:415] Job imagecache1-q82c5 created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-22, runtime: containerd://1.4.4)
I0730 05:37:39.030576 1 image_manager.go:415] Job imagecache1-6hvtv created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-22, runtime: containerd://1.4.4)
I0730 05:37:39.230910 1 image_manager.go:415] Job imagecache1-gpbfx created (delete:- quay.io/bitnami/redis:6.2.5 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0730 05:37:39.431366 1 image_manager.go:415] Job imagecache1-4bnpx created (delete:- quay.io/bitnami/mariadb:10.5.11 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0730 05:42:39.432314 1 image_manager.go:221] Job imagecache1-pkq69 expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-26)
I0730 05:42:39.432385 1 image_manager.go:221] Job imagecache1-69qxk expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-27)
I0730 05:42:39.432400 1 image_manager.go:221] Job imagecache1-sctsk expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-27)
I0730 05:42:39.432410 1 image_manager.go:221] Job imagecache1-k7w8p expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-20)
I0730 05:42:39.432427 1 image_manager.go:221] Job imagecache1-lfrtf expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-24)
I0730 05:42:39.432436 1 image_manager.go:221] Job imagecache1-7ksk8 expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-25)
I0730 05:42:39.432445 1 image_manager.go:221] Job imagecache1-ss78m expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-20)
I0730 05:42:39.432454 1 image_manager.go:221] Job imagecache1-26ff5 expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-21)
I0730 05:42:39.432463 1 image_manager.go:221] Job imagecache1-4d4mr expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-24)
I0730 05:42:39.432475 1 image_manager.go:221] Job imagecache1-k5vkd expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-23)
I0730 05:42:39.432483 1 image_manager.go:221] Job imagecache1-q9r4l expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-25)
I0730 05:42:39.432492 1 image_manager.go:221] Job imagecache1-6hvtv expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-22)
I0730 05:42:39.432500 1 image_manager.go:221] Job imagecache1-4bnpx expired (delete: quay.io/bitnami/mariadb:10.5.11 --> node-10-120-127-27)
I0730 05:42:39.432515 1 image_manager.go:221] Job imagecache1-tgnnf expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-23)
I0730 05:42:39.432524 1 image_manager.go:221] Job imagecache1-6q8j8 expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-21)
I0730 05:42:39.432533 1 image_manager.go:221] Job imagecache1-q82c5 expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-22)
I0730 05:42:39.432541 1 image_manager.go:221] Job imagecache1-gpbfx expired (delete: quay.io/bitnami/redis:6.2.5 --> node-10-120-127-27)
I0730 05:42:39.432553 1 image_manager.go:221] Job imagecache1-hxjnv expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-26)
I0730 05:42:41.045426 1 controller.go:430] Starting to sync image cache imagecache1(statusupdate)
I0730 05:42:41.096181 1 controller.go:664] Annotation kubefledged.io/purge-imagecache removed from imagecache(imagecache1)
I0730 05:42:41.096224 1 controller.go:633] Completed sync actions for image cache imagecache1(statusupdate)
I0730 05:42:41.096311 1 event.go:282] Event(v1.ObjectReference{Kind:"ImageCache", Namespace:"kube-fledged", Name:"imagecache1", UID:"c4b5579d-f075-40fe-aa99-4ebe73fefa63", APIVersion:"kubefledged.io/v1alpha2", ResourceVersion:"63611135", FieldPath:""}): type: 'Warning' reason: 'ImageCachePurge' Image deletion failed for some images. Please see "failures" section
@senthilrch senthilrch added the bug Something isn't working label Jul 30, 2021
@senthilrch senthilrch added this to the v0.8.1 milestone Jul 30, 2021
@senthilrch senthilrch self-assigned this Jul 30, 2021
@senthilrch senthilrch mentioned this issue Jul 30, 2021
Merged
senthilrch added a commit that referenced this issue Jul 30, 2021
@senthilrch
fix issue #89 change hostpath filetype to socket
d8ab2e7
@AnchorArray AnchorArray mentioned this issue Sep 16, 2021
Merged
AnchorArray added a commit to noteable-io/kube-fledged that referenced this issue Sep 16, 2021
@AnchorArray @Chili-Man @senthilrch
aberg/DEVOPS 247 (#3)
c3c3be7 
* Move custom resource definitions to crds directory

* modified travis build conditions

* Initial commit for v0.8.0

* Bumped up versions of go, alpine, operator sdk, docker, cri-tool

* upgrade go in travis ci

* update dependencies

* code changes for upgraded dependencies

* fix unit tests

* make release-amd64 to build all 4 amd64 images

* v1alpha1 -> v1alpha2, kubefledged.k8s.io -> kubefledged.io

* ignore build binaries

* add labels to manifests

* updates to imagecache manifest

* update apigroup apiversion in validatingwebhook

* issue senthilrch#70 deploy-using-yaml: wait for webhook-server running

* issue senthilrch#66: upgrade crd api version to v1

* updates to helm chart & operator

* update clusterrole to list and watch

* update signer name in csr

* remove v1alpha1 cr

* issue senthilrch#70 add init container to wait for webhook-server

* helm chart fix webhook service name

* helm chart update

* fix chart apiVersion

* updated readme for helm chart installation

* updated name of helm operator cr

* fix issue senthilrch#75: workaround

* Ensure validating webhook configuration client config service name for the webhook server mataches the correct webhook service name

* Ensure validating webhook configuration client config service name for the webhook server mataches the correct webhook service name

* add init option to webhook server

* update manifests

* updated helm chart

* updated makefile and manifests

* updated helm operator

* makefile wait for operator ready

* get imagecache before updating status

* pre-install hook for validatingwebhookconfiguration

* fix golint errors

* modify refresh/purge annotation key to kubefledged.io/xxx

* cri-client-image name as env instead of cmd flag

* read busybox image from env

* use busybox image from gcr.io to overcome dockerhub ratelimiting

* fix issue senthilrch#89 change hostpath filetype to socket

* delete pre-install hook in "make remove-operator-and-kubefledged"

* add annotations to validatingwebhookconfiguration

* add "helm repo update" to readme

* continue processing job deletion when not found

* check if refresh-cache annotation exists

* add "helm repo update" to readme

* update helm chart to use release namespace

* update design proposal document

* expose helm parameters in operator CR

* document helm parameters

* update release version to v0.8.2

* set status to known when unable to fetch pod

* add check for image pull/delete status unknown

* update log messages

* deploy controller and operator to same namespace

* fix unit test errors

* restore Kubefledged CR during "remove-operator-and-kubefledged"

* Update README.md

* Update README.md

* Update design-proposal.md

* Update README.md

Co-authored-by: Diego Rodriguez <diego@noteable.io>
Co-authored-by: Senthil Raja Chermapandian <senthilrch@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@senthilrch senthilrch

Labels
bug Something isn't working
Projects
None yet
Milestone
v0.8.1
Development

No branches or pull requests
1 participant
@senthilrch