Skip to content
This repository has been archived by the owner on Dec 15, 2021. It is now read-only.

Starting kubeless | kubeless-controller-manager in CrashLoopBackOff #1152

Open
iamshreeram opened this issue Jul 19, 2020 · 14 comments
Open

Comments

@iamshreeram
Copy link

iamshreeram commented Jul 19, 2020

Is this a BUG REPORT or FEATURE REQUEST?:
BUG

What happened:
Installed kubeless with below

kube-shell> kubectl create ns kubeless
namespace/kubeless created
kube-shell> kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml
configmap/kubeless-config created
deployment.apps/kubeless-controller-manager created
serviceaccount/controller-acct created
customresourcedefinition.apiextensions.k8s.io/functions.kubeless.io created
customresourcedefinition.apiextensions.k8s.io/httptriggers.kubeless.io created
customresourcedefinition.apiextensions.k8s.io/cronjobtriggers.kubeless.io created
kube-shell> kubectl --namespace kubeless get pods
NAME                                          READY   STATUS             RESTARTS   AGE
kubeless-controller-manager-67d484f5d-xf2mj   0/3     CrashLoopBackOff   9          107s

What you expected to happen:
Kubeless Pods being created

How to reproduce it (as minimally and precisely as possible):

kubectl create ns kubeless
kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
Client Version: v1.16.6-beta.0
Server Version: v1.18.3
  • Kubeless version (use kubeless version): v1.0.7-dirty
  • Cloud provider or physical cluster: Local Mac

Pod description :

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  2m25s                  default-scheduler  Successfully assigned kubeless/kubeless-controller-manager-67d484f5d-xf2mj  to minikube
  Normal   Created    2m22s (x2 over 2m23s)  kubelet, minikube  Created container http-trigger-controller
  Normal   Created    2m22s (x2 over 2m23s)  kubelet, minikube  Created container kubeless-function-controller
  Normal   Started    2m22s (x2 over 2m23s)  kubelet, minikube  Started container kubeless-function-controller
  Normal   Pulled     2m22s (x2 over 2m23s)  kubelet, minikube  Container image "kubeless/http-trigger-controller:v1.0.1" already present on machine
  Normal   Started    2m21s (x2 over 2m23s)  kubelet, minikube  Started container http-trigger-controller
  Normal   Started    2m21s (x2 over 2m22s)  kubelet, minikube  Started container cronjob-trigger-controller
  Normal   Pulled     2m21s (x2 over 2m23s)  kubelet, minikube  Container image "kubeless/cronjob-trigger-controller:v1.0.1" already present on machine
  Normal   Created    2m21s (x2 over 2m23s)  kubelet, minikube  Created container cronjob-trigger-controller
  Warning  BackOff    2m20s (x2 over 2m21s)  kubelet, minikube  Back-off restarting failed container
  Warning  BackOff    2m20s (x2 over 2m21s)  kubelet, minikube  Back-off restarting failed container
  Warning  BackOff    2m20s (x2 over 2m21s)  kubelet, minikube  Back-off restarting failed container
  Normal   Pulled     2m8s (x3 over 2m23s)   kubelet, minikube  Container image "kubeless/function-controller:v1.0.7" already present on machine
@andresmgot
Copy link
Contributor

I @iamshreeram, can you get the logs of the container failing? First you need to get which one of the 3 containers is failing (you can that info from the pod description), then get the logs with kubectl logs -n kubeless kubeless-controller-manager-67d484f5d-xf2mj -c <container>. That should give more info about what's gone wrong.

@iamshreeram
Copy link
Author

@andresmgot , Thanks for looking at the issue.

Below is the error for all 3 containers [kubeless-function-controller, http-trigger-controller, cronjob-trigger-controller] in all 3 pods :

time="2020-07-20T12:17:04Z" level=info msg="Running Kubeless controller manager version: v1.0.7"
time="2020-07-20T12:17:04Z" level=fatal msg="Unable to read the configmap: Error while fetching config location: customresourcedefinitions.apiextensions.k8s.io \"functions.kubeless.io\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"

@andresmgot
Copy link
Contributor

ah, I see the problem. You are using the manifest that doesn't include the RBAC configuration (and RBAC is now always enabled by default). Install Kubeless like this:

kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-v1.0.7.yaml

(rather than using the non-rbac version)

@iamshreeram
Copy link
Author

@andresmgot , Thanks for response.

Does this mean users can never use non-rbac - kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml? If so, can this be removed from documentation?

@andresmgot
Copy link
Contributor

While it's not usual, people can disable RBAC in their clusters. If they do so, the manifest with RBAC rules will fail, that's why we offer the two options. In the documentation, we point to the other manifest but if you think of another way of making it more prominent please send a PR to help others:

https://github.com/kubeless/kubeless/blob/master/docs/quick-start.md

@iamshreeram
Copy link
Author

@andresmgot , I think we can just remove - kubeless-non-rbac-$RELEASE.yaml is used for non-RBAC Kubernetes cluster. from the doc and provide documentation to disable RBAC.

Let me know. thanks!

@andresmgot
Copy link
Contributor

Yes, we can do that. I don't think that file is used very often.

@fc7
Copy link

fc7 commented Jan 8, 2021

I had the same issue, and it took me a while until I found this page. I also vote for the above proposal.

@shubhomoy
Copy link

I am having the same issue but not due to RBAC

kubeless-controller-manager-5d9bf7f5c6-hs7s5   2/3     CrashLoopBackOff   5          3m30s

The cronjob-trigger-controller container is crashing. Following with following the logs

time="2021-05-02T20:30:09Z" level=info msg="Running Kubeless cronjob trigger controller version: v1.0.3"
time="2021-05-02T20:30:09Z" level=fatal msg="Unable to read the configmap: Error while fetching config location: v1beta1.CustomResourceDefinition: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 535 ...s\":{\"k:{\\\"... at {\"kind\":\"CustomResourceDefinition\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"metadata\":{\"name\":\"functions.kubeless.io\",\"uid\":\"1fb634d2-032e-472b-a8c7-10edd69e96f8\",\"resourceVersion\":\"582946\",\"generation\":1,\"creationTimestamp\":\"2021-05-02T20:29:48Z\",\"managedFields\":[{\"manager\":\"kube-apiserver\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1\",\"time\":\"2021-05-02T20:29:48Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:acceptedNames\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:conditions\":{\"k:{\\\"type\\\":\\\"Established\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NamesAccepted\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NonStructuralSchema\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}}}}},{\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"time\":\"2021-05-02T20:29:48Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:conversion\":{\".\":{},\"f:strategy\":{}},\"f:group\":{},\"f:names\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:preserveUnknownFields\":{},\"f:scope\":{},\"f:version\":{},\"f:versions\":{}}}}]},\"spec\":{\"group\":\"kubeless.io\",\"version\":\"v1beta1\",\"names\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1beta1\",\"served\":true,\"storage\":true}],\"conversion\":{\"strategy\":\"None\"},\"preserveUnknownFields\":true},\"status\":{\"conditions\":[{\"type\":\"NamesAccepted\",\"status\":\"True\",\"lastTransitionTime\":\"2021-05-02T20:29:48Z\",\"reason\":\"NoConflicts\",\"message\":\"no conflicts found\"},{\"type\":\"Established\",\"status\":\"True\",\"lastTransitionTime\":\"2021-05-02T20:29:48Z\",\"reason\":\"InitialNamesAccepted\",\"message\":\"the initial names have been accepted\"},{\"type\":\"NonStructuralSchema\",\"status\":\"True\",\"lastTransitionTime\":\"2021-05-02T20:29:48Z\",\"reason\":\"Violations\",\"message\":\"spec.preserveUnknownFields: Invalid value: true: must be false\"}],\"acceptedNames\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"storedVersions\":[\"v1beta1\"]}}"

@andresmgot
Copy link
Contributor

@shubhomoy Kubernetes +1.18 is not supported for the cronjob trigger so that may be the reason. See #1130

@eduard93
Copy link

eduard93 commented Jul 11, 2021

I got the same error as @shubhomoy on:

kubeless version
Kubeless version: v1.0.8

kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

The error (parsing position is slightly different, but the error is the same):

kubectl logs kubeless-controller-manager-5d9bf7f5c6-sqhvg -n kubeless -c cronjob-trigger-controller
time="2021-07-11T19:53:44Z" level=info msg="Running Kubeless cronjob trigger controller version: v1.0.3"
time="2021-07-11T19:53:44Z" level=fatal msg="Unable to read the configmap: Error while fetching config location: v1beta1.CustomResourceDefinition: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 901 ...s\":{\"k:{\\\"... at {\"kind\":\"CustomResourceDefinition\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"metadata\":{\"name\":\"functions.kubeless.io\",\"uid\":\"f0b89e5a-0ca7-4db2-b133-f65a50fc2566\",\"resourceVersion\":\"2620\",\"generation\":1,\"creationTimestamp\":\"2021-07-11T19:36:54Z\",\"managedFields\":[{\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"time\":\"2021-07-11T19:36:54Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:conversion\":{\".\":{},\"f:strategy\":{}},\"f:group\":{},\"f:names\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:preserveUnknownFields\":{},\"f:scope\":{},\"f:version\":{},\"f:versions\":{}}}},{\"manager\":\"kube-apiserver\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1\",\"time\":\"2021-07-11T19:36:55Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:acceptedNames\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:conditions\":{\"k:{\\\"type\\\":\\\"Established\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NamesAccepted\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NonStructuralSchema\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}}}}}]},\"spec\":{\"group\":\"kubeless.io\",\"version\":\"v1beta1\",\"names\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1beta1\",\"served\":true,\"storage\":true}],\"conversion\":{\"strategy\":\"None\"},\"preserveUnknownFields\":true},\"status\":{\"conditions\":[{\"type\":\"NonStructuralSchema\",\"status\":\"True\",\"lastTransitionTime\":\"2021-07-11T19:36:54Z\",\"reason\":\"Violations\",\"message\":\"spec.preserveUnknownFields: Invalid value: true: must be false\"},{\"type\":\"NamesAccepted\",\"status\":\"True\",\"lastTransitionTime\":\"2021-07-11T19:36:54Z\",\"reason\":\"NoConflicts\",\"message\":\"no conflicts found\"},{\"type\":\"Established\",\"status\":\"True\",\"lastTransitionTime\":\"2021-07-11T19:36:55Z\",\"reason\":\"InitialNamesAccepted\",\"message\":\"the initial names have been accepted\"}],\"acceptedNames\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"storedVersions\":[\"v1beta1\"]}}"
Error pretty print
{
 "kind":"CustomResourceDefinition",
 "apiVersion":"apiextensions.k8s.io/v1beta1",
 "metadata":{
    "name":"functions.kubeless.io",
    "uid":"f0b89e5a-0ca7-4db2-b133-f65a50fc2566",
    "resourceVersion":"2620",
    "generation":1,
    "creationTimestamp":"2021-07-11T19:36:54Z",
    "managedFields":[
       {
          "manager":"kubectl-create",
          "operation":"Update",
          "apiVersion":"apiextensions.k8s.io/v1beta1",
          "time":"2021-07-11T19:36:54Z",
          "fieldsType":"FieldsV1",
          "fieldsV1":{
             "f:spec":{
                "f:conversion":{
                   ".":{
                      
                   },
                   "f:strategy":{
                      
                   }
                },
                "f:group":{
                   
                },
                "f:names":{
                   "f:kind":{
                      
                   },
                   "f:listKind":{
                      
                   },
                   "f:plural":{
                      
                   },
                   "f:singular":{
                      
                   }
                },
                "f:preserveUnknownFields":{
                   
                },
                "f:scope":{
                   
                },
                "f:version":{
                   
                },
                "f:versions":{
                   
                }
             }
          }
       },
       {
          "manager":"kube-apiserver",
          "operation":"Update",
          "apiVersion":"apiextensions.k8s.io/v1",
          "time":"2021-07-11T19:36:55Z",
          "fieldsType":"FieldsV1",
          "fieldsV1":{
             "f:status":{
                "f:acceptedNames":{
                   "f:kind":{
                      
                   },
                   "f:listKind":{
                      
                   },
                   "f:plural":{
                      
                   },
                   "f:singular":{
                      
                   }
                },
                "f:conditions":{
                   "k:{""type"":""Established""}":{
                      ".":{
                         
                      },
                      "f:lastTransitionTime":{
                         
                      },
                      "f:message":{
                         
                      },
                      "f:reason":{
                         
                      },
                      "f:status":{
                         
                      },
                      "f:type":{
                         
                      }
                   },
                   "k:{""type"":""NamesAccepted""}":{
                      ".":{
                         
                      },
                      "f:lastTransitionTime":{
                         
                      },
                      "f:message":{
                         
                      },
                      "f:reason":{
                         
                      },
                      "f:status":{
                         
                      },
                      "f:type":{
                         
                      }
                   },
                   "k:{""type"":""NonStructuralSchema""}":{
                      ".":{
                         
                      },
                      "f:lastTransitionTime":{
                         
                      },
                      "f:message":{
                         
                      },
                      "f:reason":{
                         
                      },
                      "f:status":{
                         
                      },
                      "f:type":{
                         
                      }
                   }
                }
             }
          }
       }
    ]
 },
 "spec":{
    "group":"kubeless.io",
    "version":"v1beta1",
    "names":{
       "plural":"functions",
       "singular":"function",
       "kind":"Function",
       "listKind":"FunctionList"
    },
    "scope":"Namespaced",
    "versions":[
       {
          "name":"v1beta1",
          "served":true,
          "storage":true
       }
    ],
    "conversion":{
       "strategy":"None"
    },
    "preserveUnknownFields":true
 },
 "status":{
    "conditions":[
       {
          "type":"NonStructuralSchema",
          "status":"True",
          "lastTransitionTime":"2021-07-11T19:36:54Z",
          "reason":"Violations",
          "message":"spec.preserveUnknownFields: Invalid value: true: must be false"
       },
       {
          "type":"NamesAccepted",
          "status":"True",
          "lastTransitionTime":"2021-07-11T19:36:54Z",
          "reason":"NoConflicts",
          "message":"no conflicts found"
       },
       {
          "type":"Established",
          "status":"True",
          "lastTransitionTime":"2021-07-11T19:36:55Z",
          "reason":"InitialNamesAccepted",
          "message":"the initial names have been accepted"
       }
    ],
    "acceptedNames":{
       "plural":"functions",
       "singular":"function",
       "kind":"Function",
       "listKind":"FunctionList"
    },
    "storedVersions":[
       "v1beta1"
    ]
 }
}

I have tried to add preserveUnknownFields: false at the end of kubeless-v1.0.8.yaml:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: cronjobtriggers.kubeless.io
spec:
  group: kubeless.io
  names:
    kind: CronJobTrigger
    plural: cronjobtriggers
    singular: cronjobtrigger
  scope: Namespaced
  version: v1beta1
  preserveUnknownFields: false

But it causes:

The CustomResourceDefinition "cronjobtriggers.kubeless.io" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: because otherwise all fields are pruned

Which makes sense. Can someone please point me, where are the object which openAPIV3Schema must describe is defined/located? I can write a schema from that and try to run kubeless with that.

Another question - can cronjobtriggers be safely removed? I'm only interested in http and cli triggers.

Links:

@eduard93
Copy link

eduard93 commented Jul 12, 2021

Trivial fix (preserveUnknownFields: false) didn't work, the same error persists.

New yaml for cronjobtriggers.kubeless.io
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: cronjobtriggers.kubeless.io
spec:
  group: kubeless.io
  names:
    kind: CronJobTrigger
    plural: cronjobtriggers
    singular: cronjobtrigger
  scope: Namespaced
  version: v1beta1
  versions: 
  - name: v1beta1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              schedule:
                type: string
              jobTemplate:
                type: object
  - name: v1
    served: false
    storage: false
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              schedule:
                type: string
              jobTemplateA:
                type: object
  preserveUnknownFields: false

@huangjiasingle
Copy link

huangjiasingle commented Jul 25, 2021

@eduard93 @eduard93 @shubhomoy @iamshreeram the reason of the crashbackoff is cronjob-trigger-controller version v1.0.3 dependence kubeless version define as github.com/kubeless/kubeless v1.0.0-alpha.6, the kubeless v1.0.0-alpha.6 dependece client-go version was very low (k8s.io/client-go v5.0.0+incompatible), the decode interface not compatible the high version kubernetes crd resource.

how to solve this problem?

rebuild the cronjob-trigger-controller: git clone https://github.com/kubeless/cronjob-trigger.git && cd cronjob-trigger, and the modify the go.mod, change the github.com/kubeless/kubeless v1.0.0-alpha.6 to github.com/kubeless/kubeless v1.0.8, the v1.0.8 is the version which you to use kubeless’s version , and the go mod tidy , rebuild the cronjob-trigger-controller image, that will be succedded.

@SCLogo
Copy link

SCLogo commented Dec 1, 2021

@eduard93 @eduard93 @shubhomoy @iamshreeram the reason of the crashbackoff is cronjob-trigger-controller version v1.0.3 dependence kubeless version define as github.com/kubeless/kubeless v1.0.0-alpha.6, the kubeless v1.0.0-alpha.6 dependece client-go version was very low (k8s.io/client-go v5.0.0+incompatible), the decode interface not compatible the high version kubernetes crd resource.

how to solve this problem?

rebuild the cronjob-trigger-controller: git clone https://github.com/kubeless/cronjob-trigger.git && cd cronjob-trigger, and the modify the go.mod, change the github.com/kubeless/kubeless v1.0.0-alpha.6 to github.com/kubeless/kubeless v1.0.8, the v1.0.8 is the version which you to use kubeless’s version , and the go mod tidy , rebuild the cronjob-trigger-controller image, that will be succedded.

Did you manage to work kubeless with this modification on k8s > 1.18? I am trying to move our clusters to 1.22

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants