Failed to put CRD error, frameworkcontroller frequently restarts

[mf-giwoong-lee]

I set up kubernetes system with kubeadm!

I have one master node and one worker node.

I use calico as pod-network.

There are some problem when setting up frameworkcontroller.

Frameworkcontroller restarts frequently (every 1min) and I found this message when typing kubectl logs frameworkcontroller-0

I1028 02:01:51.888234      10 controller.go:207] Initializing frameworkcontroller
I1028 02:01:51.888637      10 controller.go:210] With Config:
kubeApiServerAddress: https://localhost:40443
kubeConfigFilePath: ""
kubeClientQps: 200
kubeClientBurst: 300
workerNumber: 500
largeFrameworkCompression: true
crdEstablishedCheckIntervalSec: 1
crdEstablishedCheckTimeoutSec: 60
objectLocalCacheCreationTimeoutSec: 300
frameworkCompletedRetainSec: 2592000
frameworkMinRetryDelaySecForTransientConflictFailed: 60
frameworkMaxRetryDelaySecForTransientConflictFailed: 900
logObjectSnapshot:
  framework:
    onFrameworkRetry: true
    onFrameworkDeletion: true
  task:
    onTaskRetry: true
    onTaskDeletion: true
  pod:
    onPodDeletion: true
podFailureSpec: []
I1028 02:01:51.889430      10 controller.go:427] Recovering frameworkcontroller
E1028 02:02:21.890073      10 runtime.go:69] Observed a panic: &errors.errorString{s:"Failed to put CRD: Get https://localhost:40443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/frameworks.fr
ameworkcontroller.microsoft.com: dial tcp: i/o timeout"} (Failed to put CRD: Get https://localhost:40443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/frameworks.frameworkcontroller.microsoft.
com: dial tcp: i/o timeout)
/go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/go/src/github.com/microsoft/frameworkcontroller/pkg/internal/utils.go:66
/go/src/github.com/microsoft/frameworkcontroller/pkg/controller/controller.go:428
/go/src/github.com/microsoft/frameworkcontroller/cmd/frameworkcontroller/main.go:35
/usr/local/go/src/runtime/proc.go:200
/usr/local/go/src/runtime/asm_amd64.s:1337
E1028 02:02:21.890143      10 panic.go:522] Stopping frameworkcontroller
panic: Failed to put CRD: Get https://localhost:40443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/frameworks.frameworkcontroller.microsoft.com: dial tcp: i/o timeout [recovered]
        panic: Failed to put CRD: Get https://localhost:40443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/frameworks.frameworkcontroller.microsoft.com: dial tcp: i/o timeout

goroutine 1 [running]:
github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x11f84e0, 0xc0003f8110)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/microsoft/frameworkcontroller/pkg/internal.PutCRD(0xc000338960, 0xc0003438c0, 0xc000047590, 0xc000047598)
        /go/src/github.com/microsoft/frameworkcontroller/pkg/internal/utils.go:66 +0x173
github.com/microsoft/frameworkcontroller/pkg/controller.(*FrameworkController).Run(0xc000107290, 0xc0000e4960)
        /go/src/github.com/microsoft/frameworkcontroller/pkg/controller/controller.go:428 +0x157
main.main()
        /go/src/github.com/microsoft/frameworkcontroller/cmd/frameworkcontroller/main.go:35 +0x47

I use the example yaml described in frameworkcontroller guideline https://github.com/Microsoft/frameworkcontroller/tree/master/example/run

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: frameworkcontroller
  namespace: default
spec:
  serviceName: frameworkcontroller
  selector:
    matchLabels:
      app: frameworkcontroller
  replicas: 1
  template:
    metadata:
      labels:
        app: frameworkcontroller
    spec:
      # Using the ServiceAccount with granted permission
      # if the k8s cluster enforces authorization.
      serviceAccountName: frameworkcontroller
      containers:
      - name: frameworkcontroller
        image: frameworkcontroller/frameworkcontroller
        # Using k8s inClusterConfig, so usually, no need to specify
        # KUBE_APISERVER_ADDRESS or KUBECONFIG
        env:
        - name: KUBE_APISERVER_ADDRESS
          value: https://localhost:40443 #{http[s]://host:port}
        #- name: KUBECONFIG
        #  value: {Pod Local KubeConfig File Path}
        command: [
          "bash", "-c",
          "cp /frameworkcontroller-config/frameworkcontroller.yaml . &&
          ./start.sh"]
        volumeMounts:
        - name: frameworkcontroller-config
          mountPath: /frameworkcontroller-config
      volumes:
      - name: frameworkcontroller-config
        configMap:
          name: frameworkcontroller-config

Additionally frameworkbarrier cannot find the custom resource frameworks, my guessing is that frameworkcontroller doesn't work well so it cannot build the custom resource frameworks.

I also upload my entire script for launching frameworkcontroller!

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta4/nvidia-device-plugin.yml

sleep 10

kubectl create serviceaccount frameworkcontroller --namespace default
kubectl create clusterrolebinding frameworkcontroller \
  --clusterrole=cluster-admin \
  --user=system:serviceaccount:default:frameworkcontroller

sleep 5

#kubectl create -f frameworkcontroller-with-default-config.yaml

# custom config
kubectl create -f frameworkcontroller-customized-config.yaml
kubectl create -f frameworkcontroller-with-customized-config.yaml

sleep 15

kubectl create serviceaccount frameworkbarrier --namespace default
kubectl create clusterrole frameworkbarrier --verb=get,list,watch --resource=frameworks
kubectl create clusterrolebinding frameworkbarrier --clusterrole=frameworkbarrier --user=system:serviceaccount:default:frameworkbarrier

[yqwang-ms]

Looks like your cluster network issue, the http call from pod to apiserver is timeout:
Get https://localhost:40443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/frameworks.frameworkcontroller.microsoft.com: dial tcp: i/o timeout

[mf-giwoong-lee]

Yeah, I know that the cause of the problem is timeout. But I don't know why timeout is occur.

I start api-server with apiserver-bind-port 40443.

[yqwang-ms]

If so, maybe you can find help by searching "apiserver dial tcp: i/o timeout", such as:
kubernetes-sigs/metrics-server#375 (comment)
https://stackoverflow.com/questions/60420546/how-do-i-fix-a-dial-tcp-10-96-0-1443-i-o-timeout-error-for-operator-pod-instal
etc.

Or create issue in k8s repro.

And I will close this one for FC.

[mf-giwoong-lee]

I1028 08:15:50.855129       9 controller.go:207] Initializing frameworkcontroller
I1028 08:15:50.856482       9 controller.go:210] With Config:
kubeApiServerAddress: ""
kubeConfigFilePath: ""
kubeClientQps: 80
kubeClientBurst: 120
workerNumber: 200
largeFrameworkCompression: false
crdEstablishedCheckIntervalSec: 1
crdEstablishedCheckTimeoutSec: 60
objectLocalCacheCreationTimeoutSec: 300
frameworkCompletedRetainSec: 2592000
frameworkMinRetryDelaySecForTransientConflictFailed: 60
frameworkMaxRetryDelaySecForTransientConflictFailed: 900
logObjectSnapshot:
  framework:
    onFrameworkRetry: true
    onFrameworkDeletion: true
  task:
    onTaskRetry: true
    onTaskDeletion: true
  pod:
    onPodDeletion: true
podFailureSpec:
- code: -1000
  phrase: PodEvicted
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^Evicted$
    messageRegex: (?ms).*
- code: -1001
  phrase: PodNodeLost
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^NodeLost$
    messageRegex: (?ms).*
- code: -1002
  phrase: PodScheduledToInsufficientResourceNode
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^OutOf\S+$
    messageRegex: (?ms).*
- code: -1003
  phrase: PodPreemptedForCriticalPod
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^Preempting$
    messageRegex: (?ms).*
- code: -1004
  phrase: PodDeadlineExceeded
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - reasonRegex: (?i)^DeadlineExceeded$
    messageRegex: (?ms).*
- code: -1005
  phrase: PodNodeAdmissionForbidden
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - reasonRegex: (?i)^Forbidden$
    messageRegex: (?ms).*
- code: -1006
  phrase: PodNodeAdmissionUnexpectedError
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^UnexpectedAdmissionError$
    messageRegex: (?ms).*
  - reasonRegex: (?i)^UnknownReason$
    messageRegex: (?ms).*
  - reasonRegex: (?i)^InvalidNodeInfo$
    messageRegex: (?ms).*
  - reasonRegex: (?i)^UnexpectedPredicateFailureType$
    messageRegex: (?ms).*
- code: -1200
  phrase: ContainerDockerOOMKilled
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^OOMKilled$
      messageRegex: (?ms).*
      codeRange:
        min: 1
- code: -1201
  phrase: ContainerDockerCreationError
  type:
    name: Failed
    attributes: []
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?ms).*
      codeRange:
        min: 125
        max: 125
- code: -1202
  phrase: ContainerDockerStartCmdPermissionDenied
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?ms).*
      codeRange:
        min: 126
        max: 126
- code: -1203
  phrase: ContainerDockerStartCmdNotFound
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?ms).*
      codeRange:
        min: 127
        max: 127
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?msi).*(not found|cannot find|no such).*
      codeRange:
        min: 128
        max: 128
- code: -1204
  phrase: ContainerDockerStartUnknownError
  type:
    name: Failed
    attributes: []
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?ms).*
      codeRange:
        min: 128
        max: 128
- code: 130
  phrase: ContainerSigIntReceived
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 130
        max: 130
- code: 131
  phrase: ContainerSigQuitReceived
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 131
        max: 131
- code: 132
  phrase: ContainerSigIllReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 132
        max: 132
- code: 134
  phrase: ContainerSigAbrtReceived
  type:
    name: Failed
    attributes: []
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 134
        max: 134
- code: 135
  phrase: ContainerSigBusReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 135
        max: 135
- code: 136
  phrase: ContainerSigFpeReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 136
        max: 136
- code: 137
  phrase: ContainerSigKillReceived
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 137
        max: 137
- code: 139
  phrase: ContainerSigSegvReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 139
        max: 139
- code: 141
  phrase: ContainerSigPipeReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 141
        max: 141
- code: 143
  phrase: ContainerSigTermReceived
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 143
        max: 143
W1028 08:15:50.856507       9 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1028 08:15:50.857270       9 controller.go:427] Recovering frameworkcontroller
I1028 08:15:50.869345       9 utils.go:108] Create CRD frameworks.frameworkcontroller.microsoft.com
E1028 08:15:50.870802       9 runtime.go:69] Observed a panic: &errors.errorString{s:"Failed to put CRD: the server could not find the requested resource"} (Failed to put CRD: the server could not find the requested resource)
/go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/go/src/github.com/microsoft/frameworkcontroller/pkg/internal/utils.go:66
/go/src/github.com/microsoft/frameworkcontroller/pkg/controller/controller.go:428
/go/src/github.com/microsoft/frameworkcontroller/cmd/frameworkcontroller/main.go:35
/usr/local/go/src/runtime/proc.go:200
/usr/local/go/src/runtime/asm_amd64.s:1337
E1028 08:15:50.870814       9 panic.go:522] Stopping frameworkcontroller
panic: Failed to put CRD: the server could not find the requested resource [recovered]
        panic: Failed to put CRD: the server could not find the requested resource

goroutine 1 [running]:
github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x11f84e0, 0xc0003ee2b0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/microsoft/frameworkcontroller/pkg/internal.PutCRD(0xc000306f00, 0xc000351080, 0xc0003530a8, 0xc0003530b0)
        /go/src/github.com/microsoft/frameworkcontroller/pkg/internal/utils.go:66 +0x173
github.com/microsoft/frameworkcontroller/pkg/controller.(*FrameworkController).Run(0xc0000c73f0, 0xc0000ad140)
        /go/src/github.com/microsoft/frameworkcontroller/pkg/controller/controller.go:428 +0x157
main.main()
        /go/src/github.com/microsoft/frameworkcontroller/cmd/frameworkcontroller/main.go:35 +0x47

I apply canal as pod-network instead of calico

But I got above error message...

What is the resource describe in the message?

[yqwang-ms]

Check if you can get "https://{ApiServer Address}/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/". FC needs it to work.