Errors in pod logs on startup - failed to list deployments

[jeblackmore]

I installed the stable release with default settings on a Kubernetes cluster using Helm. Annotated my Deployments as per the instructions but I'm not seeing any rolling updates when I change a ConfigMap. When I checked the reloader pod logs I found this:

time="2019-12-13T15:46:02Z" level=info msg="Environment:Kubernetes"
time="2019-12-13T15:46:02Z" level=info msg="Starting Reloader"
time="2019-12-13T15:46:02Z" level=warning msg="KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces."
time="2019-12-13T15:46:02Z" level=info msg="Starting Controller to watch resource type: secrets"
time="2019-12-13T15:46:02Z" level=info msg="Starting Controller to watch resource type: configMaps"
time="2019-12-13T15:46:02Z" level=error msg="Failed to list deployments the server could not find the requested resource"
time="2019-12-13T15:46:02Z" level=error msg="Failed to list daemonSets the server could not find the requested resource"
time="2019-12-13T15:46:02Z" level=error msg="Failed to list statefulSets the server could not find the requested resource"

Then the last 3 lines just repeat periodically.

I'm wondering if its an RBAC issue, but the ClusterRole and ClusterRoleBinding seem to be there.

Any help would be greatly appreciated.

[aliartiza75]

Hi, @jeblackmore,

Kindly share you helm manifest.

[jeblackmore]

Hi.

I haven't created a Helm manifest, I am just doing:

helm install stable/reloader

on the command line.

[aliartiza75]

@jeblackmore Kindly share your deployment manifest.

[jeblackmore]

Do you mean the output of

helm get manifest reloader

?

[aliartiza75]

no, I want the manifest of one of your deployments in which you added the reloader specific annotations.

[jeblackmore]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "core-demo.name" . }}
  labels:
{{ include "core-demo.labels" . | indent 4 }}
  annotations:
    configmap.reloader.stakater.com/reload: "{{ include "core-demo.name" . }}-configmap"
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ include "core-demo.name" . }}
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ include "core-demo.name" . }}
        app.kubernetes.io/instance: {{ .Release.Name }}
    spec:
    {{- with .Values.global.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      serviceAccountName: {{ template "core-demo.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          envFrom:
          - configMapRef:
              name: {{ include "core-demo.name" . }}-configmap
#          livenessProbe:
#            httpGet:
#              path: /
#              port: http
          readinessProbe:
            httpGet:
              path: /actuator/health
              port: http
            initialDelaySeconds: 30
            timeoutSeconds: 5
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
    {{- end }}

[jeblackmore]

With the corresponding ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "core-demo.name" . }}-configmap
  labels:
    app: {{ include "core-demo.name" . }}
data:
  SPRING_APPLICATION_JSON: |-
    {
      "management": {
        "endpoints": {
          "web": {
            "exposure": {
              "include": "{{ .Values.actuatorEndpoints }}"
            }
          }
        }
      }
    }

[jeblackmore]

I assumed that the error messages in the Pod log indicated that something isn't configured correctly with Reloader rather than with my deployments. Does Reloader just output those errors if it doesn't find any annotated deployments?

[aliartiza75]

@jeblackmore the errors show that it is not able to find the deployments in your cluster. I will try to replicate the scenario to find out the exact issue.

what is the version of your k8s cluster?

[jeblackmore]

Thanks. Its v1.16.2

[jeblackmore]

Hi. Any luck reproducing it?

[rasheedamir]

@aliartiza75 can you plz take a look?

[aliartiza75]

@jeblackmore. What is the helm version you are currently using?

[waseem-h]

@jeblackmore The upstream stable helm chart repo is not updated to the latest version and it doesn't support kubernetes 1.16.x. You need to install reloader from our chart repository by running the following(Make sure that you have uninstalled the previous version):

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install stakater/reloader

Closing it for now. Feel free to re-open if you still see the issue in the latest version

[jeblackmore]

Many thanks @waseem-h, that fixes it for me.