Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Permission denied error in the setenv container #175

Closed
Evantage-WS opened this issue Nov 29, 2021 · 15 comments · Fixed by #181
Closed

Permission denied error in the setenv container #175

Evantage-WS opened this issue Nov 29, 2021 · 15 comments · Fixed by #181

Comments

@Evantage-WS
Copy link

Hi,

Testing this on a minikube cluster, I do see an error in the setenv container

+ crictl info --output go-template --template '{{.config.containerdRootDir}}'
/bin/sh: crictl: Permission denied
+ echo 'CONTAINER_ROOT_DIR='

minikube was started with:

minikube start -p k8s-test \
  --cpus 4 \
  --memory 6144 \
  --container-runtime=containerd

Any idea how to fix this?
@wenchajun
Copy link
Member

wenchajun commented Nov 29, 2021 •
edited

You can run the following command

 whereis crictl

Then modify the path https://github.com/fluent/fluentbit-operator/blob/master/charts/fluentbit-operator/templates/fluentbit-operator-deployment.yaml#L49 ,
change to the path you got
For example, You can run the following command to modify

kubectl edit deploy fluentbit-operator -n kubesphere-logging-system

@Evantage-WS
Copy link
Author

Hi @wenchajun,

Thanks, but my crictl is in the path from the helm-chart:

# whereis crictl
crictl: /usr/bin/crictl /etc/crictl.yaml /usr/local/bin/crictl

I think the problem is that it should be run with higher priviledges.

@wenchajun
Copy link
Member

Could you show me your deployment yaml?

kubectl edit deploy fluentbit-operator -n kubesphere-logging-system

@Evantage-WS
Copy link
Author 

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: fluentbit-operator
    meta.helm.sh/release-namespace: kubesphere-logging-system
  labels:
    app.kubernetes.io/component: operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluentbit-operator
  name: fluentbit-operator
  namespace: kubesphere-logging-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: operator
      app.kubernetes.io/name: fluentbit-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: operator
        app.kubernetes.io/name: fluentbit-operator
    spec:
      containers:
      - image: kubesphere/fluentbit-operator:v0.12.0
        imagePullPolicy: IfNotPresent
        name: fluentbit-operator
        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /fluentbit-operator
          name: env
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - /bin/sh
        - -c
        - set -ex; echo CONTAINER_ROOT_DIR=$(crictl info --output go-template --template
          '{{.config.containerdRootDir}}') > /fluentbit-operator/fluent-bit.env
        image: docker:19.03
        imagePullPolicy: IfNotPresent
        name: setenv
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /fluentbit-operator
          name: env
        - mountPath: /var/run/containerd/containerd.sock
          name: containerdsock
          readOnly: true
        - mountPath: /usr/local/bin/crictl
          name: crictl
          readOnly: true
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: fluentbit-operator
      serviceAccountName: fluentbit-operator
      terminationGracePeriodSeconds: 30
      volumes:
      - name: env
      - hostPath:
          path: /var/run/containerd/containerd.sock
          type: ""
        name: containerdsock
      - hostPath:
          path: /usr/local/bin/crictl
          type: ""
        name: crictl

@wenchajun
Copy link
Member 

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: fluentbit-operator
    meta.helm.sh/release-namespace: kubesphere-logging-system
  labels:
    app.kubernetes.io/component: operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluentbit-operator
  name: fluentbit-operator
  namespace: kubesphere-logging-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: operator
      app.kubernetes.io/name: fluentbit-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: operator
        app.kubernetes.io/name: fluentbit-operator
    spec:
      containers:
      - image: kubesphere/fluentbit-operator:v0.12.0
        imagePullPolicy: IfNotPresent
        name: fluentbit-operator
        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /fluentbit-operator
          name: env
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - /bin/sh
        - -c
        - set -ex; echo CONTAINER_ROOT_DIR=$(crictl info --output go-template --template
          '{{.config.containerdRootDir}}') > /fluentbit-operator/fluent-bit.env
        image: docker:19.03
        imagePullPolicy: IfNotPresent
        name: setenv
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /fluentbit-operator
          name: env
        - mountPath: /var/run/containerd/containerd.sock
          name: containerdsock
          readOnly: true
        - mountPath: /usr/local/bin/crictl   //change this to /usr/bin/crictl
          name: crictl
          readOnly: true
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: fluentbit-operator
      serviceAccountName: fluentbit-operator
      terminationGracePeriodSeconds: 30
      volumes:
      - name: env
      - hostPath:
          path: /var/run/containerd/containerd.sock
          type: ""
        name: containerdsock
      - hostPath:
          path: /usr/local/bin/crictl  //change this to /usr/bin/crictl
          type: ""
        name: crictl

because your crictl path is different from the origin path,so you can change this to solve this problem

@Evantage-WS
Copy link
Author

Done that, do see now:

+ echo 'CONTAINER_ROOT_DIR='

@wenchajun
Copy link
Member

You can delete the pod and run it again. Can you run this command on the command line and see if it returns a result

crictl info

or

crictl info --output go-template --template '{{.config.containerdRootDir}}'

@Evantage-WS
Copy link
Author

Do you want me to use the original deployment from the helm? Where do you want me to exec into a shell? There is no shell available in the fluentbit-operator pod.

@wenchajun
Copy link
Member

wenchajun commented Nov 29, 2021 •
edited

You just need to execute it from the command line, in your local machine

@Evantage-WS
Copy link
Author

Hi:

With /usr/bin/crictl:

/usr/bin/crictl crictl info --output go-template --template '{{.config.containerdRootDir}}'
/var/lib/containerd

With /usr/local/bin/crictl:

/usr/local/bin/crictl crictl info --output go-template --template '{{.config.containerdRootDir}}'
sh: 9: /usr/local/bin/crictl: Permission denied

@wenchajun
Copy link
Member

Done that, do see now:

+ echo 'CONTAINER_ROOT_DIR='

Is this the all message? You can delete the fluentbit-operator pod and check the message.

@Evantage-WS
Copy link
Author

This is all I see after deleting the pod again

+ echo 'CONTAINER_ROOT_DIR='

@wenchajun
Copy link
Member

I have used helm installation and cannot reproduce your problem. And I think the fluentbit-operator function works properly. If you can, please provide more error information.

@Evantage-WS
Copy link
Author

Hi @wenchajun,

I have removed the minikube cluster and installed a vm with kubernetes 1.22.3 (kubeadm install). I still have the problem, in the logs of the setenv container I still see:

+ echo 'CONTAINER_ROOT_DIR='

Is it possible to make the path: /usr/local/bin/crictl configurable in the values.yaml file?

@wenchajun
Copy link
Member

Hi @wenchajun,

I have removed the minikube cluster and installed a vm with kubernetes 1.22.3 (kubeadm install). I still have the problem, in the logs of the setenv container I still see:

+ echo 'CONTAINER_ROOT_DIR='

Is it possible to make the path: /usr/local/bin/crictl configurable in the values.yaml file?

OK, I will fix it. Do you want to commit a PR to resolve this?

@wenchajun wenchajun mentioned this issue Dec 7, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Set the crictl path to a variable wenchajun/fluentbit-operator
2 participants
@wenchajun @Evantage-WS