Failed to pull image "docker.io/bitnami/bitnami-shell:11-debian-11-r114"

[utkarshatembhurkar]

Name and Version

docker.io/bitnami/rabbitmq:3.11.15-debian-11-r3

What architecture are you using?

None

What steps will reproduce the bug?

I am using this configuration with Kustomize to deploy to GKE and getting the error:

Failed to pull image "[docker.io/bitnami/bitnami-shell:11-debian-11-r114](http://docker.io/bitnami/bitnami-shell:11-debian-11-r114)": rpc error: code = Unknown desc = failed to pull and unpack image "[docker.io/bitnami/bitnami-shell:11-debian-11-r114](http://docker.io/bitnami/bitnami-shell:11-debian-11-r114)": failed to resolve reference "[docker.io/bitnami/bitnami-shell:11-debian-11-r114](http://docker.io/bitnami/bitnami-shell:11-debian-11-r114)": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  labels:
    app.kubernetes.io/name: rabbitmq
    app.kubernetes.io/component: server
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: rabbitmq
      app.kubernetes.io/component: server
  serviceName: rabbitmq-discovery

  replicas: 1
  updateStrategy:
      # https://www.rabbitmq.com/upgrade.html
      # https://cloud.google.com/kubernetes-engine/docs/how-to/updating-apps
      type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: rabbitmq
        app.kubernetes.io/component: server
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/tls-skip-verify: 'true'
        vault.hashicorp.com/agent-requests-cpu: 5m
        vault.hashicorp.com/agent-requests-mem: 16Mi
        vault.hashicorp.com/agent-limits-cpu: 10m
        vault.hashicorp.com/agent-limits-mem: 32Mi
    spec:
      serviceAccountName: rabbitmq-sa
      terminationGracePeriodSeconds: 180
      securityContext:
        fsGroup: 1001
      # This init container copies the config files from read-only ConfigMap to writable location.
      initContainers:
      - name: volume-permissions
        image: docker.io/bitnami/bitnami-shell:11-debian-11-r114
        imagePullPolicy: "IfNotPresent"
        command:
          - /bin/bash
        args:
          - -euc
          - |
            # first source the variables to have the RABBITMQ_ERL_COOKIE
            source <(cat /vault/secrets/*.env)
            mkdir -p "/bitnami/rabbitmq/mnesia"
            chown "1001:1001" "/bitnami/rabbitmq/mnesia"
            find "/bitnami/rabbitmq/mnesia" -mindepth 1 -maxdepth 1 -not -name ".snapshot" -not -name "lost+found" | \
            xargs -r chown -R "1001:1001"

            # Copy the mounted configuration to both places.
            cp /rabbitmqconfig/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf
            # Change permission to allow to add more configurations via variables
            chown 1001:1001 /etc/rabbitmq/rabbitmq.conf
            chmod 660 /etc/rabbitmq/rabbitmq.conf
            cp /rabbitmqconfig/enabled_plugins /etc/rabbitmq/enabled_plugins
        securityContext:
          runAsUser: 0
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 200m
            memory: 256Mi
        volumeMounts:
        - name: data
          mountPath: /bitnami/rabbitmq/mnesia
        - name: configmap
          mountPath: /rabbitmqconfig
        - name: config
          mountPath: /etc/rabbitmq

      containers:
      - name: rabbitmq
        image: docker.io/bitnami/rabbitmq:3.11.15-debian-11-r3
        imagePullPolicy: "IfNotPresent"
        securityContext:
          runAsUser: 1001 
          privileged: false
          allowPrivilegeEscalation: false
        command: ['/bin/bash']
        args:
          - -c
          - |
            source <(cat /vault/secrets/*.env)
            /opt/bitnami/scripts/rabbitmq/entrypoint.sh
            /opt/bitnami/scripts/rabbitmq/run.sh
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 300m
            memory: 512Mi
        lifecycle:
        
          # postStart:
          #   exec:
          #     command:
          #     - /bin/bash
          #     - -ec
          #     - |
          #       # Wait for the RabbitMQ to be ready.
          #       until rabbitmq-diagnostics -q check_running; do
          #         sleep 5
          #       done

          #       # By default, RabbitMQ does not have Highly Available policies enabled,
          #       # using the following command to enable it.
          #       rabbitmqctl set_policy ha-all "." '{"ha-mode":"all", "ha-sync-mode":"automatic"}' --apply-to all --priority 0
          preStop:
            exec:
              command:
                - /bin/bash
                - -ec
                - |
                  if [[ -f /opt/bitnami/scripts/rabbitmq/nodeshutdown.sh ]]; then
                      /opt/bitnami/scripts/rabbitmq/nodeshutdown.sh -t "180" -d "false"
                  else
                      rabbitmqctl stop_app
                  fi
        env:
        - name: JAVA_OPTS
          value: >-
                -Dlog4j2.formatMsgNoLookups=true
                -Dlog4j2.disable.jmx=true
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: DISCOVERY_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: K8S_SERVICE_NAME
          value: 'gted-archive-system-rabbitmq-discovery'
        - name: RABBITMQ_USE_LONGNAME
          value: 'true'
        - name: RABBITMQ_FORCE_BOOT
          value: "no"
        - name: RABBITMQ_NODENAME
          value: 'rabbit@$(MY_POD_NAME).gted-archive-system-rabbitmq-discovery.$(DISCOVERY_NAMESPACE).svc.cluster.local'
        - name: K8S_HOSTNAME_SUFFIX
          value: '.gted-archive-system-rabbitmq-discovery.$(DISCOVERY_NAMESPACE).svc.cluster.local'
        - name: RABBITMQ_MNESIA_DIR
          value: "/bitnami/rabbitmq/mnesia/$(RABBITMQ_NODENAME)"
        - name: RABBITMQ_LOGS
          value: "-"
        - name: RABBITMQ_ULIMIT_NOFILES
          value: "65536"
        - name: RABBITMQ_LOAD_DEFINITIONS
          value: "no"
        - name: RABBITMQ_DEFINITIONS_FILE
          value: "/app/load_definition.json"
        ports:
        - name: clustering
          containerPort: 25672
        - name: amqp
          containerPort: 5672
        - name: amqp-ssl
          containerPort: 5671
        - name: prometheus
          containerPort: 15692
        - name: http
          containerPort: 15672
        volumeMounts:
        - name: config
          mountPath: /etc/rabbitmq
        - name: data
          mountPath: /bitnami/rabbitmq/mnesia
        livenessProbe:
          exec:
            command:
            - rabbitmq-diagnostics
            - -q
            - ping
          initialDelaySeconds: 60
          timeoutSeconds: 30
          periodSeconds: 60
        readinessProbe:
          exec:
            command:
            - rabbitmq-diagnostics
            - -q
            - status
          initialDelaySeconds: 20
          timeoutSeconds: 30
          periodSeconds: 60

      volumes:
      - name: configmap
        configMap:
          name: rabbitmq-config
          items:
          - key: rabbitmq.conf
            path: rabbitmq.conf
          - key: enabled_plugins
            path: enabled_plugins
      - name: config
        emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app.kubernetes.io/name: rabbitmq
        app.kubernetes.io/component: server
    spec:
      accessModes:
      - ReadWriteOnce
      storageClassName: standard-rwo
      resources:
        requests:
          storage: 8Gi

What do you see instead?

Failed to pull image "[docker.io/bitnami/bitnami-shell:11-debian-11-r114](http://docker.io/bitnami/bitnami-shell:11-debian-11-r114)": rpc error: code = Unknown desc = failed to pull and unpack image "[docker.io/bitnami/bitnami-shell:11-debian-11-r114](http://docker.io/bitnami/bitnami-shell:11-debian-11-r114)": failed to resolve reference "[docker.io/bitnami/bitnami-shell:11-debian-11-r114](http://docker.io/bitnami/bitnami-shell:11-debian-11-r114)": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[avinal]

I am having the same error with bitnami/postgresql from around 5 hrs ago. The bitnami/bitnami-shell repo is gone from the docker hub and I can't find a breaking change/deprecation notice anywhere.

[jack1902]

TL;DR: try to use bitnami/os-shell instead, but the default container user is different to the original bitnami/bitnami-shell

As a slight work-around, you can use https://hub.docker.com/r/bitnami/os-shell but be aware that certain older bitnami charts don't set the runAsUser so elements of older charts might still break.

Glad to see i'm not the only one hitting this issue, i have internal mirrors but its broken other things i had which referenced the bitnami-shell image

[utkarshatembhurkar]

using os-shell helped me.
Thank you

[davejhahn]

This is a pretty big breaking change--we are running into this after doing some other upgrades where the pods migrated and couldn't start up. Unless you are on a chart version that has this image, you will run into this. We are on 19.6.0, I only see a couple since.

[carrodher]

The bitnami/bitnami-shell image was deprecated some months ago in favor of bitnami/os-shell (see bitnami/containers#41421). That change was announced with a deprecation notice in the DockerHub README.
After 7 months and following the retention policy defined at README.md#retention-policy, the bitnami/bitnami-shell repo was removed in favor of bitnami/bitnami-shell-archived where all the existing images tags are present.

This process is followed in that way to not keep unmaintained and insecure images not receiving updates for a long time in DockerHub. With the relocation of the images to a different repo, users are aware that they are using a deprecated image. If they want to continue using it, the only required change is the replacement of bitnami/bitnami-shell with bitnami/bitnami-shell-archived in their values.yaml. Although the preferred options are 1) using a newer version of the Helm chart bundling this image or, 2) at least, replacing the image with the maintained bitnami/os-shell.

[avinal]

Hey @carrodher I am using the configuration below, it uses initContainer to generate and copy certificate, and that needs bitnami/bitnami-shell to work.

  source:
    chart: postgresql
    repoURL: https://charts.bitnami.com/bitnami
    targetRevision: 12.2.7
    helm:
      releaseName: postgres
      parameters:
        - name: image.tag
          value: 13.10.0
        - name: tls.enabled
          value: "true"
        - name: tls.autoGenerated
          value: "true"

Are the newer images updated to use bitnami/os-shell? I am looking for the commit/code for the change.

[carrodher]

bitnami-shell and os-shell are equivalent images, just a rename was performed to deliver this image not only as part of the opensource catalog (Bitnami Application Catalog) but also in the enterprise catalog (VMware Tanzu Application Catalog).

If os-shell doesn't work for your use case, feel free to open a separate issue so we can investigate what happened. In any case, using bitnami-shell-archived where previously bitnami-shell was used should work fine since both images are the same, even with the same tags.

[jack1902]

@carrodher can see that bitnami/bitnami-shell is available again? Just checking, the bitnami/bitnami-shell-deprecated that you mention doesn't seem to work https://hub.docker.com/r/bitnami/bitnami-shell-deprecated

I was using an old version of elasticsearch chart which deployed the bitnami/bitnami-shell image which had:

# Uses the old `bitnami/bitnami-shell`
- image: {{ include "elasticsearch.sysctl.image" . }}
  imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
  command:
    - /bin/bash
    - -ec
    - |
      {{- include "elasticsearch.sysctlIfLess" (dict "key" "vm.max_map_count" "value" "262144") | nindent 14 }}
      {{- include "elasticsearch.sysctlIfLess" (dict "key" "fs.file-max" "value" "65536") | nindent 14 }}
  securityContext:
    privileged: true

In newer versions of the chart, it uses the bitnami/os-shell, but a subtle different in the deployment specs for the sysctl image:

securityContext:
  privileged: true
  runAsUser: 0

I appreciate i am on an older chart-version but bitnami/os-shell isn't a direct replacement for bitnami/bitnami-shell due to the older chart version not having the runAsUser: 0 (i get permission denied when using os-shell whereas bitnami-shell didn't encounter that issue)

[carrodher]

Just checking, the bitnami/bitnami-shell-deprecated that you mention doesn't seem to work https://hub.docker.com/r/bitnami/bitnami-shell-deprecated

Sorry @jack1902, unfortunately, that was a typo on my latest comment (I just edited it). The proper repo name is -archived instead of -deprecated, see https://hub.docker.com/r/bitnami/bitnami-shell-archived.

Replacing bitnami-shell with bitnami-shell-archived should work out of the box since it is an exact copy of the images from one repo to another

[jack1902]

@carrodher cheers, will swap over to https://hub.docker.com/r/bitnami/bitnami-shell-archived as a stop-gap whilst upgrades to newer charts happen to make use of os-shell (FYI, the link you posted in your last comment is to deprecated not archived

[carrodher]

Hello everyone,

Following your valuable feedback, we have made adjustments to our retention policy to better serve the Bitnami Helm charts community:

• For regular images, the time window in which both repositories (bitnami/foo and bitnami/foo-archived) coexist has been extended from 3 to 6 months. After that period, bitnami/foo will be fully removed.
• Special images, like bitnami/bitnami-shell, which are extensively used in Helm charts, will now have an extended coexistence period of 1 year.

In the case of bitnami/bitnami-shell, it has been recovered and will be removed on July 2024, precisely 1 year after the initial deprecation date.

We strongly encourage everyone to update their deployments to a recent Helm chart version or consider using bitnami/os-shell whenever possible to ensure continued access to updates.