Image pulling works for individual images or for containers, fails with (re)deploying a stack

[ivoras]

Before you start please confirm the following.

• Yes, I've searched similar issues on GitHub.
• Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

We have a local GitLab registry with two peculiarities: #1 is that it's behind a TLS client authentication proxy, and #2 is that this proxy also munges the host name of the service, as the same GitLab registry is deployed in a private network under a different name.

Pulling images from docker CLI works, and it also works while pulling individual images from Portainer - either individually for images, or for an already existing container, but it fails with a 403 Forbidden when pulling the same images as a part of a stack (docker-compose file referencing the images).

Expected Behavior

(Re)deploying a stack works, including pulling required images.

Actual Behavior

failed to pull images of the stack: image1 Pulling image2 Pulling image3 Pulling image4 Pulling image5 Error image2 Error image1 Error image3 Error Error response from daemon: Head "https://fqdn/v2/path/manifests/on-prod-v4": denied: access forbidden

The fqdn is the proxy address and it's indeed accessible publicly (as we can pull docker images from it with the docker CLI and individual images with Portainer).

Steps to Reproduce

Not sure. You'd have to have a nginx proxy which does TLS client auth for incoming requests, and rewrites the host name to the internal one.

Portainer logs or screenshots

failed to pull images of the stack: image1 Pulling image2 Pulling image3 Pulling image4 Pulling image5 Error image2 Error image1 Error image3 Error Error response from daemon: Head "https://fqdn/v2/path/manifests/on-prod-v4": denied: access forbidden

Portainer version

2.19.0

Portainer Edition

Community Edition (CE)

Platform and Version

Docker 24.0.6

OS and Architecture

Ubuntu 22.04

Browser

Chrome

What command did you use to deploy Portainer?

services:

  portainer:
    image: portainer/portainer-ce:latest
    restart: unless-stopped
    command: --sslcert /srv/cert/eu.crt --sslkey /srv/cert/eu.key
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /srv/cert:/srv/cert
      - /etc/docker/certs.d:/etc/docker/certs.d
      - portainer-data:/data
    ports:
      - 9443:9443
    networks:
      - internal
    security_opt:
      - no-new-privileges:true

volumes:
  portainer-data:

networks:
  internal:

### Additional Information

_No response_

[JcMinarro]

I have similar error.

On my case I have multiple stacks, each of them working with different private registries (Both of them private registries on Docker Hub).

I can see that only the "most recent updated" docker hub registry is used to re-deploy images.

For example, imagine I have two docker hub accounts (accountA and accountB).
I added both of them as Docker Hub accounts within the registries section

Then I have 2 stcks:

version: '3.7'
services:
  app:
    container_name: projectA
    image: "accountA/projectA:main"
    restart: always

version: '3.7'
services:
  app:
    container_name: projectB
    image: "accountB/projectB:main"
    restart: always

If the last registry I added/updated was accountB, only the second stack will be able to be re-deployed, but the first stack will fail because of authentication permission.

If I go to the "Images Section" I am able to re-pull accountA/projectA:main and accountB/projectB:main images by selecting the appropriate registry.

[arbianqx]

I have similar error.

On my case I have multiple stacks, each of them working with different private registries (Both of them private registries on Docker Hub).

I can see that only the "most recent updated" docker hub registry is used to re-deploy images.

For example, imagine I have two docker hub accounts (accountA and accountB). I added both of them as Docker Hub accounts within the registries section

Then I have 2 stcks:

version: '3.7'
services:
  app:
    container_name: projectA
    image: "accountA/projectA:main"
    restart: always

version: '3.7'
services:
  app:
    container_name: projectB
    image: "accountB/projectB:main"
    restart: always

If the last registry I added/updated was accountB, only the second stack will be able to be re-deployed, but the first stack will fail because of authentication permission.

If I go to the "Images Section" I am able to re-pull accountA/projectA:main and accountB/projectB:main images by selecting the appropriate registry.

I have the exact problem and it's really frustrating. Any updates on this?