Problem with Gitlab private registry and Swarm Stack using web editor

[Daxterpol]

Before you start please confirm the following.

• Yes, I've searched similar issues on GitHub.
• Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

For a long time now, creating a stack using the web editor and filling a docker-compose in using a private registry which is existing in the Registries menu and that we can browse normally and see the related image, does not work as attended.
Stack fails to deploy with the following error :

Feb  7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.056607119+01:00" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
Feb  7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.057113552+01:00" level=info msg="Ignoring extra error returned from registry" error="unauthorized: authentication required"
Feb  7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.060066974+01:00" level=error msg="pulling image failed" error="pull access denied for registry.gitlab.com/toto/toto_project/toto-img, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" module=node/agent/taskmanager node.id=os43f7oac0t1ldwqn6i6gxd3g service.id=hyz3frcbj571blhnbhnh0g58e task.id=xmwjcgjpfxvi6jzep19q0y62z
Feb  7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.060366644+01:00" level=error msg="fatal task error" error="No such image: registry.gitlab.com/toto/toto_project/toto-img" module=node/agent/taskmanager node.id=os43f7oac0t1ldwqn6i6gxd3g service.id=hyz3frcbj571blhnbhnh0g58e task.id=xmwjcgjpfxvi6jzep19q0y62z

Even if we try to do a docker login, same error is logged without the "repository does not exist or may require 'docker login'" log.
Sometimes, it works for a couple of days before being stuck again with these errors at the next webhook or manual polling.

The same container is successfully deployed if using Services and choosing the registry in the UI, instead of Stack which does not permit to use docker compose.

It sounds like the stack is not able to successfully use Registries credentials added successfully to portainer.

Expected Behavior

Stack is successfully deployed using the Registries set in Portainer and the associatied token with no error.

Actual Behavior

Stack is not successfully deployed using the Registries set in Portainer and the associatied token with authorization / authentication errror.

Steps to Reproduce

1. Add a private gitlab registry using the Registries menu with a token that allow to use images (or with all permission, same result)
2. Deploy a stack with the web editor using an image in the previously created gitlab registry as a container image
3. Stack fails to deploy the image with the previous authentication/authorization error

Portainer logs or screenshots

Feb  7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.056607119+01:00" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
Feb  7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.057113552+01:00" level=info msg="Ignoring extra error returned from registry" error="unauthorized: authentication required"
Feb  7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.060066974+01:00" level=error msg="pulling image failed" error="pull access denied for registry.gitlab.com/toto/toto_project/toto-img, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" module=node/agent/taskmanager node.id=os43f7oac0t1ldwqn6i6gxd3g service.id=hyz3frcbj571blhnbhnh0g58e task.id=xmwjcgjpfxvi6jzep19q0y62z
Feb  7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.060366644+01:00" level=error msg="fatal task error" error="No such image: registry.gitlab.com/toto/toto_project/toto-img" module=node/agent/taskmanager node.id=os43f7oac0t1ldwqn6i6gxd3g service.id=hyz3frcbj571blhnbhnh0g58e task.id=xmwjcgjpfxvi6jzep19q0y62z

Portainer version

2.19.4

Portainer Edition

Business Edition (BE/EE) with Starter Plus license

Platform and Version

Docker Swarm 24.0.7

OS and Architecture

Debian 11.8 AMD64

Browser

Chrome 121

What command did you use to deploy Portainer?

version: '3.8'

services:
  appv2:
    image: registry.gitlab.com/toto/toto_project/toto-img:latest
    restart: always
    ports:
      - "6889:443"
    extra_hosts:
      - "toto.toto.chc:192.168.50.65"
    deploy:
      resources:
        limits:
          cpus: '6'
          memory: 2048M
      update_config:
        order: start-first
      restart_policy:
        condition: any

Additional Information

We did a new fresh install of Portainer 2.19.4, the result is exactly the same.

[dblaws97473]

I'm experiencing the same error in the same conditions, I can't pull my private images

[lpotherat]

Similar issue with private self-hosted gitlab instance, and multiple registries from the same instance with different access tokens.

[fabien-github]

I have the same issue too, can we have an update on this please ?

[Daxterpol]

Hi,

Any news on that issue please ?

Thanks

[arbianqx]

Having the same issue. Multiple environments and each environment has one or two stacks. Each registry is for each stack but when I check into images I see preselected different registries.