Added custom registries for specific docker images, still getting errors

[andsoitgoes]

Bug description
I've defined specific registries in portainer to try and make the process simplified as I'm still having digest errors populate my logs, however even after I'm getting multiple errors such as:

INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=lscr.io/linuxserver/duckdns:latest
and

ERR client/client.go:94 > unexpected status code | status_code=403
Expected behavior
no errors

Portainer Logs

2023-04-19T20:22:56.239907440Z 2023/04/19 20:22:56 server: Reverse tunnelling enabled
2023-04-19T20:22:56.239991660Z 2023/04/19 20:22:56 server: Fingerprint 62:f0:f2:26:b0:2d:77:38:44:c3:26:a8:7b:ae:0f:6d
2023-04-19T20:22:56.240010752Z 2023/04/19 20:22:56 server: Listening on 0.0.0.0:8000...
2023-04-19T20:22:56.543020923Z 2023/04/19 08:22PM INF main.go:902 > starting Portainer | build_number=29776 go_version=1.19.4 image_tag=linux-arm64-2.18.1 nodejs_version=18.16.0 version=2.18.1 webpack_version=5.65.0 yarn_version=1.22.19 
2023-04-19T20:22:56.563859938Z 2023/04/19 08:22PM INF http/server.go:411 > starting HTTPS server | bind_address=:9443 
2023-04-19T20:22:56.564374722Z 2023/04/19 08:22PM INF http/server.go:396 > starting HTTP server | bind_address=:9000 
2023-04-19T20:23:03.370955449Z 2023/04/19 08:23PM ERR client/client.go:94 > unexpected status code | status_code=403 
2023-04-19T20:23:06.340001409Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=lscr.io/linuxserver/ombi:development 
2023-04-19T20:23:06.340155202Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=docker.io/oznu/homebridge:no-avahi 
2023-04-19T20:23:06.341578833Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=lscr.io/linuxserver/jackett:latest 
2023-04-19T20:23:06.345347831Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=docker.io/portainer/portainer-ee:latest 
2023-04-19T20:23:06.349228215Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=lscr.io/linuxserver/duckdns:latest 
2023-04-19T20:23:06.475812175Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=lscr.io/linuxserver/prowlarr:develop 
2023-04-19T20:23:07.233998557Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=lscr.io/linuxserver/radarr:latest 
2023-04-19T20:23:07.290302833Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=lscr.io/linuxserver/swag:latest 
2023-04-19T20:23:07.290965020Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=lscr.io/linuxserver/sonarr:latest 
2023-04-19T20:23:07.339522544Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=docker.io/haugene/transmission-openvpn:latest 
2023-04-19T20:23:07.400325022Z 2023/04/19 08:23PM INF images/digest.go:62 > Can not find registry auth, try to access in anonymous | image=docker.io/containrrr/watchtower:latest 
2023-04-19T20:23:37.888291165Z 2023/04/19 08:23PM ERR client/client.go:94 > unexpected status code | status_code=403 
2023-04-19T20:26:00.205599612Z 2023/04/19 08:26PM ERR client/client.go:94 > unexpected status code | status_code=403 
2023-04-19T20:26:00.755970992Z 2023/04/19 08:26PM ERR client/client.go:94 > unexpected status code | status_code=403 
2023-04-19T20:26:13.234309065Z 2023/04/19 08:26PM ERR client/client.go:94 > unexpected status code | status_code=403 
2023-04-19T20:27:06.462620372Z 2023/04/19 08:27PM ERR client/client.go:94 > unexpected status code | status_code=403 
2023-04-19T20:27:19.738956197Z 2023/04/19 08:27PM ERR client/client.go:94 > unexpected status code | status_code=403 
2023-04-19T20:27:41.062948013Z 2023/04/19 08:27PM ERR client/client.go:94 > unexpected status code | status_code=403 

Steps to reproduce the issue:

1. Add specific registries for required containers
2. Restart Portainer
3. View logs

Technical details:

• Portainer version: 2.18.1
• Docker version (managed by Portainer): Standalone 23.0.1
• Kubernetes version (managed by Portainer): u/k
• Platform (windows/linux): Raspbian OS
• Command used to start Portainer (docker run -p 9443:9443 portainer/portainer):

sudo docker run -d \
 -p 8000:8000 \
 -p 9000:9000 \
 -p 9443:9443 \
  --name=portainer \
  --restart=always \
  --pull=always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  portainer/portainer-ee:latest

• Browser: Chrome
• Use Case (delete as appropriate): Using Portainer at Home
• Have you reviewed our technical documentation and knowledge base? Yes

Additional context
I thought by adding in the various registries it would rid me of errors, but it's just added further errors in

[tamarahenson]

@andsoitgoes

You can ignore Can not find registry auth, try to access in anonymous for now.

I am curious what these are unexpected status code | status_code=403

Can you navigate to Portainer >> Notifications. And, post a screenshot of that page for review?

Thanks!

[andsoitgoes]

See attached! Thanks so much Tamara! Really appreciate your attention with these issues, it means the world!

[tamarahenson]

@andsoitgoes

Did you use AGENT_SECRET in your Agent install? You can read more here about variables: https://github.com/portainer/agent#deployment-options

I am seeing in your Notifications the Failed loading environment unauthorized and then the request failed with status code 401 (which is related to authentication credentials) and the request failed with status 403 (related to client forbidden to accessing URLs) it appears if some kind of authentication is setup on the Agent.

What kind of Environment are you trying to connect to? Edge Agent, Docker Swarm, Standalone, K8s? Can you post your Agent install script? Did anything change recently (agent upgrade) or is this a new install?

Thanks!

[andsoitgoes]

Awesome, thanks for your reply!

Didn't make any changes other than adding in some app templates for a few different containers, as they weren't getting updated when I added in the dockerhub login, which I've removed. It seems like the 403 errors are gone as of yesterday, I can't exactly remember if that's when I removed the authentication login, I'm back to docker anonymous with the specific registries to monitor the various containers I have.

I think, for now, I'll close this. If I see the errors again, then I'll double check at that point?

Unless there is something else?

[tamarahenson]

@andsoitgoes

I am going to follow up with Product regarding the 403s. I was testing a separate issue and I started getting the 403s upon receiving an error. Here is a snippet of my log:

=====================>]   11.1MB/11.1MB\nwrite /var/lib/docker/tmp/GetImageBlob1042692503: no space left on device\n" output=
2023/04/19 10:43PM ERR github.com/portainer/portainer/api@v0.0.0-20230403061909-99331a81d4b0/http/client/client.go:94 > unexpected status code | status_code=403
2023/04/19 10:44PM ERR github.com/portainer/portainer/api@v0.0.0-20230403061909-99331a81d4b0/http/client/client.go:94 > unexpected status code | status_code=403

I got an error about disk space and then the 403s started logging. I believe they are OK to ignore as they appear after a for reals error happens.

Thanks!