Reccuring error in my logs "unexpected status code | status_code=403" #8077
Comments
|
Thank you for the information. I am going to further investigate. Interim, an HTTP 403 response code means that a client is forbidden from accessing a valid URL. The server understands the request, but it can't fulfill the request because of client-side issues. Can you post log snippets from both Portainer Server and Agent for review? What type of hardware is your Swarm cluster running on? Thanks! |
|
Hello @tamarahenson, Here my portainer logs : portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF main.go:582 > encryption key file not present | filename=portainer
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF main.go:601 > proceeding without encryption key |
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF github.com/portainer/portainer-ee/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF github.com/portainer/portainer-ee/api/datastore/backup.go:151 > creating DB backup |
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF github.com/portainer/portainer-ee/api/datastore/backup.go:74 > copying DB file | from=/data/portainer.db to=/data/backups/EE/portainer.db.071.20221119211114
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF github.com/portainer/portainer-ee/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF github.com/portainer/portainer-ee/api/datastore/migrate_data.go:121 > migrating database | migrator_version=71 portaineree_db_version=71
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF github.com/portainer/portainer-ee/api/datastore/migrate_data.go:139 > migrator_edition=EE portainer_edition=EE store_edition=EE
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF github.com/portainer/portainer-ee/api/cloud/cluster_setup.go:77 > starting cloud cluster setup service |
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF main.go:558 > instance already has defined environments, skipping the environment defined via CLI |
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 21:11:14 server: Reverse tunnelling enabled
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 21:11:14 server: Fingerprint de:48:e6:0a:1d:f4:0b:b7:fe:97:ac:a5:bc:8f:1b:d1
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 21:11:14 server: Listening on 0.0.0.0:8000...
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF main.go:887 > starting Portainer | build_number=24946 go_version=1.19.3 image_tag=linux-amd64-2.16.1 nodejs_version=18.12.1 version=2.16.0 webpack_version=5.65.0 yarn_version=1.22.19
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF github.com/portainer/portainer-ee/api/http/server.go:392 > starting HTTPS server | bind_address=:9443
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:11PM INF github.com/portainer/portainer-ee/api/http/server.go:377 > starting HTTP server | bind_address=:9000
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:26PM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:27PM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:27PM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:27PM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:27PM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:27PM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:28PM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:28PM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
portainer_portainer.1.p2gsg0vu6k9s@frhb65918ds | 2022/11/19 09:28PM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403Here my agent logs : portainer_agent.0.6euwud0ahg91@frhb65918ds | 2022/11/19 09:09PM INF github.com/portainer/agent/cmd/agent/main.go:79 > agent running on Docker platform |
portainer_agent.0.6euwud0ahg91@frhb65918ds | 2022/11/19 09:09PM INF github.com/portainer/agent/cmd/agent/main.go:94 > agent running on a Swarm cluster node. Running in cluster mode |
portainer_agent.0.6euwud0ahg91@frhb65918ds | 2022/11/19 21:09:18 [INFO] serf: EventMemberJoin: frhb65918ds-da6a668853f7 10.0.193.3
portainer_agent.0.6euwud0ahg91@frhb65918ds | 2022/11/19 09:09PM INF github.com/portainer/agent/edge/registry/server.go:101 > starting registry credential server |
portainer_agent.0.6euwud0ahg91@frhb65918ds | 2022/11/19 09:09PM INF github.com/portainer/agent/http/server.go:100 > starting Agent API server | api_version=2.16.1 server_addr=0.0.0.0 server_port=9001 use_tls=true
portainer_agent.0.6euwud0ahg91@frhb65918ds | 2022/11/19 21:25:56 http: TLS handshake error from 10.0.193.6:53124: EOF
portainer_agent.0.6euwud0ahg91@frhb65918ds | 2022/11/19 21:26:10 http: TLS handshake error from 10.0.193.6:53118: EOF
portainer_agent.0.6euwud0ahg91@frhb65918ds | 2022/11/19 21:29:30 http: TLS handshake error from 10.0.193.6:59420: EOF
portainer_agent.0.6euwud0ahg91@frhb65918ds | 2022/11/19 21:31:12 http: TLS handshake error from 10.0.193.6:38986: read tcp 10.0.193.3:9001->10.0.193.6:38986: read: connection reset by peerHere my hardware config :
|
|
I accidentally reproduced your issue in my cluster. I am trying to connect to an environment via an agent and am receiving context deadline exceeded toasts in the UI when this happens:
In my case, I had an environment connected to a Portainer server via an Agent. I tried connecting that same environment to a different Portainer server. Thanks! |
|
Hi, I just upgraded from Portainer |
|
Still happening on 2.19.1.
|
|
starting Portainer | build_number=35428 go_version=1.20.5 image_tag=linux-amd64-2.19.4 nodejs_version=18.19.0 version=2.19.4 webpack_version=5.88.1 yarn_version=1.22.21 2024/01/08 04:13PM ERR github.com/portainer/portainer/api/http/client/client.go:92 > unexpected status code | status_code=403 |
|
Same here. I get the error message every time the Portainer browser tab gets focus or loses focus. |
|
Problem still existing. |
|
Sadly, this issue is still present in the recently released Portainer 2.20.0. |
I confirm, I was about to file the same bug. I read the thread above but I am not sure whether there is anything tangible about that bug (whether it is a problem (I do not see any), is there a workaround (if needed), or how to silence it) |
It was the same on my end. 2024/04/24 06:50AM ERR github.com/portainer/portainer/api/http/client/client.go:92 > unexpected status code | status_code=403 multiple times from the interval 6:18 am to 6:50 am I upgraded to 2.20.1 and it fixed the problem. (idk if it stopped cause i updated or if it just stopped outright) |
|
I can confirm that this problem is still present in Portainer 2.20.1 😢. However, I finally found the cause of the issue after modifying Portainer to log the URL in addition to the status code. As it can be seen there, the URL that causes the sporadic 403 errors is: This URL is called by Portainer itself to check if there are newer versions available in GitHub. Why does it return 403? Because Portainer is calling this URL very frequently, and GitHub is applying rate limiting. If you open a browser with that URL after seeing those 403 errors, you will get the error from GitHub: Portainer seems to be calling this URL when you switch windows or tabs in the browser, as you can see this error much more frequently when you do so. This is because the Portainer UI triggers a version check everytime the window/tab regains focus (not a good idea in my opinion and should be considered a bug). The reason this error does not appear always is because it depends on the state of rate-limiting from Github for your current public IP address. If you leave Portainer for a bit (enough for the rate limiting to be lifted) you will not see these errors until the rate limit is triggered again. @jamescarppe (or anyone in Portainer Team) does the above help fixing this quite long standing issue? :) EDIT: The frequency of calling GitHub API should be fixed, but perhaps it also would be a good idea to add a command-line flag to disable new version checks entirely if desired? |
Wow, this is impressive @hhromic - it is not common to see such good bug reports. I will have to get better at digging down |
|
Looks like the Portainer team already fixed this in #11679 some days ago (even before my investigation above). Looking forward to see this annoying bug gone :) |
|
Yep, someone beat you to the root cause in #11596 so I raised it internally, and the team have a fix in place for our next release. The fix reduces the check frequency to once every 24 hours (except when hard refreshing in the browser), which should mean simply using Portainer should no longer push the rate limit up. Once the release including this is out we'll loop back to this (and the other) thread to confirm this is fixed, and if so we'll close the issue(s) off. |
|
We've just released 2.20.3 which includes a fix for the overzealous update checks - are you able to test and confirm whether this has cleared up the log messages? |
|
@jamescarppe in my case these lines are still there: While using Portainer 2.20.3 EE |
|
That's a different message so will be a different issue. It's informational so nothing you should be concerned with, but I'll flag it with the team as we might not need to be outputting those on regular log levels. |
Bug description
I have a reccuring error in my logs every minutes :
Expected behavior
No errors.
Portainer Logs
Steps to reproduce the issue:
DOMAIN=portainer.example.com IMAGE=portainer/portainer-ee docker stack deploy -c <(curl -so - https://raw.githubusercontent.com/ethibox/awesome-stacks/master/stacks/portainer.yml) portainerTechnical details:
The text was updated successfully, but these errors were encountered: