[BE] Unable to retrieve networks, stacks, versions, etc. #8777

LPwithPaul · 2023-04-13T13:31:52Z

Bug description
After Updating Portainer-EE to V2.17.1 and the Portainer_agent on my Synology NAS to 2.17.1, the Errors "Unable to retrieve stacks", "Unable to retrieve networks", "Unable to retrieve version"... pop up while accessing the Environment from the control panel.

Expected behavior
The Environment should be accessible without errors and the Containers hosted within Docker should show up.

Portainer Logs
portainer_agent on Synology NAS: https://pastebin.com/LMFM5hEB
portainer-ee on VPS: https://pastebin.com/dWYPBkDR

Steps to reproduce the issue:

tbh I don't know -_-

Update portainer-ee and portainer_agent(on Synology NAS) to V2.17.1
Login on the Portainer control panel
Click on the Environment of the NAS
See error

Technical details:

Portainer version: V2.17.1
Docker version (managed by Portainer): VPS: 20.10.23 - NAS: 20.10.3
Kubernetes version (managed by Portainer): -
Platform (windows/linux): VPS: Ubuntu 20.04 - NAS: DSM 7.1.1-42962 Update 4
Command used to start Portainer (docker run -p 9443:9443 portainer/portainer): Regular Docker Commands as found in the Docs(except the volumes Binding on the NAS. That is customized)
Browser: Safari, Opera(different machine)
Use Case (delete as appropriate): Using Portainer at Home AND Using Portainer in a Commercial setup.
Have you reviewed our technical documentation and knowledge base? Yes

Additional context
The connection to the portainer_agent is handled over a Cloudflare tunnel(in order to not expose any Ports of my Home network) but accessing the hostname:port in a Browser outputs the regular API stating "{"message":"Missing request signature headers","details":"Unauthorized"}"

Edit: The Agent on a different VPS(Same Version as NAS) works completely fine...

The text was updated successfully, but these errors were encountered:

tamarahenson · 2023-04-15T00:01:49Z

@LPwithPaul

Thank you for the information. I am going to further investigate. I will update you as I learn more.

Thanks!

tamarahenson · 2023-04-15T00:07:34Z

@LPwithPaul

Can you post the steps you took to upgrade Portainer? I see in your logs you upgraded from 2.16.0.

Thanks!

LPwithPaul · 2023-04-15T10:57:33Z

@tamarahenson

Hey! First of all thanks for looking at this.

To upgrade i deleted the portainer container(it has a persistent volume), pulled the latest image from docker and then created a new container with the same settings as the old one.

Simple as that.

Thank you!

IrineSistiana · 2023-04-18T07:10:18Z

Hi. I thought I encountered the same issue. However, both my protainer and agent are newly installed, rather than from an upgrade.

Bug description:

Agent connection is good, but none of the pages are working, except "Host".

Portainer version: v2.18.1 CE
Docker version (managed by Portainer): 23.0.3 (API: 1.42)
Kubernetes version (managed by Portainer): -
Platform (windows/linux): protainer: raspberrypi, agent:ubuntu 22.04
Command used to start Portainer (docker run -p 9443:9443 portainer/portainer): Identical command from docs plus an AGENT_SECRAT env.
Command used to start Agent: Identical command from docs plus an AGENT_SECRAT env.
Browser: Chrome v112

Portainer directly connects to the agent. No proxy etc.

Portainer logs:

I noticed that, when one error message pops up in the web UI, the Portainer will print "http: proxy error: invalid character '\u0083' looking for beginning of value" at the same time.

It seems different error messages have different logs.

{"time":1681801007,"message":"http: proxy error: invalid character '\\u0083' looking for beginning of value"}
{"time":1681801217,"message":"http: proxy error: invalid character '\\b' after top-level value"}
{"time":1681801217,"message":"http: proxy error: invalid character '!' looking for beginning of value"}
{"time":1681801218,"message":"http: proxy error: invalid character '\\u0083' looking for beginning of value"}
{"time":1681801219,"message":"http: proxy error: invalid character '\\u0083' looking for beginning of value"}
{"time":1681801220,"message":"http: proxy error: invalid character '\\b' after top-level value"}

LPwithPaul · 2023-04-18T08:43:09Z

@IrineSistiana

Hi, first of all.

This is great to hear for me, as it confirms for me, that it wasn't just some dumb error of me 😄

Do you have your Agent hostet on a VPS or on a Synology like me?

Is the portainer host and the agent in the same network?

Thanks ✌️

LPwithPaul · 2023-04-18T09:23:12Z

Hi @IrineSistiana

Okay, then I suppose we have the same Issue, as i have also seen that error message.

But with me the working Agent and Host are on different networks. Maybe that will be helpful to the guys and girls that look into this issue further than we can.

Greets✌️

tamarahenson · 2023-04-18T13:06:44Z

@IrineSistiana

Does everything work as expected on 2.16.2?

@LPwithPaul

Have you upgraded to 2.18.1? Can you post results?

Thanks!

IrineSistiana · 2023-04-18T13:08:53Z

Hi @tamarahenson

I just install a new 2.16.2 Portainer and agnet to test this. And that does not work.

IrineSistiana · 2023-04-18T13:17:02Z

2.16.2 log

2023/04/18 12:58PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:530 > encryption key file not present | filename=portainer
2023/04/18 12:58PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:549 > proceeding without encryption key |
2023/04/18 12:58PM INF github.com/portainer/portainer/api/database/boltdb/db.go:124 > loading PortainerDB | filename=portainer.db
2023/04/18 12:59PM INF github.com/portainer/portainer/api/internal/ssl/ssl.go:80 > no cert files found, generating self signed SSL certificates |
2023/04/18 12:59:01 server: Reverse tunnelling enabled
2023/04/18 12:59:01 server: Fingerprint 33:20:20:4c:ae:18:ee:58:fe:86:c9:e8:ef:1c:27:ae
2023/04/18 12:59:01 server: Listening on 0.0.0.0:8000...
2023/04/18 12:59PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:789 > starting Portainer | build_number=25294 go_version=1.19.3 image_tag=linux-arm64-2.16.2 nodejs_version=18.12.1 version=2.16.2 webpack_version=5.68.0 yarn_version=1.22.19
2023/04/18 12:59PM INF github.com/portainer/portainer/api/http/server.go:337 > starting HTTPS server | bind_address=:9443
2023/04/18 12:59PM INF github.com/portainer/portainer/api/http/server.go:322 > starting HTTP server | bind_address=:9000
{"time":1681822744,"message":"http: TLS handshake error from 10.8.0.3:55208: remote error: tls: unknown certificate"}
{"time":1681822744,"message":"http: TLS handshake error from 10.8.0.3:55204: remote error: tls: unknown certificate"}
{"time":1681822744,"message":"http: TLS handshake error from 10.8.0.3:55206: remote error: tls: unknown certificate"}
{"time":1681822745,"message":"http: TLS handshake error from 10.8.0.3:55223: remote error: tls: unknown certificate"}
{"time":1681822745,"message":"http: TLS handshake error from 10.8.0.3:55225: remote error: tls: unknown certificate"}
{"time":1681822746,"message":"http: TLS handshake error from 10.8.0.3:55226: remote error: tls: unknown certificate"}
{"time":1681822746,"message":"http: TLS handshake error from 10.8.0.3:55227: remote error: tls: unknown certificate"}
{"time":1681822748,"message":"http: TLS handshake error from 10.8.0.3:55230: remote error: tls: unknown certificate"}
{"time":1681822799,"message":"http: proxy error: invalid character '(' after top-level value"}
{"time":1681822800,"message":"http: proxy error: invalid character '@' after top-level value"}
{"time":1681822800,"message":"http: proxy error: invalid character '(' after top-level value"}
{"time":1681822807,"message":"http: proxy error: invalid character '(' after top-level value"}
{"time":1681822904,"message":"http: proxy error: invalid character '¸' after top-level value"}
{"time":1681822906,"message":"http: proxy error: invalid character '¸' after top-level value"}
{"time":1681822907,"message":"http: proxy error: invalid character '¼' after top-level value"}
{"time":1681822907,"message":"http: proxy error: invalid character '@' after top-level value"}
{"time":1681822907,"message":"http: proxy error: invalid character '¼' after top-level value"}
{"time":1681822918,"message":"http: proxy error: invalid character '¼' after top-level value"}
{"time":1681822918,"message":"http: proxy error: invalid character '@' after top-level value"}
{"time":1681822918,"message":"http: proxy error: invalid character '¼' after top-level value"}
{"time":1681822920,"message":"http: proxy error: invalid character '@' after top-level value"}
{"time":1681822921,"message":"http: proxy error: invalid character '¼' after top-level value"}
{"time":1681822923,"message":"http: proxy error: invalid character '¼' after top-level value"}
{"time":1681823106,"message":"http: TLS handshake error from 10.8.0.3:55554: remote error: tls: bad certificate"}
{"time":1681823142,"message":"http: proxy error: invalid character '¸' after top-level value"}
{"time":1681823142,"message":"http: proxy error: invalid character '@' after top-level value"}
{"time":1681823143,"message":"http: proxy error: invalid character '¸' after top-level value"}
{"time":1681823144,"message":"http: proxy error: invalid character '@' after top-level value"}
{"time":1681823146,"message":"http: proxy error: invalid character '¸' after top-level value"}
{"time":1681823147,"message":"http: proxy error: invalid character '¸' after top-level value"}
{"time":1681823189,"message":"http: proxy error: invalid character '@' after top-level value"}
{"time":1681823192,"message":"http: proxy error: invalid character '@' after top-level value"}
{"time":1681823193,"message":"http: proxy error: invalid character '¸' after top-level value"}
{"time":1681823193,"message":"http: proxy error: invalid character '¸' after top-level value"}

LPwithPaul · 2023-04-18T19:14:41Z

Have you upgraded to 2.18.1? Can you post results?
@tamarahenson

After updating both the EE and the Agent to "latest" it still doesn't work.

I just deleted my Portainer data volume and re-install it again. And the problem is gone.
Should I try this too? There shouldn't be any serious loss of data, should it?

tamarahenson · 2023-04-21T09:00:15Z

@IrineSistiana

Upon review of the logging snippets:

{"time":1681801007,"message":"http: proxy error: invalid character '\\u0083' looking for beginning of value"}
{"time":1681801217,"message":"http: proxy error: invalid character '\\b' after top-level value"}
{"time":1681801217,"message":"http: proxy error: invalid character '!' looking for beginning of value"}
{"time":1681801218,"message":"http: proxy error: invalid character '\\u0083' looking for beginning of value"}
{"time":1681801219,"message":"http: proxy error: invalid character '\\u0083' looking for beginning of value"}
{"time":1681801220,"message":"http: proxy error: invalid character '\\b' after top-level value"}

You may be encountering an issue with the characters of your AGENT_SECRET. Are they quoted and/or escaped properly?

Thanks!

IrineSistiana · 2023-04-22T03:54:56Z

@LPwithPaul Try to disable the "Brotli" in your Cloudflare domain dashboard, under speed/optimization section. Works for me.

It seems this "http: proxy error: invalid character" error is caused by accidentally reading compressed data from resp.Body and io.Copy() the data to the downstream as uncompressed data.

I set up a local http reverse proxy to dump requests between the agent and portioner. All response jsons are good. And they are all compressed. After I intensionally remove the "accept-encoding" header to disable compression. The problem is gone.

I'm not good at Go, so I can't dig into it further.

@tamarahenson

Update:

I'm sure this issue is caused by the Brotli compressed response. The Portainer forwards browser header, which "accept-encoding" contains "br". If the remote responses with Brotli compressed data (Agent itself don't support it, but a reverse proxy may do). Then

portainer/api/http/proxy/factory/utils/response.go

Line 100 in 3654109

isGzip := response.Header.Get("Content-Encoding") == "gzip"

portainer/api/http/proxy/factory/utils/json.go

Line 41 in 3654109

if isGzip {

Json decoder recevied raw Brotli compressed data.

LPwithPaul · 2023-04-24T12:40:43Z

I can confirm, that disabling Brotli has worked. It's all back to normal now. Thanks a lot for your help.

db00t · 2023-10-01T12:41:07Z

Set correct time.
Command in Ubuntu:
timedatectl set-timezone Europe/London
Worked for me after I got a lot of "Unable to retrieve" messages on a freshly build.

jcmendez · 2024-05-27T01:13:24Z

I'm running into similar error messages, and I believe something related to the dates is the problem as @db00t mentions, as in the logs, there are mixed messages where one of the times seem to be the local time stamp on EDT and others seem to be UTC, as it is already next day...

2024/05/26 08:59PM http: proxy error: context canceled 
2024/05/26 09:00PM http: proxy error: context canceled 
2024/05/26 09:00PM http: proxy error: context canceled 
2024/05/27 01:01AM WRN platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/05/27 01:01AM WRN platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/05/27 01:03AM WRN platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/05/26 09:03PM http: proxy error: context canceled 
2024/05/26 09:03PM http: proxy error: context canceled 
2024/05/26 09:03PM http: proxy error: context canceled 
2024/05/26 09:03PM http: proxy error: context canceled 
2024/05/26 09:03PM http: proxy error: context canceled 
2024/05/26 09:04PM http: proxy error: context canceled 
2024/05/26 09:04PM http: proxy error: context canceled 
2024/05/26 09:06PM http: proxy error: context canceled 
2024/05/26 09:06PM http: proxy error: context canceled 
2024/05/27 01:09AM WRN platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

b14ckw1d0w · 2024-06-05T16:27:01Z

Cloudflare has removed the option to disable Brotli for free members meaning I'm back to having non working portainer agents again.

Is there any working alternatives? I couldn't get my head around edge agents but would assume if they are also going through cloudflare the same compression issues would be there too.

IrineSistiana · 2024-06-06T00:08:14Z

Hi @b14ckw1d0w.

Disable br compression in your brower should also work, if I remember the bug correctly(?). But anyways, it seems this bug hasn't been fixed.

I.m not using Cloudflare right now. So I cannot reproduce it. Maybe you can open a new issus if you can reproduce the bug and provide log for the dev team easily.

(Although i don't think log and reproducing are important because the bug is kinda straightforward.) #8777 (comment)

LPwithPaul added bug/need-confirmation kind/bug Applied to Bugs labels Apr 13, 2023

LPwithPaul changed the title ~~[EE] Unable to retrieve networks, stacks, versions, etc.~~ [BE] Unable to retrieve networks, stacks, versions, etc. Apr 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BE] Unable to retrieve networks, stacks, versions, etc. #8777

[BE] Unable to retrieve networks, stacks, versions, etc. #8777

LPwithPaul commented Apr 13, 2023 •

edited

Loading

tamarahenson commented Apr 15, 2023

tamarahenson commented Apr 15, 2023 •

edited

Loading

LPwithPaul commented Apr 15, 2023

IrineSistiana commented Apr 18, 2023 •

edited

Loading

LPwithPaul commented Apr 18, 2023

This comment was marked as outdated.

LPwithPaul commented Apr 18, 2023

This comment was marked as outdated.

tamarahenson commented Apr 18, 2023

IrineSistiana commented Apr 18, 2023

IrineSistiana commented Apr 18, 2023

LPwithPaul commented Apr 18, 2023 •

edited

Loading

This comment was marked as outdated.

tamarahenson commented Apr 21, 2023

This comment was marked as outdated.

IrineSistiana commented Apr 22, 2023 •

edited

Loading

LPwithPaul commented Apr 24, 2023

db00t commented Oct 1, 2023

jcmendez commented May 27, 2024

b14ckw1d0w commented Jun 5, 2024

IrineSistiana commented Jun 6, 2024

[BE] Unable to retrieve networks, stacks, versions, etc. #8777

[BE] Unable to retrieve networks, stacks, versions, etc. #8777

Comments

LPwithPaul commented Apr 13, 2023 • edited Loading

tamarahenson commented Apr 15, 2023

tamarahenson commented Apr 15, 2023 • edited Loading

LPwithPaul commented Apr 15, 2023

IrineSistiana commented Apr 18, 2023 • edited Loading

LPwithPaul commented Apr 18, 2023

This comment was marked as outdated.

LPwithPaul commented Apr 18, 2023

This comment was marked as outdated.

tamarahenson commented Apr 18, 2023

IrineSistiana commented Apr 18, 2023

IrineSistiana commented Apr 18, 2023

LPwithPaul commented Apr 18, 2023 • edited Loading

This comment was marked as outdated.

tamarahenson commented Apr 21, 2023

This comment was marked as outdated.

IrineSistiana commented Apr 22, 2023 • edited Loading

LPwithPaul commented Apr 24, 2023

db00t commented Oct 1, 2023

jcmendez commented May 27, 2024

b14ckw1d0w commented Jun 5, 2024

IrineSistiana commented Jun 6, 2024

LPwithPaul commented Apr 13, 2023 •

edited

Loading

tamarahenson commented Apr 15, 2023 •

edited

Loading

IrineSistiana commented Apr 18, 2023 •

edited

Loading

LPwithPaul commented Apr 18, 2023 •

edited

Loading

IrineSistiana commented Apr 22, 2023 •

edited

Loading