Ingress proxy unpacks assest and work slowly

[nurikk]

Describe the issue you are experiencing

Hello from zigbee2mqtt team!

Recently many users started complaining that z2m frontend open slowly.

zigbee2mqtt/hassio-zigbee2mqtt#236
nurikk/zigbee2mqtt-frontend#1124

After some debugging i can see that when frontend is loaded through ingress proxy, it's loaded very slowly:
there are few symptoms:

• Assets aren't gzip compressed
• Missing caching headers
• Sometimes response comes in form of headers from underlying backend as is, kind of http response parsing issue

Here're some screenshots

This how it looks if we request frontend directly, without ingress
First load (no cache)

After refresh

What is the used version of the Supervisor?

supervisor-2021.10.8

What type of installation are you running?

Home Assistant Supervised

Which operating system are you running on?

Debian

What is the version of your installed operating system?

Ubuntu 18.04.6 LTS

What version of Home Assistant Core is installed?

core-2021.11.5

Steps to reproduce the issue

1. Install zigbee2mqtt addon
2. Start it
3. Open web ui

Anything in the Supervisor logs that might be useful for us?

no

Additional information

nurikodd.duckdns.org.har.zip

Attached HAR archive from chrome dev tools

[nurikk]

Current workaround is to use websocat exported port for frontend and access directly on websocat port

[nurikk]

Hey guys, may be you can help? @danielwelch @ciotlosm

[Bascht74]

right now the Z2M frontend looks like this every second time. If you press reload, everything is fine...

[Bascht74]

@frenck Hi, I am sorry that I "quoted" you directly in that issue, but I saw in the last HA shows that you use Z2M yourself. There is an issue with the frontend and ingress (which seems to be a part of HA supervisor). Maybe you face the problem yourself (you need to reload the frontend sometimes that it shows correctly)
@nurikk made this issue about it but no one seems to pick it up. Do you have any hint how to proceed?
I saw problems with ingress when HA started using it but now I am not sure if there is somebody who will cover the ingress part...

[frenck]

@Bascht74 i don't use z2m as an add-on.

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[Enzokot]

Keep alive

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[Bascht74]

Bump

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[Enzokot]

Keep alive

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[Enzokot]

Keep alive

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[Enzokot]

Keep alive

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[Bascht74]

bump.

@danielwelch @ciotlosm can you take a look?

[Bascht74]

Anybody out here?

[Bascht74]

@balloob Hello Paul, normally HA issues are assigned by a bot, but here for the supervisor this doesn't seem to work.
I don't know what to do, just keeping this issue open.
Is there a better way or should i let the bot close the issue, as nobody seems to be able to take a look?
Thx for a small hint about that as I am clueless here...

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[Bascht74]

Still no reaction and no assignment. Is that right?

[mdegat01]

Hey sorry for the delay on this one @Bascht74 . I'm digging into this now.

So first thing I noticed is that the cache-control header seems to be the same for both ingress and non-ingress connections to Z2M. And in my testing just now I do actually see Z2M hitting the cache when using ingress after the first refresh. I realize that its been a while since this bug was first reported, do you see that change as well? Or are you still seeing Z2M skipping the cache?

Dev tools for Z2M direct access on second refresh with cache allowed:

Dev tools for Z2M ingress access on second refresh with cache allowed:

Now for the other two parts. The response actually gets unpacked twice along the way to the client unfortunately. Since ingress everything from the HA frontend (including ingress requests) go to core first. Core then sees it is an ingress request and so passes it to supervisor. Supervisor then checks that it has a valid ingress session and if so passes it on to the addon. The process then runs in reverse for the response.

The request handling makes sense but the response handling is a bit illogical. Core obviously has to process the response since it is often asked to be the TLS endpoint by users when they add their SSL certs to its HTTP config. But supervisor never has to handle SSL and is basically just pass through. When I asked about this it seems there's a limitation in aiohttp around this. We let aiohttp handle encoding and apparently it always decodes response, there is no way to pass a response straight through without decoding it. I can research that more as that seems like the obvious improvement, perhaps something has changed.

Since supervisor currently has to decode the response it is currently leveraging that to decide whether to stream it in chunks or not:

supervisor/supervisor/api/ingress.py

Lines 189 to 219 in 0991f52

	# Simple request
	if (
	hdrs.CONTENT_LENGTH in result.headers
	and int(result.headers.get(hdrs.CONTENT_LENGTH, 0)) < 4_194_000
	):
	# Return Response
	body = await result.read()
	return web.Response(
	headers=headers,
	status=result.status,
	content_type=result.content_type,
	body=body,
	)
	# Stream response
	response = web.StreamResponse(status=result.status, headers=headers)
	response.content_type = result.content_type
	try:
	await response.prepare(request)
	async for data in result.content.iter_chunked(4096):
	await response.write(data)
	except (
	aiohttp.ClientError,
	aiohttp.ClientPayloadError,
	ConnectionResetError,
	) as err:
	_LOGGER.error("Stream error with %s: %s", url, err)
	return response

I'm suspicious that this is the source of your 0B responses but I'm having a hard time reproducing that. Do you still see that behavior?

The thing that seems like the top priority here is why isn't the response gzipped? I definitely see exactly what you're talking about there. My understanding is that this should be handled by aiohttp. The request headers include the accepted encodings so it should be encoding the output in one of the accepted formats. That doesn't seem to be happening which is confusing, will look into why.

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.