MQTT won't start after HA OS upgrade

[blacknell]

Describe the issue you are experiencing

After upgrading to HA OS 11.0 MQTT addon won't start

Relates issue
Relates #3253

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Which add-on are you reporting an issue with?

Mosquitto broker

What is the version of the add-on?

6.3.1

Steps to reproduce the issue

1. Upgraded to 11.0
2. Restart fails

...

System Health information

System Information

version	core-2023.10.0
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.11.5
os_name	Linux
os_version	6.1.21-v8
arch	aarch64
timezone	Europe/London
config_dir	/config

Home Assistant Community Store

GitHub API	ok
GitHub Content	ok
GitHub Web	ok
GitHub API Calls Remaining	4998
Installed Version	1.32.1
Stage	running
Available Repositories	1312
Downloaded Repositories	8

Home Assistant Cloud

logged_in	false
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Home Assistant Supervisor

host_os	Home Assistant OS 11.0
update_channel	stable
supervisor_version	supervisor-2023.10.0
agent_version	1.6.0
docker_version	24.0.6
disk_total	458.4 GB
disk_used	77.1 GB
healthy	true
supported	true
board	rpi4-64
supervisor_api	ok
version_api	ok
installed_addons	Samba share (10.0.2), Mosquitto broker (6.3.1), Duck DNS (1.15.0), Check Home Assistant configuration (3.11.0), Terminal & SSH (9.7.1), Home Assistant Google Drive Backup (0.111.1), Zigbee2MQTT (1.33.1-1), InfluxDB (4.8.0), File editor (5.6.0), Grafana (9.1.0), ESPHome (2023.9.3)

Dashboards

dashboards	15
resources	6
views	36
mode	storage

Recorder

oldest_recorder_run	5 October 2023 at 05:49
current_recorder_run	17 October 2023 at 10:11
estimated_db_size	1369.58 MiB
database_engine	sqlite
database_version	3.41.2

Anything in the Supervisor logs that might be useful for us?

2023-10-17 12:09:02: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:03] INFO: Service restart after closing
[12:09:04] INFO: Starting mosquitto MQTT broker...
1697540944: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:04: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:04] INFO: Service restart after closing
[12:09:06] INFO: Starting mosquitto MQTT broker...
1697540946: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:06: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:06] INFO: Service restart after closing
[12:09:07] INFO: Starting mosquitto MQTT broker...
1697540947: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:07: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:08] INFO: Service restart after closing
[12:09:09] INFO: Starting mosquitto MQTT broker...
1697540949: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:09: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:09] INFO: Service restart after closing
[12:09:11] INFO: Starting mosquitto MQTT broker...
1697540951: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:11: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:11] INFO: Service restart after closing
[12:09:12] INFO: Successfully send discovery information to Home Assistant.
[12:09:12] INFO: Starting mosquitto MQTT broker...
1697540952: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:12: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[12:09:13] INFO: Successfully send service information to the Supervisor.
[11:09:13] INFO: Service restart after closing
[12:09:14] INFO: Successfully send discovery information to Home Assistant.
[12:09:14] INFO: Starting mosquitto MQTT broker...
1697540954: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:14: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:15] INFO: Service restart after closing
[12:09:16] INFO: Successfully send service information to the Supervisor.
[12:09:16] INFO: Successfully send discovery information to Home Assistant.
[12:09:17] INFO: Starting mosquitto MQTT broker...
1697540957: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:17: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:18] INFO: Service restart after closing
[12:09:18] INFO: Successfully send service information to the Supervisor.
[12:09:18] INFO: Successfully send discovery information to Home Assistant.
[12:09:19] INFO: Starting mosquitto MQTT broker...
1697540959: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:19: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:20] INFO: Service restart after closing
[12:09:21] INFO: Successfully send service information to the Supervisor.
[12:09:21] INFO: Successfully send discovery information to Home Assistant.
[12:09:21] INFO: Starting mosquitto MQTT broker...
1697540961: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:21: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:22] INFO: Service restart after closing
[12:09:23] INFO: Successfully send discovery information to Home Assistant.
[12:09:23] INFO: Successfully send service information to the Supervisor.
[12:09:24] INFO: Starting mosquitto MQTT broker...
1697540964: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:24: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[12:09:24] INFO: Successfully send discovery information to Home Assistant.
[11:09:25] INFO: Service restart after closing
[12:09:26] INFO: Successfully send service information to the Supervisor.
[12:09:26] INFO: Starting mosquitto MQTT broker...
1697540966: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:26: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:27] INFO: Service restart after closing
[12:09:28] INFO: Successfully send discovery information to Home Assistant.
[12:09:28] INFO: Successfully send service information to the Supervisor.
[12:09:29] INFO: Starting mosquitto MQTT broker...
1697540969: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:29: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[12:09:29] INFO: Successfully send discovery information to Home Assistant.
[11:09:30] INFO: Service restart after closing
[12:09:31] INFO: Successfully send service information to the Supervisor.
[12:09:31] INFO: Starting mosquitto MQTT broker...
1697540971: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:31: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[12:09:32] INFO: Successfully send discovery information to Home Assistant.
[11:09:33] INFO: Service restart after closing
[12:09:33] INFO: Successfully send discovery information to Home Assistant.
[12:09:33] INFO: Successfully send service information to the Supervisor.
[12:09:34] INFO: Starting mosquitto MQTT broker...
1697540974: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:34: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:35] INFO: Service restart after closing
[12:09:36] INFO: Successfully send discovery information to Home Assistant.
[12:09:36] INFO: Successfully send service information to the Supervisor.

Anything in the add-on logs that might be useful for us?

No response

Additional information

/mnt/data/supervisor/addons/data/core_mosquitto/options.json

{
  "logins": [],
  "require_certificate": true,
  "certfile": "fullchain.pem",
  "keyfile": "privkey.pem",
  "customize": {
    "active": true,
    "folder": "mosquitto"
  }
}

mosquitto/mqtt.conf

#log_timestamp_format %Y-%m-%d %H:%M:%S %Z

[doronazl]

same here, no ones on this issue ?

[homeerik]

After upgrading to OS 11 i have random stops of Zigbee. Sometime al is working fine for a few days en suddenly all Zigbee devices are unavailable. I can restart HA anymore and only remove power. After reboot al is working fine until it is happening again.

Before the upgrade Zigbee and HA was rock solid.

I use the combination:
Sonoff 3.0 P version usb stick
Mosquitto broker
Zigbee2MQTT

Log files are not showing anything.

[blacknell]

I could not wait long for a reply. This meant I had to remove and install mosquitto.

[inonoob]

Dear all,

I also have the same issue. I've remove at least 3 time mosquitto and re-installed it. That fixes the issue for some time but then it starts again with the same error message:

[12:14:24] INFO: Successfully send service information to the Supervisor. [10:14:24] INFO: Service restart after closing [12:14:24] INFO: Successfully send discovery information to Home Assistant. [12:14:25] INFO: Starting mosquitto MQTT broker... 2023-10-20 12:14:25: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persist__restore_sub: Assertion "client_id" failed. [12:14:26] INFO: Successfully send service information to the Supervisor. [10:14:26] INFO: Service restart after closing

Something seems wrong.

Home assistant OS 11.0 and

[doronazl]

Circling back just to share what i did.
I have a lot of important sensors relying on the stability of mqtt, cant have this kind of unplanned problems and sit around waiting for an update again.

What i did is reverted to the backup before the update and went the route of seperating the mqtt from home assistant.
All you need is a raspberry pi 3, installed mosquitto mqtt on it, changed settings to route to the new server and uninstalled the mosquito addon, now no update from home assistant cant effect my mqtt server uptime, i then updated the OS and moved on from this problem.
This may not be a fix but a workaround, but its also making my system more resilient, happy with the results

[inonoob]

@doronazl

This is what I'm doing. I move now MQTT to a virtual debian machine with influxdb and Grafana with Promox.
I don't trust the add-ons anymore.
After an update in home assistant. The system just remove all my data I had collected in influxdb (3month in total lost). Since then I move now all to a dedicated machine for that task. And Home assistant is also going to move from the Pi to the Virtual machine.

[doronazl]

@doronazl

This is what I'm doing. I move now MQTT to a virtual debian machine with influxdb and Grafana with Promox. I don't trust the add-ons anymore. After an update in home assistant. The system just remove all my data I had collected in influxdb (3month in total lost). Since then I move now all to a dedicated machine for that task. And Home assistant is also going to move from the Pi to the Virtual machine.

Your way probably better, im just not a fan of VMs unless you know exactly what youre doing, otherwise lots of simple things can get complicated later when having to deal with those layers.

Im old fashioned, my HA is on a NUC, my mqtt on a rpi, they are both battery backed to keep things running at all times

[inonoob]

It might not be the best way but I had a server laying around and found a good use for it.

And for the authors of the add-on. I still would like to say thank you. I know open source can be sometimes difficult. Doing stuff in the spear time. So big thank you again. I learnt a lot with it.

[DarthSonic]

Solution is to uninstall and the re-install mosquitto addon. Configuration and everything is still intact after that.

[blacknell]

Solution is to uninstall and the re-install mosquitto addon. Configuration and everything is still intact after that.

This wasn't my experience. Uninstalling meant I lost the persistent mosquito database and so all retained messages.

I lost a lot of devices since the discovery was deleted. I'm not sure why the devices were deleted since the discovery payloads weren't cleared but that's what happened.

[DarthSonic]

Solution is to uninstall and the re-install mosquitto addon. Configuration and everything is still intact after that.

This wasn't my experience. Uninstalling meant I lost the persistent mosquito database and so all retained messages.

I lost a lot of devices since the discovery was deleted. I'm not sure why the devices were deleted since the discovery payloads weren't cleared but that's what happened.

Did not notice that, as I do not need any persistent currently.

[kazikskrz]

Is there a quick fix available without reinstalling?

[skank01]

Same problem here

[jt-nti]

If it's any help, I managed to get slightly further with a mosquitto.conf file (using the customize flag) and setting persistence to false. I think that's causing its own issues but at least Zigbee2MQTT is connecting now.

[rootiam]

If it's any help, I managed to get slightly further with a mosquitto.conf file (using the customize flag) and setting persistence to false. I think that's causing its own issues but at least Zigbee2MQTT is connecting now.

I have the same issue. This didn't work for me.

[jt-nti]

I've had a chance to get hold of the persistance db out of the mosquitto addon and have a look using the mosquitto_db_dump tool, and there are some null client IDs in there which kind of makes sense based on the log...

DB_CHUNK_SUB:
	Length: 38
	Client ID: (null)
	Topic: stat/tasmota_000000/RESULT
	QoS: 0
	Subscription ID: 0
	Options: 0x00

I wonder if there's a mosquitto issue letting null client IDs through but then failing to process them? They were all tasmota related in my db, so that seems worth a look as well.

[hackex]

I've got into the same issue of error in mosquitto addon:
"persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.[". Sadly, none of my mosquitto messages get received because of this error. I hope for a fix.

[CaptainBalou]

Same here.

[00:26:21] INFO: Starting mosquitto MQTT broker... 2023-11-18 00:26:21: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persist__restore_sub: Assertion client_id' failed.
[23:26:22] INFO: Service restart after closing
[00:26:22] INFO: Successfully send discovery information to Home Assistant.
[00:26:22] INFO: Successfully send service information to the Supervisor.
[00:26:23] INFO: Starting mosquitto MQTT broker...
2023-11-18 00:26:23: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion client_id' failed. [23:26:23] INFO: Service restart after closing [00:26:23] INFO: Successfully send discovery information to Home Assistant. [00:26:24] INFO: Successfully send service information to the Supervisor. [00:26:24] INFO: Starting mosquitto MQTT broker... 2023-11-18 00:26:24: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persist__restore_sub: Assertion client_id' failed.
[23:26:24] INFO: Service restart after closing
[00:26:25] INFO: Successfully send discovery information to Home Assistant.
[00:26:25] INFO: Starting mosquitto MQTT broker...
2023-11-18 00:26:25: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion client_id' failed. [00:26:25] INFO: Successfully send service information to the Supervisor. [23:26:26] INFO: Service restart after closing [00:26:26] INFO: Successfully send discovery information to Home Assistant. [00:26:27] INFO: Starting mosquitto MQTT broker... 2023-11-18 00:26:27: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persist__restore_sub: Assertion client_id' failed.
[00:26:27] INFO: Successfully send service information to the Supervisor.
[23:26:27] INFO: Service restart after closing
[00:26:28] INFO: Successfully send discovery information to Home Assistant.
[00:26:28] INFO: Starting mosquitto MQTT broker...
2023-11-18 00:26:28: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion client_id' failed. [00:26:28] INFO: Successfully send service information to the Supervisor. [23:26:29] INFO: Service restart after closing [00:26:29] INFO: Successfully send discovery information to Home Assistant. [00:26:30] INFO: Starting mosquitto MQTT broker... . . .

[fsaris]

Same issue here. After reinstall it worked again (for now).

[jt-nti]

@hackex @CaptainBalou @fsaris do you have any Tasmota devices? That seems to be what caused the problem for me- at least everything has been working since fixing the mosquitto.db and removing the problematic Tasmota device. If not, it would be interesting to see which client has the null id in your case. Unfortunately it's a bit fiddly to find that out, so it might be nice to update the mosquitto addon to (optionally?) log the output of a db_dump at start up?

[fsaris]

Yes, I have tasmota devices.

So you expect that one of the devices broke the mosquitto db?

[stfnhmplr]

Same here. I have also Tasmota devices.

2023-11-19 10:50:17: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[09:50:17] INFO: Service restart after closing
[10:50:18] INFO: Starting mosquitto MQTT broker...
2023-11-19 10:50:18: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[09:50:18] INFO: Service restart after closing
[10:50:19] INFO: Starting mosquitto MQTT broker...

[jt-nti]

@fsaris it's a possibility. If you're able to build and run the db_dump tool, it should show the null client ID. (I still have three Tasmotas which aren't causing any problems so I'm not sure why one of them had the null client ID.)

[inonoob]

I can give the feedback I don't have Tasmota devices but I have that issue. I have AI on the Edge. I don't know if that might be a problem aswell. So it might not only be for one device brand only.

[obfri]

Same problem detected today - even with MQTT 6.4.0. Using Tasmota devices.

[obfri]

my tasmota devices showing MQTT login errors - nothing changed:
20:52:16.971 MQT: Attempting connection...
20:52:16.988 MQT: Connect failed to 192.168.xxx.xxx:1883, rc -2. Retry in 20 sec
20:52:37.964 MQT: Attempting connection...
20:52:37.985 MQT: Connect failed to 192.168.xxx.xxx:1883, rc -2. Retry in 30 sec

did some more investigations:
failure started after MQTT 6.4.0 update (done Nov 20)
full restore of Nov19 full backup eliminates failure

[jt-nti]

The good news is that there's a fix coming soon (hopefully 🤞) from mosquitto which should help...

eclipse/mosquitto@fd097d5

I still think it would be nice if the home assistant addon could include the mosquitto_db_dump tool as well- not really sure where to start but I might attempt to have a look over the Christmas break.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jt-nti]

It would be good to use the updated version of mosquitto when it's available, and including the db_dump tool might still be nice to help debug any future problems.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jt-nti]

It doesn't look like the mosquitto fix has been released yet.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.