MQTT update from 6.0.2 to 6.1.1 gives 500 error message

[camyb1981]

Describe the issue you are experiencing

I'm running HASS core-2022.4.7 with MQTT addon on version 6.0.2, when I try to update to MQTT 6.1.1 I get the below error.

Addon Error message:
Can't install homeassistant/amd64-addon-mosquitto:6.1.1: 500 Server Error for http+docker://localhost/v1.41/images/homeassistant/amd64-addon-mosquitto:6.1.1/json: Internal Server Error ("readlink /mnt/data/docker/overlay2/l: invalid argument")

At the moment all is working fine, except for the update which is failing. Other updates on other addons do not have any issues....it is only the MQTT which is failing and I'm a bit lost in the reason why. Looks like an INVALID ARGUMENT is passed...but again this is an automated process so the invalid argument must be hidden somewhere.

What could be the issue?

What type of installation are you running?

Home Assistant Supervised

Which operating system are you running on?

Home Assistant Operating System

Which add-on are you reporting an issue with?

Mosquitto broker

What is the version of the add-on?

6.0.2

Steps to reproduce the issue

1. Go to the CONFIGURATION of HASS
2. Click on the ADDONS option
3. Click on the MQTT addon
4. Stop MQTT
5. Click on the UPDATE button (see picture)
6. Then error is shown
   ...

Anything in the Supervisor logs that might be useful for us?

22-05-04 10:49:27 INFO (SyncWorker_3) [supervisor.docker.interface] Updating image homeassistant/amd64-addon-mosquitto:6.0.2 to homeassistant/amd64-addon-mosquitto:6.1.1

22-05-04 10:49:27 INFO (SyncWorker_3) [supervisor.docker.interface] Downloading docker image homeassistant/amd64-addon-mosquitto with tag 6.1.1.

22-05-04 10:49:28 ERROR (SyncWorker_3) [supervisor.docker.interface] Can't install homeassistant/amd64-addon-mosquitto:6.1.1: 500 Server Error for http+docker://localhost/v1.41/images/homeassistant/amd64-addon-mosquitto:6.1.1/json: Internal Server Error ("readlink /mnt/data/docker/overlay2/l: invalid argument")

Anything in the add-on logs that might be useful for us?

No response

Additional information

No response

[homonto]

same here, all my mqtt clients on raspberry pi lost connection with mosquitto broker after upgrade - had to return to 6.0.2
but ESP32 clients did not have this problem

Core
Version
core-2022.4.7

Supervisor
Version
supervisor-2022.05.0

Host
Operating System
Home Assistant OS 7.6

[camyb1981]

Is there anybody who can look into this? Are there other people who run into the same issue and maybe can share how they solved it?

[Krzysztonek]

I'd appreciate it if someone would investigate this issue. I'm able to install the 6.1.1 version, but no client connections to the MQTT server can be established (authorization error).

Might be that Broker v.6.1.1. is corrupted? v.6.0.2 worked quite nicely.

[mdegat01]

So there's multiple issues in here. @camyb1981 is showing an issue installing an update. @homonto and @Krzysztonek you appear to be talking about post-update authorization issues. Can you two make a new issue and move that discussion there? The authorization issues aren't the same thing.

@camyb1981 First of all can you clarify how you are running Home Assistant? You put Home Assistant Operating System and Home Assistant Supervised which doesn't make sense. Those are two different installation methods, which are you using?

As for the error presented, that is confusing. We install all addons the same way so I don't see how you could get an invalid argument here. My guess is that the addon did not shut down properly the first time and now its failing to start it again since the addon's data folder is already mounted in another container. Do you have access to the docker cli? If so can you run docker container ls -a and show me the output? Should show whether the container is in a bad state.

[homonto]

@mdegat01 you are right - I am working on ... rewriting the scripts on my Raspberries as till now they did not use username/password (all other clients use authentication)
So I believe I can announce my problem not being a add-on problem but rather ... my problem
I am not done yet but in case I don't succeed I will open new topic

[mdegat01]

@homonto I was wondering about that. Yea there was an unannounced breaking change in the 6.1.0 release because, well, I thought it had already been done. Part of PR #2007 was this:

Support for anonymous logins has been removed

So I assumed that capability had been removed. However it turns out in v1.6 of Mosquitto if the config did not explicitly say allow_anonymous false it defaulted to allowing anonymous logins. In v2.0+ of Mosquitto (which is what 6.1.0 updated Mosquitto to) Mosquitto fixed this with their own breaking change as allowing anonymous logins is a bad security practice. Now anonymous logins are disabled unless the config explicitly states allow_anonymous true. So the breaking change that was supposed to be part of v6.0.0 of this addon has now actually happened in v6.1.0.

If this is what broke you (you too @Krzysztonek ) then you can work around this by using customize option to set allow_anonymous true in your config. But I would advise you both against that and instead provide credentials to all your mqtt clients.

[homonto]

@mdegat01 topic closed, my python scripts got extra security ;-)
Thank you a lot!

btw:
allow_anonymous: true
in config did not change much but lots of messages like:

1651779830: Client <unknown> disconnected, not authorised.
1651779830: New connection from 192.168.1.72:46079 on port 1883.

maybe I did not know the syntax
but I prefered to add user/password now

[camyb1981]

So there's multiple issues in here. @camyb1981 is showing an issue installing an update. @homonto and @Krzysztonek you appear to be talking about post-update authorization issues. Can you two make a new issue and move that discussion there? The authorization issues aren't the same thing.

@camyb1981 First of all can you clarify how you are running Home Assistant? You put Home Assistant Operating System and Home Assistant Supervised which doesn't make sense. Those are two different installation methods, which are you using?

As for the error presented, that is confusing. We install all addons the same way so I don't see how you could get an invalid argument here. My guess is that the addon did not shut down properly the first time and now its failing to start it again since the addon's data folder is already mounted in another container. Do you have access to the docker cli? If so can you run docker container ls -a and show me the output? Should show whether the container is in a bad state.

Thanks for your reply, my bad about the form which wasn't filled in correctly. I'm running HASS from a Oracle VirtualBox with the original HASS distro. This seems to me like a dedicated HASS OS as I'm not really able to perform any specific linux command from the commandline. I do have the terminal addon installed which does some kind of Linux interfacing. But I hope this is more clear now.

This doesn't change the fact of the issue I'm having. I'm not doing anything special...I just go from HASS to the addon page, click the MQTT addon and from there I see that a new version is available....When I hit the UPDATE button I get the 500 error message...that's all. So indeed I don't touch or pass any input args with the request, that is also why I'm stuck on the update.

What I maybe could do is DELETE the addon and reinstall, but I'm a bit worried that it all will stop working once I do this and that I need to reconfigure all my zigbee2mqtt devices.

Does anybody know what the risk is of removing the MQTT addon and reinstall? Will this break my zigbee2mqtt process?
Of course I can make a full backup and then do it....but to be honest just the idea that I'll be spending the whole day behind the HASS to get it all to work again holds me back on this.

It would be so much easier if the update would just work and not throw a 500 error message. A proper fix for this would be great...if there is no fix and the issue is only with my instance...then I'll need to take the time and reinstall I guess.

[Krzysztonek]

Hi @camyb1981
I hope you don't need to remove or reinstall the MQTT addon. First, please check HA Core version you're using.
I discovered that I've been using core-2022.5.0 and it was not compatible with MQTT 6.1.1.
Once I updated my HA Core to version 2022.5.1. the problem with MQTT authenticating simply disappeared.
From now on my HA + MQTT addon works fine for me. :)

[camyb1981]

So there's multiple issues in here. @camyb1981 is showing an issue installing an update. @homonto and @Krzysztonek you appear to be talking about post-update authorization issues. Can you two make a new issue and move that discussion there? The authorization issues aren't the same thing.

@camyb1981 First of all can you clarify how you are running Home Assistant? You put Home Assistant Operating System and Home Assistant Supervised which doesn't make sense. Those are two different installation methods, which are you using?

As for the error presented, that is confusing. We install all addons the same way so I don't see how you could get an invalid argument here. My guess is that the addon did not shut down properly the first time and now its failing to start it again since the addon's data folder is already mounted in another container. Do you have access to the docker cli? If so can you run docker container ls -a and show me the output? Should show whether the container is in a bad state.

Below you see the output of the command docker container ls -a:

[homonto]

reinstalling is not the big thing - always have a FULL backup BEFORE you do it though ;-)
you cannot be afraid of reinstalling - one day something goes wrong and then what? you actually should be PREPARED for the worst while hoping for the best ;-)

[camyb1981]

core-202

Indeed I'm running on

Version	core-2022.5.0
Installation Type	Home Assistant OS

I don't see any update available in HASS....how did you update from core-2022.5.0 to core-2022.5.1?

[homonto]

[camyb1981]

ot be afraid of reinstalli

reinstalling is not the big thing - always have a FULL backup BEFORE you do it though ;-) you cannot be afraid of reinstalling - one day something goes wrong and then what? you actually should be PREPARED for the worst while hoping for the best ;-)

hehee, I know, because I already did it 2 times. So yes I know that pain and yes I make full backups before, not after ;)
But even having a full backup once did not worked for me :(

[Krzysztonek]

2022.5.1 just simply appeared on my dashboard today early morning. Seems you need to wait a bit for your turn. Have a coffee meantime ;)

[homonto]

ot be afraid of reinstalli

reinstalling is not the big thing - always have a FULL backup BEFORE you do it though ;-) you cannot be afraid of reinstalling - one day something goes wrong and then what? you actually should be PREPARED for the worst while hoping for the best ;-)

hehee, I know, because I already did it 2 times. So yes I know that pain and yes I make full backups before, not after ;) But even having a full backup once did not worked for me :(

that is why I always have a copy of the config files to ... redo from the scratch ;-)
It happened probably once only in my history with HA

[camyb1981]

I've restarted HASS and now I see 2022.5.1 available...hmmm.
Let's go...

[Krzysztonek]

The problem is that some of the updates of HA components sometimes aren't being released in the correct order.. moreover, sometimes they depend on each other and it makes some confusion.
Rebooting a machine is the only way to have them always displayed correctly.

[camyb1981]

Update did not solve the issue...same error message

Version	core-2022.5.1
Installation Type	Home Assistant OS
Development	false
Supervisor	true
Docker	true
User	root
Virtual Environment	false
Python Version	3.9.9
Operating System Family	Linux
Operating System Version	5.10.108
CPU Architecture	x86_64
Timezone	Europe/Amsterdam

[Krzysztonek]

Then I'd suggest installing a newly issued core update v.2022.5.2.

[camyb1981]

I think I'll need to delete the addon and reinstall...even with v.2022.5.3 the issue is still here. Thank you all for thinking along with me on this.

[mdegat01]

@camyb1981 can you check your free space on your file system? Should be in Settings -> System -> Storage. Docker starts having strange issues when it has no room left to put images so perhaps you're hitting that?

EDIT: Before trying any docker commands, try running ha su repair before updating. Forgot to suggest that first, that does a bunch of tasks to clean things up and should (hopefully) make the below commands unnecessary.

---

Do you have access to the docker CLI set up? If so can try just doing this directly:

docker pull homeassistant/amd64-addon-mosquitto:6.1.1

That's what supervisor does behind the scenes, sometimes doing it directly can give some more information. Could also simply try running the following to see if it frees up enough space to get past this:

docker system prune -a

There's not really any harm to this command it simply removes unused images. HA should re-download the ones it needs again if something got accidently purged.

Does anybody know what the risk is of removing the MQTT addon and reinstall? Will this break my zigbee2mqtt process?

For MQTT there's actually really minimal risk here. Almost everything that goes through the broker is not stored, it is just immediately passed off to listeners and then forgotten. The only exceptions are messages with retain: true. Most likely the only retained messages are discovery and availability ones and clients using those tend to resend them each time they connect to the broker.

Since I happen to use Zigbee2MQTT I can tell you unless you've customized your config to force it to send other messages with retain: true the only retained messages are availability and discovery. And Z2M definitely re-sends those on reconnect. Actually Z2M basically resends everything on connect since it keeps a DB of the last known state of all devices and resends that on reconnect too. So there's basically no risk of losing anything with Z2M.

So while I can't be 100% certain since I don't know all the clients you use I'd say its most likely that you don't lose anything by uninstalling and then reinstalling with the same config. Just a brief downtime for MQTT.

[camyb1981]

Finally now with the version 6.1.2 the installation was successful. In the end the update it did was from 6.0.2 --> 6.1.2 without any errors or issues. Now lets see if all still works.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.