Controller not reachable after update 1.0.0

[Bounties1337]

Add-on updated to 1.0.0.
http nor https onto controller possible.

However, Log looks good saying controller is starting... .

Hassio Log says the following:

Any Idea?

[bagg3rs]

I am getting the same error and logs after the upgrade also.

I reverted back to v0.23.2 version successfully, https://github.com/hassio-addons/addon-unifi/releases/tag/v0.23.2
Which states it's the last 32bit version for armhf, armv7 & i386.
There is a duplicate issue logged earlier also

[pro2call-nl]

Here also same issue.

HASSIO CORE core-2021.8.5
Supervisor supervisor-2021.06.8

64BIT OS Raspberry pi 4 8gb

restored the backup working OK now

[bitcrumb]

Same here, logs mention the unifi controller successfully started. No visible errors in the add-on logs.
But the controller is unreachable (I had/have https configured).

The "Log Viewer" does repeatedly show this error:
2021-08-13 00:15:48 ERROR (MainThread) [homeassistant.components.unifi] Error connecting to the UniFi controller at 192.168.1.4: Error requesting data from 192.168.1.4: Cannot connect to host 192.168.1.4:8443 ssl:default [Connect call failed ('192.168.1.4', 8443)]

edit
Reverting back to v0.23.2 version was succesful.

[coffeepotatochips]

version 0.23.2 broke the use of Let's Encrypt SSL and I had to uninstall unifi controller and disable SSL. A painful process as I had not made a backup of the config file and I have to reset my unifi devices for re-adoption.

For version 1.0.0, this broke the web UI access. Logs as follows:

:1023 Permanent error code on channel bind request: 403 - Forbidden IP
:1023 Permanent error code on channel bind request: 403 - Forbidden IP
:484 All TURN channel bindings failed

21-08-13 09:11:52 WARNING (MainThread) [supervisor.addons.options] Option 'ssl' does not exist in the schema for UniFi Controller (a0d7b954_unifi)
21-08-13 09:11:52 WARNING (MainThread) [supervisor.addons.options] Option 'certfile' does not exist in the schema for UniFi Controller (a0d7b954_unifi)
21-08-13 09:11:52 WARNING (MainThread) [supervisor.addons.options] Option 'keyfile' does not exist in the schema for UniFi Controller (a0d7b954_unifi)

Reverted back to version 0.23.2.

[PrathikGopal]

Same here with me, it fails to load UI after upgrade.
Reverted back from backup for now.

Tried to reinstall clean and restore from backup, the UI came back online but the login fails(even default).

[Tommatheussen]

I'm not sure if this is actually what made it work for me, but I went into the addon Configuration, selected to 'Edit in YAML'.
Removed the SSL and cert options there and saved. After that it came right back up.
The release notes state that SSL options are no longer there, maybe that caused a breaking loop or something.

[dikt1338]

Same here.

21-08-13 08:52:26 WARNING (MainThread) [supervisor.addons.options] Option 'certfile' does not exist in the schema for UniFi Controller (a0d7b954_unifi)
21-08-13 08:52:26 WARNING (MainThread) [supervisor.addons.options] Option 'keyfile' does not exist in the schema for UniFi Controller (a0d7b954_unifi)
21-08-13 08:52:26 WARNING (MainThread) [supervisor.addons.options] Option 'certfile' does not exist in the schema for UniFi Controller (a0d7b954_unifi)
21-08-13 08:52:26 WARNING (MainThread) [supervisor.addons.options] Option 'keyfile' does not exist in the schema for UniFi Controller (a0d7b954_unifi)
21-08-13 08:52:30 WARNING (MainThread) [supervisor.addons.options] Option 'certfile' does not exist in the schema for UniFi Controller (a0d7b954_unifi)
21-08-13 08:52:30 WARNING (MainThread) [supervisor.addons.options] Option 'keyfile' does not exist in the schema for UniFi Controller (a0d7b954_unifi)
21-08-13 08:53:07 INFO (SyncWorker_7) [supervisor.docker.interface] Stopping addon_a0d7b954_unifi application
21-08-13 08:53:12 INFO (SyncWorker_7) [supervisor.docker.interface] Cleaning addon_a0d7b954_unifi application
21-08-13 08:53:14 INFO (SyncWorker_5) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/unifi/aarch64 with version 1.0.0
21-08-13 08:53:26 INFO (SyncWorker_5) [supervisor.docker.interface] Stopping addon_a0d7b954_unifi application
21-08-13 08:53:30 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.RUNNING
21-08-13 08:53:30 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.SECURITY/ContextType.CORE
21-08-13 08:53:30 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.FREE_SPACE/ContextType.SYSTEM
21-08-13 08:53:30 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.PWNED/ContextType.ADDON
21-08-13 08:53:30 INFO (MainThread) [supervisor.resolution.check] System checks complete
21-08-13 08:53:30 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
21-08-13 08:53:30 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
21-08-13 08:53:30 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
21-08-13 08:53:30 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
21-08-13 08:53:31 INFO (SyncWorker_5) [supervisor.docker.interface] Cleaning addon_a0d7b954_unifi application
21-08-13 08:53:31 WARNING (SyncWorker_4) [supervisor.docker.interface] Can't grep logs from ghcr.io/hassio-addons/unifi/aarch64: 409 Client Error for http+docker://localhost/v1.41/containers/5b2720fe14599b9a1cb4817146ff1ade5f1f5041f99e18404e4f08add641a268/logs?stderr=1&stdout=1&timestamps=0&follow=0&tail=100: Conflict ("can not get logs from container which is dead or marked for removal")
21-08-13 08:53:46 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
21-08-13 08:53:46 INFO (SyncWorker_3) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/unifi/aarch64 with version 1.0.0
21-08-13 08:54:20 INFO (SyncWorker_6) [supervisor.docker.interface] Stopping addon_a0d7b954_unifi application
21-08-13 08:54:25 INFO (SyncWorker_6) [supervisor.docker.interface] Cleaning addon_a0d7b954_unifi application
21-08-13 08:54:27 INFO (SyncWorker_5) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/unifi/aarch64 with version 1.0.0
21-08-13 08:54:47 INFO (SyncWorker_5) [supervisor.docker.interface] Stopping addon_a0d7b954_unifi application
21-08-13 08:54:52 INFO (SyncWorker_5) [supervisor.docker.interface] Cleaning addon_a0d7b954_unifi application
21-08-13 08:54:54 INFO (SyncWorker_2) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/unifi/aarch64 with version 1.0.0
21-08-13 08:55:26 INFO (SyncWorker_1) [supervisor.docker.interface] Stopping addon_a0d7b954_unifi application
21-08-13 08:55:32 INFO (SyncWorker_1) [supervisor.docker.interface] Cleaning addon_a0d7b954_unifi application
21-08-13 08:55:34 INFO (SyncWorker_0) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/unifi/aarch64 with version 1.0.0
21-08-13 08:56:03 INFO (SyncWorker_4) [supervisor.docker.interface] Stopping addon_a0d7b954_unifi application
21-08-13 08:56:08 INFO (SyncWorker_4) [supervisor.docker.interface] Cleaning addon_a0d7b954_unifi application

[dikt1338]

reverted back, too.
If I could help with logs or test anything - let me know

[piouoso]

Hello,
same here, I tried also to edit the YAML and remove SSL and keyfile with no luck, I had to revert back to 0.23.2 after 10 minutes of real panic.

[marcomsousa]

same here, controller not reachable after update 1.0.0
I try to removed the SSL and cert options, but no lucky.

> [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
> [s6-init] ensuring user provided files have correct perms...exited 0.
> [fix-attrs.d] applying ownership & permissions fixes...
> [fix-attrs.d] done.
> [cont-init.d] executing container initialization scripts...
> [cont-init.d] 00-banner.sh: executing... 
> -----------------------------------------------------------
>  Add-on: UniFi Controller
>  Manage your UniFi network using a web browser
> -----------------------------------------------------------
>  Add-on version: 1.0.0
>  You are running the latest version of this add-on.
>  System: Home Assistant OS 6.2  (aarch64 / raspberrypi4-64)
>  Home Assistant Core: 2021.8.5
>  Home Assistant Supervisor: 2021.06.8
> -----------------------------------------------------------
>  Please, share the above information when looking for help
>  or support in, e.g., GitHub, forums or the Discord chat.
> -----------------------------------------------------------
> [cont-init.d] 00-banner.sh: exited 0.
> [cont-init.d] 01-log-level.sh: executing... 
> Log level is set to INFO
> [cont-init.d] 01-log-level.sh: exited 0.
> [cont-init.d] unifi.sh: executing... 
> [cont-init.d] unifi.sh: exited 0.
> [cont-init.d] done.
> [services.d] starting services
> [services.d] done.
> [15:17:00] INFO: Starting the UniFi Controller...

https://IP:8443 -> ERR_CONNECTION_REFUSED

[stevehop]

Restore the full Hass stack??? That's stupid. I just want the older Unifi version again (for which I also have a backup).

No, I clearly said a partial restore of a Home Assistant snapshot, that will allow you to restore just the old version of Unifi and your old config

[marcomsousa]

I see that in this new 1.0.0 release, mongodb was upgrade from 2.6.10-0ubuntu1[ubuntu repo] to 3.4.24 [Mongodb repo] (without be reflected on the release notes).

Could this be the problem? @frenck

0.23.2 version:
d583903#diff-aa3ca5069e32eb52aa889015b102e6bf275a3fc88789e6d699bd994e9a82ce81L29

1.0.0 version:
9592a78#diff-aa3ca5069e32eb52aa889015b102e6bf275a3fc88789e6d699bd994e9a82ce81R27

[frenck]

I've hidden all "I have the same issue" comments. Please use a 👍 on a comment or report. Those comments add nothing to debugging the issue at hand.

Additionally, this isn't a support forum either.

Thanks for understanding 👍

[frenck]

I see that in this new 1.0.0 release, mongodb was upgrade from 2.6.10-0ubuntu1[ubuntu repo] to 3.4.24 [Mongodb repo] (without be reflected on the release notes).

Could this be the problem? @frenck

Any change between the previous and last version could potentially be the problem. Could you explain why you conclude that is the problem?

[marcomsousa]

Checking the documentation and say that we need to perform more upgrades steps:

To upgrade an existing MongoDB deployment to 3.4, you must be running a 3.2-series release.

To upgrade from a version earlier than the 3.2-series, you must successively upgrade major releases until you have upgraded to 3.2-series. For example, if you are running a 3.0-series, you must 3.2 before you can upgrade to 3.4.

2.6 -> 3.0 upgrade to 3.0
3.0 -> 3.2 upgrade to 3.2
3.2 -> 3.4 upgrade to 3.4
3.4 -> 3.6 upgrade to 3.6

So we need to go first to 3.0?

[mpoli]

When I upgraded to 0.23.2 I was caught in the #235 and couldn't connect to the controller.

Installing a fresh 1.0.0 gives me a functional web interface, but trying to upgrade from 0.23.2 to 1.0.0 results in non functional web interface.

I did a manual restore from a home assistant add on backup (0.23.2), using method number 3 in this link: https://help.ui.com/hc/en-us/articles/204952144-UniFi-How-to-Create-and-Restore-a-Backup both preserving and replacing the the keystore file. All attempts result in:

root@a0d7b954-unifi:/data/unifi/data# cat /run/s6/services/unifi/logs/server.log
[2021-08-13T23:13:14,468] <localhost-startStop-1> INFO  Version - HV000001: Hibernate Validator 6.1.2.Final
[2021-08-13T23:13:15,270] <localhost-startStop-1> INFO  system - ======================================================================
[2021-08-13T23:13:15,273] <localhost-startStop-1> INFO  system - UniFi 6.2.26 (build atag_6.2.26_15319 - release/release) is started
[2021-08-13T23:13:15,274] <localhost-startStop-1> INFO  system - ======================================================================
[2021-08-13T23:13:15,279] <localhost-startStop-1> INFO  system - BASE dir:/usr/lib/unifi
[2021-08-13T23:13:15,424] <localhost-startStop-1> INFO  system - Current System IP: 172.30.33.4
[2021-08-13T23:13:15,424] <localhost-startStop-1> INFO  system - Hostname: a0d7b954-unifi.local.hass.io
[2021-08-13T23:13:15,429] <localhost-startStop-1> INFO  system - ubic.env: prod
[2021-08-13T23:13:16,065] <localhost-startStop-1> WARN  system - Valid keystore is missing or invalid. Generating one ...
[2021-08-13T23:13:16,066] <localhost-startStop-1> INFO  system - Generating Certificate[UniFi]... please wait...
[2021-08-13T23:13:17,211] <localhost-startStop-1> INFO  system - Certificate[UniFi] generated!
[2021-08-13T23:13:17,903] <localhost-startStop-1> INFO  db     - waiting for db connection...
[2021-08-13T23:13:19,636] <db-server> INFO  db     - DbServer stopped
[2021-08-13T23:13:19,637] <db-server> WARN  db     - DbServer not shutdown cleanly and need repairing on next startup
[2021-08-13T23:13:23,693] <db-server> ERROR system - [exec] error, rc=100, cmdline=[bin/mongod, --dbpath, /usr/lib/unifi/data/db, --port, 27117, --unixSocketPrefix, /usr/lib/unifi/run, --logRotate, reopen, --logappend, --logpath, /usr/lib/unifi/logs/mongod.log, --pidfilepath, /usr/lib/unifi/run/mongod.pid, --smallfiles, --bind_ip, 127.0.0.1, --repair]
[2021-08-13T23:13:25,350] <db-server> INFO  db     - DbServer stopped
[2021-08-13T23:13:29,406] <db-server> INFO  db     - DbServer stopped
[2021-08-13T23:13:33,464] <db-server> INFO  db     - DbServer stopped
[2021-08-13T23:13:37,521] <db-server> INFO  db     - DbServer stopped
[2021-08-13T23:13:41,576] <db-server> INFO  db     - DbServer stopped

So even trying to manually upgrade from a borked 0.23.2 to a fresh new 1.0.0 by simply copying the /data dir inside the docker volume results in non-functional and in no really useful log entries.

Could this be a mongodb version thing?

EDIT:
From /usr/lib/unifi/logs/mongod.log:

2021-08-14T12:53:50.427-0300 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27117, unixDomainSocket: { pathPrefix: "/usr/lib/unifi/run" } }, processManagement: { pidFilePath: "/usr/lib/unifi/run/mongod.pid" }, storage: { dbPath: "/usr/lib/unifi/data/db", mmapv1: { smallFiles: true } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/usr/lib/unifi/logs/mongod.log" } }
2021-08-14T12:53:50.432-0300 I -        [initandlisten] Detected data files in /usr/lib/unifi/data/db created by the 'mmapv1' storage engine, so setting the active storage engine to 'mmapv1'.
2021-08-14T12:53:50.432-0300 I STORAGE  [initandlisten] exception in initAndListen: 18656 Cannot start server with an unknown storage engine: mmapv1, terminating
2021-08-14T12:53:50.432-0300 I NETWORK  [initandlisten] shutdown: going to close listening sockets...

So Mongo engine missing from an old implementation compared to the new. Migration path issue?

[frenck]

Checking the documentation and say that we need to perform more upgrades steps:

According to the server itself, it can upgrade from 2.x -> 3.4 directly (which has been done for a lot of people. Considering 24 participants in this issue and over 5K instances already upgraded/migrated, I would think there is an edge case. One that happens and is real, but still...

As said, the upgrade path for MongoDB has been tested in different scenarios. In the end, is just odd in some cases it reports a missing mmapv1 engine, while that is still part of the 3.x MongoDB series (dropped in 4.x).

I'm in the dark at this point, and I'm really concerned about a path moving forward. I've broken my head over this for days to figure out the current path and getting things reliable. With this, as a result, I'm afraid it starts to look like it isn't something that can be fixed.

Honestly, if anybody from Ubiquiti reads this: Get rid of MongoDB or at least try to keep up (oh and if you are at it... time ditch Java 8, originating from 2014, as well...) 😢

[pro2call-nl]

I had some free time today.

what i have done:

1. made a backup in the unifi controller
2. removed 0.23.2
3. installed 1.0.0
4. restored backup.

All OK now.

this is maybe the easiest step for the 24 users

[marcomsousa]

@frenck what storageEngine you use with the in 3.4 version, MMAPv1 or WiredTiger?

[frenck]

@marcomsousa That is controlled by UniFi, it loads and configures MongoDB (its not a separate process)

[marcomsousa]

@marcomsousa That is controlled by UniFi, it loads and configures MongoDB (its not a separate process)

When I use the Unifi addon 1.0.0 from scratch, the storageEngine used are the wireTiger.

2021-08-16T15:15:10.442+0200 I - [initandlisten] Detected data files in /usr/lib/unifi/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

I think the mongodb 3.4.24 ARM64-bit from Mongo repo doesn't have included the mmapv1 storageEngine.

@frenck if true, and if you want to reproduce our issues, you can do this:

1. Install pi with 32-bit version of home assistant
2. Install Addon Unifi 0.23.2 using 32-bit. ('wiredTiger' isn't support in 32-bit mongodb version, so will use the mmapv1 storageEngine).
3. Backup home-assistant and download snapshot
4. Install pi with 64-bit version of home assistant
5. Restore backup of home-assistant (everything will work)
6. Update Addon Unifi from 0.23.2 to 1.0.0

This will reproduce our error because mongodb ARM64-bit, don't have include by default the mmapv1 storageEngine.
If true, the fix is to build/compile mongodb 3.4.24 ARM 64-bit from sources with mmapv1 storageEngine enable.

[marcomsousa]

Who is using HA 64-bit restored from an old HA 32-bit version? I think this will 100% the users that are complaining.

[bitcrumb]

@marcomsousa Now that you mention it. I indeed restored a Home Assisant setup from a 32-bit Home Assistant OS onto a fresh 64 bit HA OS version. Since I wanted to be able to upgrade to Unifi Add-on 1.0.0.

[bcutter]

Who is using HA 64-bit restored from an old HA 32-bit version? I think this will 100% the users that are complaining.

It depends, tending to say "no that's not the case":

My whole HA instance was running on 32 bit (Pi 3B+ hardware) in the beginning before I updated to HASS OS 64 bit on Pi 4 (migrated restoring a full snapshot). BUT I didn't use Unifi addon before that migration.

[piouoso]

Who is using HA 64-bit restored from an old HA 32-bit version? I think this will 100% the users that are complaining.

My HA is native 64 bit (PI4).
Anyway, I was able to do a clean install and restore the backup.
Now version 1 is working!

[Alexthestampede]

I'm also having this problem but I'd disagree on the idea of the 32 bit to 64 being the issue, I bought and setup from zero a Pi 4 specifically for the UniFi controller, so there wasn't a previous HA to restore :)

[tykeal]

I never ran an HA32, I've always been doing HA64 ever since I setup my Pi4 8GB which is the first time I installed this addon.

After futzing around with getting NGinx Proxy Manager working so I could deal with a new subdomain along with my HSTS configuration, I too got this working by doing an uninstall, re-install + backup restore. I also had to rest my integration since everything changed with having to setup a new subdomain :-/

[frenck]

I've locked down the issue, as people keep posting unrelated stuff here. This is not about an issue with other error messages, this issue is also not a support forum on how Home Assistant snapshots/backups work. If you have this exact issue, please upvote the issue instead of writing "Me too" response as well. This is not unique to this project or issue, this is how generally GitHub works.

It is super distracting to find solutions and focus on the problem at hand with all that noise. I've asked a couple of times nows, unfortunately, it is ignored leaving me no other choice than to stop this madness.

Before you downvote this comment: You don't have to agree and I don't care if you don't.

../Frenck

[frenck]

This issue is way overdue, and closing it as won't fix at this point.