Official MQTT - Nuki Hub coexistence, keypad/timecontrol sensor per c…

…ode/entry, option not to publish config, authorization log improvements, various fixes (#389)

* Coexistence with official MQTT over Wifi and Thread

* Coexistence with official MQTT over Wifi and Thread

* Arduino Core 2.0.17 cmake and README

* Coexistence with official MQTT over Wifi and Thread

* Keep updating status until state is known

* Coexistence with official MQTT over Wifi and Thread

Docker/Dockerfile_cmake

@@ -10,7 +10,7 @@ RUN tar -xf /tmp/arduino-ide.tar.xz --directory ~/
 RUN cd ~/arduino* && \
     ./install.sh && \
     ./arduino --pref "boardsmanager.additional.urls=https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json" --save-prefs && \
-    ./arduino --install-boards esp32:esp32:2.0.15
+    ./arduino --install-boards esp32:esp32:2.0.17
 
 RUN git clone --recurse-submodules https://github.com/technyon/Arduino-CMake-Toolchain.git ~/Arduino-CMake-Toolchain
 
@@ -40,6 +40,6 @@ FROM builder AS runtime
 
 COPY --from=builder /usr/src/nuki_hub/build/nuki_hub.bin /usr/src/nuki_hub/build/release/nuki_hub.bin
 COPY --from=builder /usr/src/nuki_hub/build/nuki_hub.partitions.bin /usr/src/nuki_hub/build/release/nuki_hub.partitions.bin
-COPY --from=builder /root/.arduino15/packages/esp32/hardware/esp32/2.0.15/tools/partitions/boot_app0.bin /usr/src/nuki_hub/build/release/boot_app0.bin
+COPY --from=builder /root/.arduino15/packages/esp32/hardware/esp32/2.0.17/tools/partitions/boot_app0.bin /usr/src/nuki_hub/build/release/boot_app0.bin
 
 CMD ["/bin/bash"]

HYBRID.md

@@ -0,0 +1,50 @@
+### Hybrid mode combining official MQTT implementation and Nuki Hub ###
+
+The purpose of this mode is to have Nuki Hub work in conjunction with the official MQTT implementation by Nuki.
+
+### Requirements ###
+
+- ESP32 running Nuki Hub 8.35/8.36 or higher
+- For WiFi: Nuki lock 3.0 Pro or Nuki Lock 4.0 Pro
+- For Thread: Nuki Lock 4.0 or Nuki Lock 4.0 Pro. Note that you do ***NOT*** need to buy the remote access addon for the Nuki Lock 4.0
+- For Thread: The Nuki Lock needs to have network access to the same MQTT server as the one that Nuki Hub is conected to. Depending if the MQTT server is reachable over IPv6 you might need an OpenThread Border router that supports NAT64 and has this enabled. Currently this means an Apple Device or Home Assistant with the Matter server and OpenThread Border Router
+- The Nuki Opener does not have WiFI or Thread and thus doesn't benefit from the hybrid solutions added speed. You can however use and connect a Nuki Opener as usual which will function over regular BLE and can still connect Nuki Hub as a bridge to an Opener.
+
+### Resources ###
+
+- Discussion about setting up Nuki and Thread on the Nuki Developer forum: https://developer.nuki.io/t/smart-lock-4th-generation-set-up-home-assistant-for-remote-access-via-thread/25181
+- Nuki KB article on MQTT: https://support.nuki.io/hc/en-us/articles/14052016143249-Activate-MQTT-via-the-Nuki-app
+- Nuki KB article on Thread: https://support.nuki.io/hc/en-us/articles/18155425155217-Requirements-for-Remote-Access-via-Thread
+
+### Reasons for using Hybrid mode combining official MQTT implementation and Nuki Hub
+
+| Setup                                       | Speed         | Functionality  | Battery life | 
+|---------------------------------------------|---------------|----------------|--------------|
+| Nuki Hub paired as Bridge                   | --            | +++            | +++          |
+| Nuki Hub paired as App                      | ++            | +++            | ---          |
+| Matter over Thread                          | +++           | -              | ++           |
+| Official MQTT over WiFi                     | +++           | +              | -            |
+| Official MQTT over Thread                   | +++           | +              | ++           |
+| Hybrid Official MQTT over WiFi + Nuki Hub   | +++           | +++            | --           |
+| Hybrid Official MQTT over Thread + Nuki Hub | +++           | +++            | ++           |
+
+The Hybrid Official MQTT over Thread + Nuki Hub solution allows for the best combination of state change speed, lock action execution, functionality and battery life.
+
+### Setup ###
+
+- Follow the official instruction by Nuki on setting up MQTT over Thread or WiFi on https://support.nuki.io/hc/en-us/articles/14052016143249-Activate-MQTT-via-the-Nuki-app.
+  - Make sure to connect to the same MQTT server as Nuki Hub
+  - Make sure ***NOT*** to enable `Auto discovery`, Home Assistant will be setup using Nuki Hub
+  - Optionally enable `Allow locking`. Note that if you enable this setting it is preferred to set ACL on your MQTT broker to only allow the Nuki lock and Nuki Hub MQTT user access to the topic `nuki/NUKI-ID/lockAction` to make sure that only Nuki Hub can execute commands on the lock (otherwise ACL settings through Nuki Hub can not be 100% enforced)
+  - Make sure that MQTT is setup correctly by checking if you get a green check mark in the Nuki app
+
+- Install Nuki Hub 8.35 or higher on a supported ESP32 device
+- Make sure you are not paired as a bridge. Unpair your Nuki lock in Nuki Hub if Nuki Hub was paired as a bridge (this is mandatory even if you removed the bridge connection from the Nuki lock).
+- Enable `Enable hybrid official MQTT and Nuki Hub setup`. The `Lock: Nuki Bridge is running alongside Nuki Hub (needs re-pairing if changed)` setting will be automatically be enabled.
+- Optionally enable `Enable sending actions through official MQTT`, if not enabled lock actions will be sent over BLE as usual (slower)
+- Set `Time between status updates when official MQTT is offline (seconds)` to a positive integer. If the Nuki lock MQTT connection goes offline for whatever reason Nuki Hub will update the lock state with the set interval in seconds.
+- Optionally enable `Retry command sent using official MQTT over BLE if failed`. If sending a lock action over the official MQTT implementation fails the command will be resent over BLE if this is enabled. Requires `Enable sending actions through official MQTT` to be enabled.
+- Save your configuration
+- Consider setting the `Query intervals` on the `Advanced Nuki configuration` to high numbers (e.g. 86400) to further reduce battery drain.
+- Pair your Nuki Lock with Nuki Hub
+- Test that state changes are recieved and processed by Nuki Hub by looking at the MQTT topics using an application like `MQTT Explorer`

README.md

@@ -15,12 +15,12 @@ Feel free to join us on Discord: https://discord.gg/9nPq85bP4p
 ## Supported devices
 
 <b>Supported ESP32 devices:</b>
-- All ESP32 models with WIFI and BLE which are supported by Arduino Core 2.0.15 should work. Tested builds are provided for the ESP32, ESP32-S3 and ESP32-C3.
+- All ESP32 models with WIFI and BLE which are supported by Arduino Core 2.0.17 should work. Tested builds are provided for the ESP32, ESP32-S3 and ESP32-C3.
 - Untested builds are provided for the ESP32-Solo1.
 
 <b>Not supported ESP32 devices:</b>
 - The ESP32-S2 has no BLE and as such can't run Nuki Hub.
-- The ESP32-C6 and ESP32-H2 are not supported by Arduino Core 2.0.15 and as such Nuki Hub is not compiled against these targets (at this time).
+- The ESP32-C6 and ESP32-H2 are not supported by Arduino Core 2.0.17 and as such Nuki Hub is not compiled against these targets (at this time).
 
 <b>Supported Nuki devices:</b>
 - Nuki Smart Lock 1.0
@@ -81,9 +81,14 @@ When pairing is successful, the web interface should show "Paired: Yes" (it migh
 MQTT nodes like lock state and battery level should now reflect the reported values from the lock.<br>
 <br>
 <b>Note: It is possible to run Nuki Hub alongside a Nuki Bridge.
-This is not recommended and will lead to excessive battery drain and can lead to either device missing updates.
+This is not recommended (unless when using [hybrid mode](/HYBRID.md)) and will lead to excessive battery drain and can lead to either device missing updates.
 Enable "Register as app" before pairing to allow this. Otherwise the Bridge will be unregistered when pairing the Nuki Hub.</b>
 
+## Hybrid mode
+
+Hybrid mode allows you to use the official Nuki MQTT implemenation on a Nuki Lock 3.0 Pro, Nuki Lock 4.0 or Nuki Lock 4.0 Pro in conjunction with Nuki Hub.<br>
+See [hybrid mode](/HYBRID.md) for more information.
+
 ## Support
 
 If you haven't ordered your Nuki product yet, you can support me by using my referrer code when placing your order:<br>
@@ -124,6 +129,9 @@ In a browser navigate to the IP address assigned to the ESP32.
 - Enable MQTT logging: Enable to fill the maintenance/log MQTT topic with debug log information.
 - Check for Firmware Updates every 24h: Enable to allow the Nuki Hub to check the latest release of the Nuki Hub firmware on boot and every 24 hours. Requires the Nuki Hub to be able to connect to github.com. The latest version will be published to MQTT and will be visible on the main page of the Web Configurator.
 - Disable some extraneous non-JSON topics: Enable to not publish non-JSON keypad and config MQTT topics.
+- Enable hybrid official MQTT and Nuki Hub setup: Enable to combine the official MQTT over Thread/WiFi with BLE. Improves speed of state changes. Needs the official MQTT to be setup first. Also requires Nuki Hub to be paired as app and unregistered as a bridge using the Nuki app. See [hybrid mode](/HYBRID.md)
+- Enable sending actions through official MQTT: Enable to sent lock actions through the official MQTT topics (e.g. over Thread/WiFi) instead of using BLE. Needs "Enable hybrid official MQTT and Nuki Hub setup" to be enabled. See [hybrid mode](/HYBRID.md)
+- Time between status updates when official MQTT is offline (seconds): Set to a positive integer to set the maximum amount of seconds between actively querying the Nuki lock for the current lock state when the official MQTT is offline, default 600.
 
 #### IP Address assignment
 
@@ -150,13 +158,15 @@ In a browser navigate to the IP address assigned to the ESP32.
 - Query interval keypad (Only available when a Keypad is detected): Set to a positive integer to set the maximum amount of seconds between actively querying the Nuki device for the current keypad state, default 1800.
 - Number of retries if command failed: Set to a positive integer to define the amount of times the Nuki Hub retries sending commands to the Nuki Lock or Opener when commands are not acknowledged by the device, default 3.
 - Delay between retries: Set to the amount of milliseconds the Nuki Hub waits between resending not acknowledged commands, default 100.
-- Nuki Bridge is running alongside Nuki Hub: Enable to allow Nuki Hub to co-exist with a Nuki Bridge by registering Nuki Hub as an (smartphone) app instead of a bridge. Changing this setting will require re-pairing. Enabling this setting is strongly discouraged as described in the "[Pairing with a Nuki Lock or Opener](#pairing-with-a-nuki-lock-or-opener)" section of this README
+- Lock: Nuki Bridge is running alongside Nuki Hub: Enable to allow Nuki Hub to co-exist with a Nuki Bridge by registering Nuki Hub as an (smartphone) app instead of a bridge. Changing this setting will require re-pairing. Enabling this setting is strongly discouraged as described in the "[Pairing with a Nuki Lock or Opener](#pairing-with-a-nuki-lock-or-opener)" section of this README, ***unless when used in [Hybrid mode](/HYBRID.md)e (Official MQTT / Nuki Hub co-existance)***
+- Opener: Nuki Bridge is running alongside Nuki Hub: Enable to allow Nuki Hub to co-exist with a Nuki Bridge by registering Nuki Hub as an (smartphone) app instead of a bridge. Changing this setting will require re-pairing. Enabling this setting is strongly discouraged as described in the "[Pairing with a Nuki Lock or Opener](#pairing-with-a-nuki-lock-or-opener)" section of this README
 - Presence detection timeout: Set to a positive integer to set the amount of seconds between updates to the presence/devices MQTT topic with the list of detected bluetooth devices, set to -1 to disable presence detection, default 60.
 - Restart if bluetooth beacons not received: Set to a positive integer to restart the Nuki Hub after the set amount of seconds has passed without receiving a bluetooth beacon from the Nuki device, set to -1 to disable, default 60. Because the bluetooth stack of the ESP32 can silently fail it is not recommended to disable this setting.
 
 ### Access Level Configuration
 
 #### Nuki General Access Control
+- Publish Nuki configuration information: Enable to publish information about the configuration of the connected Nuki device(s) through MQTT.
 - Publish keypad entries information (Only available when a Keypad is detected): Enable to publish information about keypad codes through MQTT, see the "[Keypad control](#keypad-control-optional)" section of this README
 - Also publish keypad codes (Only available when a Keypad is detected): Enable to publish the actual keypad codes through MQTT, note that is could be considered a security risk
 - Add, modify and delete keypad codes (Only available when a Keypad is detected): Enable to allow configuration of keypad codes through MQTT, see the "[Keypad control](#keypad-control-optional)" section of this README
@@ -271,6 +281,7 @@ In a browser navigate to the IP address assigned to the ESP32.
 - battery/maxTurnCurrent: The highest current of the turn motor during the last lock action (A) (Lock only).
 - battery/lockDistance: The total distance during the last lock action in centidegrees (Lock only).
 - battery/keypadCritical: 1 if the battery level of a connected keypad is critical, otherwise 0.
+- battery/doorSensorCritical (only available in hybdrid mode): 1 if the battery level of a connected doorsensor is critical, otherwise 0.
 - battery/basicJson: The current battery state (critical, charging, level and keypad critical) of the Nuki Lock/Opener as JSON data.
 - battery/advancedJson: : The current battery state (critical, batteryDrain, batteryVoltage, lockAction, startVoltage, lowestVoltage, lockDistance, startTemperature, maxTurnCurrent and batteryResistance) of the Nuki Lock/Opener as JSON data.
 
@@ -478,9 +489,9 @@ To change Nuki Lock/Opener keypad settings set the `keypad/actionJson` topic to
 | action           | Required | Required | Required | The action to execute                                                                                            | "delete", "add", "update"              |
 | codeId           | Required | Not used | Required | The code ID of the existing code to delete or update                                                             | Integer                                |
 | code             | Not used | Required | Optional | The code to create or update                                                                       | 6-digit Integer without zero's, can't start with "12"|
-| enabled          | Not used | Not used | Optional | Enable or disable the code, always enabled on add, disabled if not set on update                                 | 1 = enabled, 0 = disabled              |
-| name             | Not used | Required | Required | The name of the code to create or update                                                                         | String, max 20 chars                   |
-| timeLimited      | Not used | Optional | Optional | If this authorization is restricted to access only at certain times, disabled if not set (requires enabled = 1)  | 1 = enabled, 0 = disabled              |
+| enabled          | Not used | Not used | Optional | Enable or disable the code, always enabled on add                                                                | 1 = enabled, 0 = disabled              |
+| name             | Not used | Required | Optional | The name of the code to create or update                                                                         | String, max 20 chars                   |
+| timeLimited      | Not used | Optional | Optional | If this authorization is restricted to access only at certain times, requires enabled = 1                        | 1 = enabled, 0 = disabled              |
 | allowedFrom      | Not used | Optional | Optional | The start timestamp from which access should be allowed (requires enabled = 1 and timeLimited = 1)               | "YYYY-MM-DD HH:MM:SS"                  |
 | allowedUntil     | Not used | Optional | Optional | The end timestamp until access should be allowed (requires enabled = 1 and timeLimited = 1)                      | "YYYY-MM-DD HH:MM:SS"                  |
 | allowedWeekdays  | Not used | Optional | Optional | Weekdays on which access should be allowed (requires enabled = 1 and timeLimited = 1)     | Array of days: "mon", "tue", "wed", "thu" , "fri" "sat", "sun"|
@@ -682,11 +693,11 @@ Navigate to "Nuki Configuration" and change the "MQTT Nuki Smartlock Path" or "M
 
 ### The Nuki battery is draining quickly.
 
-This often is a result of enabling "Register as app".<br>
+This often is a result of enabling "Register as app" when not using [Hybrid mode](/HYBRID.md) (Official MQTT / Nuki Hub co-existance).<br>
 Doing so will cause Nuki Hub to constantly query the lock and as such cause excessive battery drain.<br>
 To prevent this behaviour, unpair Nuki Hub, disable "Register as app", and re-pair.<br>
 <br>
-<b>Never enable "Register as app" unless you intend to use a Nuki Bridge in addition to Nuki Hub!</b>
+<b>Never enable "Register as app" unless you intend to use a Nuki Bridge in addition to Nuki Hub or you are using Hybrid mode!</b>
 
 ## Building from source
 <b>Docker (Preferred)</b><br>

src/MqttTopics.h

@@ -24,6 +24,19 @@
 #define mqtt_topic_lock_address "/lock/address"
 #define mqtt_topic_lock_retry "/lock/retry"
 
+#define mqtt_topic_official_lock_action "/lockAction"
+//#define mqtt_topic_official_mode "/mode"
+#define mqtt_topic_official_state "/state"
+#define mqtt_topic_official_batteryCritical "/batteryCritical"
+#define mqtt_topic_official_batteryChargeState "/batteryChargeState"
+#define mqtt_topic_official_batteryCharging "/batteryCharging"
+#define mqtt_topic_official_keypadBatteryCritical "/keypadBatteryCritical"
+#define mqtt_topic_official_doorsensorState "/doorsensorState"
+#define mqtt_topic_official_doorsensorBatteryCritical "/doorsensorBatteryCritical"
+#define mqtt_topic_official_connected "/connected"
+#define mqtt_topic_official_commandResponse "/commandResponse"
+#define mqtt_topic_official_lockActionEvent "/lockActionEvent"
+
 #define mqtt_topic_config_action "/configuration/action"
 #define mqtt_topic_config_action_command_result "/configuration/commandResult"
 #define mqtt_topic_config_basic_json "/configuration/basicJson"
@@ -50,10 +63,12 @@
 #define mqtt_topic_battery_max_turn_current "/battery/maxTurnCurrent"
 #define mqtt_topic_battery_lock_distance "/battery/lockDistance"
 #define mqtt_topic_battery_keypad_critical "/battery/keypadCritical"
+#define mqtt_topic_battery_doorsensor_critical "/battery/doorSensorCritical"
 #define mqtt_topic_battery_basic_json "/battery/basicJson"
 #define mqtt_topic_battery_advanced_json "/battery/advancedJson"
 
 #define mqtt_topic_keypad "/keypad"
+#define mqtt_topic_keypad_codes "/keypad/codes"
 #define mqtt_topic_keypad_command_action "/keypad/command/action"
 #define mqtt_topic_keypad_command_id "/keypad/command/id"
 #define mqtt_topic_keypad_command_name "/keypad/command/name"
@@ -64,6 +79,8 @@
 #define mqtt_topic_keypad_json_action "/keypad/actionJson"
 #define mqtt_topic_keypad_json_command_result "/keypad/commandResultJson"
 
+#define mqtt_topic_timecontrol "/timecontrol"
+#define mqtt_topic_timecontrol_entries "/timecontrol/entries"
 #define mqtt_topic_timecontrol_json "/timecontrol/json"
 #define mqtt_topic_timecontrol_action "/timecontrol/action"
 #define mqtt_topic_timecontrol_command_result "/timecontrol/commandResult"