Homeassistant sketch not publishing status back to MQTT topics

[sj-louw]

For some reason I am able to arm/disarm (write to panel) via Homeassistant and Homebridge, but nothing seems to be coming into the dsc/Get/ZoneX/# mqtt topics (reading from panel then posting to these zone topics), for zone status... Do you maybe have any idea...?

Further testing seems that the Arduino reads the incoming messages in the dsc/Set topic fine. It then successfully executes the panel "A", "S" and "D" commands, but never posts the status back to the dsc/Get topic so that armed status can be read by Homeassistant/Homebridge. The same goes for the dsc/Get/ZoneX topics. The Arduino does not publish zone updates there. I am testing with the sketch in pull request: #15

[taligentx]

Let's narrow this down to see if it's a decoding issue or MQTT issue. If you can, load the Status sketch and check the output over serial - are the zone states and armed/alarm status reported correctly?

If so, you can check the MQTT communication - I've used MQTTBox to simultaneously view all of the different topics and see what is happening.

Also, thanks for the PR! I'm catching up after holidays and will add the example - I may need to reorganize the examples to cover Arduino and esp8266 separately.

[sj-louw]

Hi.

I have loaded the Status sketch and tried again, but nothing is printed to the serial terminal. I went back, disconnected my circuit and verified everything. Even made sure the transistor is still working... Please see the pics attached of my wiring. Is it possible for you to verify that my wiring is correct?

1. The panel side, full circuit not drawn in:
   

2. My circuit, its soldered on a PCB, reason why it looks weird :)
   

3. My understanding of the circuit shown in the codes' README:
   

I will continue to fiddle around, but I guess the wiring needs to be in order before anything will work correctly. After the wiring is confirmed, I will continue with testing MQTT, etc.

[taligentx]

Thanks for the detailed info! It's very helpful - check the DSC green/data line in your second image. If the right side connections go to the Arduino, your image shows the 10k resistor connected to the panel/15k/ground. It should be connected to the Arduino/15k/ground instead.

It may also help troubleshooting to temporarily remove the transistor and the connection to the Arduino write pin - you only need the dscClockPin and dscReadPin connected to read from the panel.

[sj-louw]

Hi, I have modified my PCB as you mentioned, but still no go. I have from scratch build a new circuit, also, but as soon as I connect the "Write" wire to the panel, the keypad beeps and loose connection to the panel. Without the "Write" wire connected, it reads and prints to serial fine from the panel. The transistor Im using is 2N3904.

Is it not possible to reuse a pre-made and working circuit from a previous project (https://github.com/denvera/dscmod), as per screenshot below: (With that I could read and write to the panel) The only issue as far as I can see is that your circuit has dedicated Read and Write Pins on the Arduino, where the below seems to only use a single green wire?

[taligentx]

Good to hear that the clock and read lines are working on their own - are you seeing any further invalid data (CRC Error, 0x00 commands, etc)?

If all of the data is valid with the write pin disconnected, verify the wiring matches this for the 2N3904:

If you're willing to troubleshoot the issue, you can wire the writing connections directly without the PCB to ensure there isn't some other interaction involved. In the past I've plugged the base resistor directly into the Arduino, soldered the resistor directly to the transistor, and soldered the ground and data wires directly to the transistor hanging in the air.

You can also check voltages to make sure there isn't some problem with the components themselves:
Arduino dscWritePin to ground: 0v
2N3904 pin 3 to ground: 5v-13.6v (depends on how the multimeter reads the signal, one of mine reads ~8.5v)

I've considered using a level shifter - it is possible, but would at the very least require altering the code to invert the write signal. dscWritePin is normally low and goes high to write, which turns on the transistor and brings the data line low.

Reducing down to two pins is also possible with some rewriting. I'll add it to the todo list - at this point, the higher priorities are protocol decoding and integration with home automation. I like the idea of reducing the number of pins used, but I also like that the current implementation uses very common parts that anyone can find, even salvaging from scrap equipment.

[sj-louw]

Hi, thanks for helping to troubleshoot this. I have verified the transistor connections, and it is indeed correct. When reading, there is no more CRC errors. The last attempt from my side was a brand new circuit with new parts, without a PCB, Just loose wires hanging all over the show :)

Regarding to make use on one single PIN, I do understand your point, and I agree with you to keep it at two PINS and using easy to find parts.

On a side note, I think I might be on to something. It seems like when you use an Arduino ethernet shield the interrupt PIN is disconnected. Here is an explanation I found of what needs to be done to make it work. (http://garagebox.org/arduino/342-complete-soft-of-explaination-on-interrupt-imr-usage-of-ethernet-shield.html) Then also what seems to be important, is to make sure the sketch is written according to the "rules to write a sketch in IMR function", at the bottom of that post. Can you verify that your code follows that rules?

Then the sketch must be written as ISR function example: void ISR_function() { ..................... }, whatever this means... :)

If this is indeed the case, we have to remove my previous merge, as Arduino with Ethernet will not function then with the code as is.

So in short, it seems there must be two things in place for the Ethernet shield if your code depends on using hardware interrupts:

1. The hardware mod on the ethernet shield
2. The sketch must be written to make use of the hardware interrupts (#include <utility/w5100.h>)

And then the sketch: (I am not sure where the ISR_function must be used. Got a suspicion it might be in your actual library. Maybe you have more insight here?)

#include <SPI.h>
#include <Ethernet.h>
#include <utility/w5100.h> //Important if you want to use interrupt.
#include <PubSubClient.h>
#include <dscKeybusInterface.h>

// Set the MAC address
byte mac[] = {0x00, 0x1A, 0x11, 0xB2, 0xDE, 0x0B};

// Set fallback IP address if DHCP fails
IPAddress ip(169, 254, 254, 254);
// byte server[] = {172, 16, 100, 200};

// Set the broker server IP
const char* mqttServer = "";
const char* mqttUsername = "";
const char* mqttPassword = "";
const char* accessCode = "";  // An access code is required to disarm and night arm

const char* mqttClientName = "dscKeybusInterface";
const char* mqttPublishTopic = "dsc/Get";    // Sends partition armed and alarm status
const char* mqttSubscribeTopic = "dsc/Set";  // Receives messages to write to the panel
const char* mqttZoneTopic = "dsc/Get/Zone";  // Sends zone status - the zone number will be appended to this topic name: dsc/Get/Zone1 ... dsc/Get/Zone64
const char* mqttFireTopic = "dsc/Get/Fire";  // Sends fire status
unsigned long mqttPreviousTime;

EthernetClient ethClient;
PubSubClient mqtt(ethClient);

// Configures the Keybus interface with the specified pins - dscWritePin is
// optional, leaving it out disables the virtual keypad
#define dscClockPin 3
#define dscReadPin 4
#define dscWritePin 5
dscKeybusInterface dsc(dscClockPin, dscReadPin, dscWritePin);


void setup() {
  Serial.begin(115200);

  // get default IMR
  byte oldIMR = W5100.readIMR();
  Serial.print("Old IMR = ");
  Serial.println(oldIMR,HEX);
  
  // enable interrupts for all sockets
  W5100.writeIMR(0x0F);

  // read again to insure it worked
  byte newIMR = W5100.readIMR();
  Serial.print("New IMR = ");
  Serial.println(newIMR,HEX);

  // Interrupt 0, Digital Pin 2
  // Interrupt 1, Digital Pin 3
  attachInterrupt(1,ISR_function, MODE)

  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore.
    for (;;)
      ;
  }

  // print your local IP address:
  Serial.println(Ethernet.localIP());

  mqtt.setClient(ethClient);
  mqtt.setServer(mqttServer, 1883);
  mqtt.setCallback(mqttCallback);
  
  if (mqttConnect()) mqttPreviousTime = millis();
  else mqttPreviousTime = 0;

  // Starts the Keybus interface and optionally specifies how to print data.
  // begin() sets Serial by default and can accept a different stream: begin(Serial1), etc.
  dsc.begin();

  Serial.println(F("DSC Keybus Interface is online."));
}


void loop() {
  mqttHandle();

  if (dsc.handlePanel() && dsc.statusChanged) {  // Processes data only when a valid Keybus command has been read
    dsc.statusChanged = false;                   // Reset the status tracking flag
    
    // Sends the access code when needed by the panel for arming
    if (dsc.accessCodePrompt && dsc.writeReady) {
       dsc.accessCodePrompt = false;
       dsc.write(accessCode);
    }
    
    // If the Keybus data buffer is exceeded, the sketch is too busy to process all Keybus commands.  Call
    // handlePanel() more often, or increase dscBufferSize in the library: src/dscKeybusInterface.h
    if (dsc.bufferOverflow) Serial.println(F("Keybus buffer overflow"));
    dsc.bufferOverflow = false;

    // Publishes exit delay status
    if (dsc.exitDelayChanged) {
      dsc.exitDelayChanged = false;  // Resets the exit delay status flag
      if (dsc.exitDelay) mqtt.publish(mqttPublishTopic, "pending", true);  // Publish as a retained message
    }

    // Publishes armed status
    if (dsc.partitionArmedChanged) {
      dsc.partitionArmedChanged = false;  // Resets the partition armed status flag
      if (dsc.partitionArmed) {
        if (dsc.partitionArmedAway) mqtt.publish(mqttPublishTopic, "armed_away", true);
        else if (dsc.partitionArmedStay) mqtt.publish(mqttPublishTopic, "armed_home", true);
      }
      else mqtt.publish(mqttPublishTopic, "disarmed", true);
    }

    // Publishes alarm status
    if (dsc.partitionAlarmChanged) {
      dsc.partitionAlarmChanged = false;  // Resets the partition alarm status flag
      if (dsc.partitionAlarm) mqtt.publish(mqttPublishTopic, "triggered", true);
    }

    // Publishes the fire alarm status
    if (dsc.fireStatusChanged) {
      dsc.fireStatusChanged = false;                         // Resets the fire alarm status flag
      if (dsc.fireStatus) mqtt.publish(mqttFireTopic, "1");  // Fire alarm tripped
      else mqtt.publish(mqttFireTopic, "0");                 // Fire alarm restored
    }

    // Publishes zones 1-64 status in a separate topic per zone
    // Zone status is stored in the openZones[] and openZonesChanged[] arrays using 1 bit per zone, up to 64 zones:
    //   openZones[0] and openZonesChanged[0]: Bit 0 = Zone 1 ... Bit 7 = Zone 8
    //   openZones[1] and openZonesChanged[1]: Bit 0 = Zone 9 ... Bit 7 = Zone 16
    //   ...
    //   openZones[7] and openZonesChanged[7]: Bit 0 = Zone 57 ... Bit 7 = Zone 64
    if (dsc.openZonesStatusChanged) {
      dsc.openZonesStatusChanged = false;                           // Resets the open zones status flag
      for (byte zoneGroup = 0; zoneGroup < 8; zoneGroup++) {
        for (byte zoneBit = 0; zoneBit < 8; zoneBit++) {
          if (bitRead(dsc.openZonesChanged[zoneGroup], zoneBit)) {  // Checks an individual open zone status flag
            bitWrite(dsc.openZonesChanged[zoneGroup], zoneBit, 0);  // Resets the individual open zone status flag

            // Appends the mqttZoneTopic with the zone number
            char zonePublishTopic[strlen(mqttZoneTopic) + 2];
            char zone[3];
            strcpy(zonePublishTopic, mqttZoneTopic);
            itoa(zoneBit + 1 + (zoneGroup * 8), zone, 10);
            strcat(zonePublishTopic, zone);

            if (bitRead(dsc.openZones[zoneGroup], zoneBit)) {
              mqtt.publish(zonePublishTopic, "1", true);            // Zone open
            }
            else mqtt.publish(zonePublishTopic, "0", true);         // Zone closed
          }
        }
      }
    }

    mqtt.subscribe(mqttSubscribeTopic);
  }
}


// Handles messages received in the mqttSubscribeTopic
void mqttCallback(char* topic, byte* payload, unsigned int length) {

  // Handles unused parameters
  (void)topic;
  (void)length;

  // Arm stay
  if (payload[0] == 'S' && !dsc.partitionArmed && !dsc.exitDelay) {
    while (!dsc.writeReady) dsc.handlePanel();  // Continues processing Keybus data until ready to write
    dsc.write('s');  // Virtual keypad arm stay
  }

  // Arm away
  else if (payload[0] == 'A' && !dsc.partitionArmed && !dsc.exitDelay) {
    while (!dsc.writeReady) dsc.handlePanel();
    dsc.write('w');  // Virtual keypad arm away
  }

  // Disarm
  else if (payload[0] == 'D' && (dsc.partitionArmed || dsc.exitDelay)) {
    while (!dsc.writeReady) dsc.handlePanel();
    dsc.write(accessCode);
  }
}


void mqttHandle() {
  if (!mqtt.connected()) {
    unsigned long mqttCurrentTime = millis();
    if (mqttCurrentTime - mqttPreviousTime > 5000) {
      mqttPreviousTime = mqttCurrentTime;
      if (mqttConnect()) {
        Serial.println(F("MQTT disconnected, successfully reconnected."));
        mqttPreviousTime = 0;
      }
      else Serial.println(F("MQTT disconnected, failed to reconnect."));
    }
  }
  else mqtt.loop();
}


bool mqttConnect() {
  if (mqtt.connect(mqttClientName, mqttUsername, mqttPassword)) {
    Serial.print(F("MQTT connected: "));
    Serial.println(mqttServer);
    mqtt.subscribe(mqttSubscribeTopic);
  }
  else {
    Serial.print(F("MQTT connection failed: "));
    Serial.println(mqttServer);
  }
  return mqtt.connected();
}

[taligentx]

I'd like to narrow this down to see if there's a conflict with the W5100:

• Which Arduino board are you using?
• Which pins are you using?
• Are the voltages I listed earlier correct when using the Arduino board by itself with no other add ons and only USB, ground, and pins 3,4,5 connected? You could also try removing the USB and powering the Arduino directly with the panel in case there's some strange ground loop issue.

I suspect that when you are connecting the write pin, the transistor is turning on and pulling the data line low for some reason. Checking the voltages would help determine if this is the case.

The W5100 shield uses SPI alone by default and doesn't use the interrupt mode unless you've configured the board for it. I would start with SPI-only without the interrupt mode to get the basic connectivity working. I've previously successfully tested the Uno with the ENC26J60 ethernet module over SPI-only and the UIPEthernet library - the MQTT examples worked well and didn't show any problems over the few days I had it setup.

One possible conflict I saw is that the SD card on the shield uses pin 4 - the documentation states that pin 4 should be set high to disable the SD card on the shield. If so, you would need to use a different pin for dscReadPin.

[sj-louw]

@taligentx

• Which Arduino board are you using? Arduino UNO clone
• Which pins are you using? 3 (CLK), 4 (Read), 5 (Write)
• Tonight when I actually wanted to test the voltages, both my Arduino's decided they want to die :( I cannot communicate with them, trying to upload new sketches or even trying to re-burn the bootloader. So Im sort of stuck at this point :( (I have ordered some NodeMCU modules that should arrive in the week)

I am very keen to get this setup working, so I will be continuing when my ESPs arrive.

[sj-louw]

@taligentx

I have build a third circuit today, with brand new parts and Im getting the same result, as soon as I connect the custom circuit (No Arduino connected), the existing keypad looses connection.

The voltage readings before the circuit connected between panel ground and various panel pins:

• Green: 8.6v
• Yellow: 6.7v
• Red: 13.6v
• Black: 0v

The voltage readings through the custom circuit (dscKeybusInterface ref) between its ground and its various pins (While connected to the panel):

• "Arduino PIN 3" (Yellow/CLK): 0.5v
• "Arduino PIN 4" (Green/Read): 1.7v
• "Arduino PIN 5" (Green/Write): 0.5v

Note that no Arduino or ESP was connected to the above circuit during testing and the main keypad still lost connection and started to behave erratically.

For reference, here is a photo of my latest attempt to the circuit:

[taligentx]

I'll reproduce the setup without having an Arduino connected and see what happens here. You should connect the base resistor to ground (the write pin) - otherwise the transistor base pin is floating and would have undefined behavior.

[taligentx]

@sj-louw I've been testing BC547, BC549, and a few random NPN transistors connected to the PC1555MX and PC1864 to attempt to get the Keybus to react (even connecting the collector and emitter backwards) and haven't been able to replicate this - very strange.

To debug this, I recommend starting from single components (no circuit board) step by step and see exactly which step/component causes the problem - check the Keybus after every step and let's see where it is failing.

Panel connected to AC power, battery, and keypad only (no Arduino/esp8266):

1. Connect NPN collector to DSC green - check keybus
2. Connect NPN emitter to DSC Aux (-) - check keybus (and so on for each following step)
3. Connect NPN base to 1k resistor
4. Connect other end of 1k resistor to DSC Aux (-)
5. Connect a 15k resistor to DSC green
6. Connect the other end of the DSC green 15k resistor to a 10k resistor
7. Connect the other end of the DSC green 10k resistor to DSC Aux (-)
8. Connect a second 15k resistor to DSC yellow
9. Connect the other end of the DSC yellow 15k resistor to a second 10k resistor
10. Connect the other end of the DSC yellow 10k resistor to DSC Aux (-)

Hopefully this will narrow down the problem!

Edit: autocorrect typo

[rogueturnip]

Just a thought, if anyone out there has experience with KiCad, Eagle or some other circuit design program, it would be very beneficial to build a circuit and gerber that people could send off to some place like Seedstudio to get built. The ultimate would be with the design to have a socket to plug the MCU into.

[sj-louw]

@taligentx Thank you very much for the detailed info and effort in trying to debug. I will try again over the weekend and report back.

@rogueturnip I agree. We need to get some sort of Schematic in Gerber format for people (including myself :) ) to send off for manufacturing or to print themselves. This is one of the first things I would like to do, as soon as I can figure out the exact schematic, working with my 1864 panel.

[PascalSI]

I can easily make a scheme in Gerber format without any problems)))

[sj-louw]

@@taligentx I have done exactly step 1 - 10 as you described, and it did not interfere with the main keypad. I went ahead and connected it up to my nodeMCU, with the homeassistant sketch and the diagram now as seen below: Can you please verify where the part I marked as red must go?

Reading from panel now definitely works, as I see the Zone messages coming into MQTT and Homeassistant updates the separate zone statuses accordingly. If I try to arm the alarm from Homeassistant, I can see the MQTT messages going from Homeassistant into the dsc/Set topic, but the panel does not react on it.

Note: Last night I've tested with one of my previous versions of the circuit and could again only get writing to the panel working - no reading, so at least we know the MQTT setup is working as expected and can probably focus on the electronics to get both reading and writing at the same time going. Thanks

@PascalSI That would be great if you can make a Gerber schematic with a verified version of the circuit. It would be great if this can be designed for a single-sided PCB, so that people have the option to print it themselves too. Thanks.

[taligentx]

@sj-louw Great to hear! What is different now compared to last time when you just had a circuit attached and no Arduino where you saw the keypad have problems?

You can test the following (a few extra steps as a precaution):

1. Load the KeybusReader sketch to the NodeMCU
2. Disconnect the NodeMCU from power/USB
3. Disconnect the panel battery
4. Disconnect the panel power so it is powered down
5. Disconnect the end of the 1k resistor from Aux (-)
6. Connect the end of the 1k resistor to NodeMCU D8 (so D8 should only connect to the 1k resistor)
7. Connect the NodeMCU to USB, open serial monitor and verify the interface shows as online.
8. Connect the panel power and battery and see data shows up in KeybusReader without CRC errors or invalid commands like 0x00, etc.
9. Type a few different digits into serial monitor and see if they are recognized in the KeybusReader data

If this works, try Status-MQTT-HomeAssistant and let's see what happens. I know the step-by-step seems a bit pedantic but remote debugging is always an interesting challenge and I'm interested in seeing if there's any kind of design bug.

[sj-louw]

@taligentx Hi. I must admit, I took a bit of a shortcut with the last instructions. I unplugged the nodeMCU from the USB power and then only removed the 1K resistor from ground, leaving D8 connected to the 1K. Then I put the usb power back. Everything worked after that - read and write to the panel, and my main keypad is still functioning. Essentially the exact diagram you had in the README.

I have used different 10K resistors, although I have tested the previous ones. My solder joints was twisted together before I soldered with this last approach. These was the only differences.

Anyway, below is the exact working diagram and setup I have running now, with read and write to the panel working, using the Homeassistant sketch on the master branch. I have also set the MQTT_MAX_PACKET_SIZE in the PubSubClient Library from the default 128 to 256, as per https://github.com/knolleary/pubsubclient

Thanks again for your help.

[taligentx]

@sj-louw That's great to hear! For HomeAssistant, note that the default topic for alarm states will be changing in the next version from dsc/Get to dsc/Get/Partition1 (and with separate topics per partition up to dsc/Get/Partition8). Just a heads-up if you're on the master branch now and update in the future - I'll be merging develop to master once I get some feedback on how develop is working out.