[BUG] MQTT connection fails #623

tiger42 · 2024-02-29T15:09:41Z

BSB-LAN Version
#define MAJOR "3"
#define MINOR "3"
#define PATCH "3"
#define COMPILETIME "20240227001351"

Architecture
Arduino Due

Bus system
BSB

Describe the bug
Since upgrading to the latest version (any commit after 32eeec3) the connection to my MQTT broker fails with the message "Failed to connect to MQTT broker, retrying...".
Of course I have changed the connection settings in the BSB_LAN_config.h to match the new config scheme.
Before (v3.3.2): byte mqtt_broker_ip_addr[4] = {192,168,0,1};
After (v3.3.3): char mqtt_broker_addr[33] = "192.168.0.1";
I've also tried to explicitly specify the port (char mqtt_broker_addr[33] = "192.168.0.1:1883";), but that doesn't make any difference.
I've switched back and forth between the versions, and I am sure that the problem was introduced with one of the MQTT related commits on Feb 26th.

Log files - Bug reports without log files will be closed

Client ID: BSB-LAN
Will topic: bsb-lan/status
Failed to connect to MQTT broker, retrying...
Failed to connect to MQTT broker, retrying...
Failed to connect to MQTT broker, retrying...

Expected behavior
The connection should not fail.

Attach your BSB_LAN_config.h file to your bug report (remove any confidential information if necessary). Do not ZIP or otherwise compress it. Bug reports without this file attached will be closed immediately.

/************************************************************************************/
/************************************************************************************/
/* Settings -   BEGIN                                                               */
/************************************************************************************/
/************************************************************************************/

// Upon first installation, rename this file from BSB_lan_config.h.default to BSB_lan_config.h and adjust settings accordingly
// Users who still use Arduino Mega2560 boards instead of Arduin Due: Please have a look at the very end of this configuration
// where certain modules are disabled by default due to limited flash memory.


/* Select language; so far German is the most complete, with English following.
 * Available languages are: Czech (CS), German (DE), Danish (DA), English (EN), Spanish (ES), Finnish (FI),
 * French (FR), Greek (EL), Hungarian (HU), Italian (IT), Dutch (NL), Polish (PL), Russian (RU), Swedish (SV),
 * Slovenian (SI) and Turkish (TR).
 * Incomplete languages will automatically be filled up with English translations first, and if no English translation
 * is available, fallback will take place to German.
*/
#define LANG DE

/*
Allow to initialize program configuration by reading settings from EEPROM
byte UseEEPROM = 0; // Configuration is read from this config file.
                    // Configuration can be stored in EEPROM but will not be used while UseEEPROM is zero.
                    // Set zero for fallback startup in case EEPROM configuration is broken.
byte UseEEPROM = 1; // Configuration will be read from EEPROM. This is the default.
*/
byte UseEEPROM = 0;

/*
 *  Enter a MAC address, found either on the EthernetShield or use the one below.
 *  Change this if you have more than one BSB-LAN adapter in your LAN, so that there aren't any address conflicts.
 *  MAC address here only affects the LAN shield, it doesn't apply to the WiFi-ESP-solution.
*/
byte mac[6] = { 0x00, 0x80, 0x41, 0x19, 0x69, 0x90 };

/*
 * Initialize the Ethernet server library
 * with the IP address and port you want to use
 * (port 80 is default for HTTP):
*/

uint8_t network_type = 0;               // Set to LAN (0) when using Ethernet connection. Set to WLAN (1) when using WiFi. Arduino users using WiFiSpi have to activate the definement below as well.
uint16_t HTTPPort = 80;
bool useDHCP = false;                   // Set to false if you want to use a fixed IP.
byte ip_addr[4] = {192,168,0,95};       // Please note the commas instead of dots!!!  Set useDHCP to true if you want DHCP.
byte gateway_addr[4] = {192,168,0,100}; // Gateway address. This is usually your router's IP address. Please note the commas instead of dots!!! Ignored if first value is 0.
byte dns_addr[4] = {192,168,0,100};     // DNS server. Please note the commas instead of dots!!! Ignored if first value is 0.
byte subnet_addr[4] = {255,255,255,0};  // Subnet address. Please use commas instead of dots!!! Ignored if first value is 0.

char wifi_ssid[32] = "YourWiFiNetwork"; // enter your WiFi network name (SSID) here
char wifi_pass[64] = "YourWiFiPassword";// enter your WiFi password here
uint8_t bssid[6] = {0x00, 0x00, 0x00, 0x00, 0x00, 0x00};  // enter specific BSSID address here to ensure connecting to a specific router. Leave all zeros in normal circumstances.

//#define WIFISPI                         // Activate this on the Arduino to enable WiFi via WiFiSpi. DO NOT enable this on an ESP32.
#define WIFI_SPI_SS_PIN 12              // defines SPI-SS pin for Arduino-ESP8266 connection

#define MDNS_SUPPORT                   // Advertises the hostname in the local network. Disable this if you don't want your device to be found under this name in your network.
char mDNS_hostname[32] = "BSB-LAN";

/* NTP settings to acquire exact date and time via network.
 * Attention: This only works with ESP32 microcontrollers so far!
 * Use pool.ntp.org if your BSB-LAN installation can access the internet.
 * Otherwise you may also use your router's address if it can act as a NTP server.
 * The default timezone "CET-1CEST,M3.5.0,M10.5.0/3" covers most Central European countries (GMT+1) and takes care of daylight saving.
 * Use "EET-2EEST,M3.5.0/3,M10.5.0/4" for Eastern European countries (GMT+2), or
 * use "WET0WEST,M3.5.0/1,M10.5.0" for Western European countries (GMT+0).
 * See here for a full list of timezones for places all over the world:
 * https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv
*/
#define USE_NTP       // Disable this in case you don't want to use NTP
const char ntp_server[20] = "pool.ntp.org";
const char local_timezone[30] = "CET-1CEST,M3.5.0,M10.5.0/3";

// Debug options
#define DEBUG         // Compile with verbose DEBUG module if defined
byte debug_mode = 1;  // Debug mode: 0 - disabled, 1 - send debug messages to serial interface, 2 - send debug messages to telnet client
byte verbose = 0;     // If set to 1, all messages on the bus are printed to debug interface
byte monitor = 0;     // Bus monitor mode. This is only necessary for in-depth debug sessions.
bool show_unknown = true; // true - show all parameters, false - hide unknown parameters from web display (parameters will still be queried and therefore take time!)
//#define DEVELOPER_DEBUG //Very verbose debug output. This option will dramatically slow down MCU performance. Useful for developers not for users.

/* SECURITY OPTIONS
 * There are several options to control and protect access to your heating system. However, keep
 * in mind, that even activating all three options are no guarantee that a versatile intruder with
 * access to your (W)LAN won't be able to gain access. In any case, no encryption of data streams
 * is provided from the Arduino itself. Use VPN or a SSL proxy if that is a must for you and connect
 * the Arduino wired to the VPN server or SSL proxy. On the other hand, someone with this amount
 * of criminal activity will probably have it easier just to access your heating system face-to-face ;)
*/

/*
 * if PASSKEY length is non-zero, the URL has to contain the defined passkey as first element
 * e.g.
 * char PASSKEY[64] = "1234";
 * http://192.168.178.88/1234/                - to view the main website (don't forget the trailing slash!)
 * http://192.168.178.88/1234/K               - to list all categories
 * http://192.168.178.88/1234/8700/8740/8741  - to list parameters 8700, 8740 and 8741 in one request
*/
char PASSKEY[64] = "";

/* activate IP-address-based access.
You can set any ip address as trusted.
Not used if first byte is 0
*/
byte trusted_ip_addr[4] = {0,0,0,0};
byte trusted_ip_addr2[4] = {0,0,0,0};

/* Activate HTTP-Auth authentication to provide username/password based access. No encryption!
 * Credentials have to be entered in the form
 * User:Password
 */
//char USER_PASS[64] = "User:Password";
char USER_PASS[64] = "admin:REDACTED";      // HTTP-Auth will be disabled if USER_PASS string length is zero


// Compile module with one wire bus support.
#define ONE_WIRE_BUS
byte One_Wire_Pin = 0;         // Define the pin for one wire temperature sensors. 0 - disable oneWire bus

// Compile module with DHT temperature/humidity sensors support
#define DHT_BUS
// Define the pins for DHT temperature/humidity sensors (Up to 10)
uint8_t DHT_Pins[10] = {0};

// Compile module with BME280 temperature/humidity/pressure sensors support on I2C bus.
// Up to two sensors with 0x76 and 0x77 addresses.
#define BME280
byte BME_Sensors = 0; //Define number of BME280 sensors

// Compile module for calculation 24h averages for selected programs
#define AVERAGES
// Create 24h averages from these parameters and save data into averages.txt on SD-card.
parameter avg_parameters[40] = {
// parameter, destination (as in dest_address below, -1 means "default (dest_address) address")
  {8700, -1},                         // Außentemperatur
  {8326, -1}                          // Brenner-Modulation
};

/* Compile module for logging.
File logging requires a FAT32 formatted micro SD card inserted into the Ethernet shield's card slot.
Does: log bus telegrams to file
      log selected values to file
      store to SD and load selected 24h averages
      send selected values as UDP broadcast
      push selected values to an MQTT broker*/
#define LOGGER

#define UDP_LOG_PORT 6502 // fixed here, not configurable via web interface

// Setting to determine on ESP32 whether to use SD card adapter (if present) or ESP32's internal flash.
// SD card should always be preferred to protect the ESP32's flash from wearing down.
uint8_t LogDestination = 0;  // 0 = SD card, 1 = ESP32's flash memory

// Log "raw" bus telegrams. Data saved in journal.txt on SD-card.
// Telegrams logged upon boot:
// int logTelegram = LOGTELEGRAM_OFF; // nothing to log,
// int logTelegram = LOGTELEGRAM_ON;  // log all telegrams,
// int logTelegram = LOGTELEGRAM_ON + LOGTELEGRAM_UNKNOWN_ONLY;           // log unknown telegrams,
// int logTelegram = LOGTELEGRAM_ON + LOGTELEGRAM_BROADCAST_ONLY;         // log broadcast telegrams,
// int logTelegram = LOGTELEGRAM_ON + LOGTELEGRAM_UNKNOWNBROADCAST_ONLY;  // log unknown broadcast telegrams only;
int logTelegram = LOGTELEGRAM_OFF;

// Logging data from parameters
// Interval and list of parameters can be redefined through /L command during runtime
// Data will be written to "datalog.txt"
unsigned long log_interval = 15;  // Logging interval (to SD card, UDP and MQTT broker) in seconds
parameter log_parameters[40] = {
// parameter, destination (as in dest_address below, -1 means "default (dest_address) address")
  {8700, -1},                   // Außentemperatur
  {8703, -1},                   // Außentemperatur gedämpft
  {8704, -1},                   // Außentemperatur gemischt
  {8310, -1},                   // Kesseltemperatur
  {8314, -1},                   // Rücklauftemperatur
  {8326, -1},                   // Brenner-Modulation,
  {8743, -1},                   // Vorlauftemperatur
  {8750, -1},                   // Modulation HKP
  {8830, -1},                   // Trinkwassertemperatur 1
  {8832, -1}                    // Trinkwassertemperatur 2
};

// Compile MQTT extension: activate sending log_parameters to MQTT broker every log_interval seconds
#define MQTT
byte mqtt_mode = 1; // MQTT: 1 - send messages in plain text format, 2 - send messages in JSON format, 3 - send messages in rich JSON format. Use this if you want a json package of your logging information printed to the mqtt topic
// JSON payload will be of the structure: {"MQTTDeviceID": {"status":{"log_param1":"value1"}}}
// rich JSON payload will be of the structure: {"MQTTDeviceID": {"id": "parmeter number from log values", "name": "parameter name from logvalues", "value": "query result", "desc": "enum value description", "unit": "unit of measurement", "error", error code}}
char mqtt_broker_addr[33] = "192.168.0.1:1883"; // MQTT broker. Can be IP address or hostname. Optional port can be specified after trailing colon. If omitted, port defaults to 1883.
//byte mqtt_broker_ip_addr[4] = {192,168,0,1};
char MQTTUsername[65] = "REDACTED";              // Set username for MQTT broker here or set empty string if no username/password is used.
char MQTTPassword[65] = "REDACTED";    // Set password for MQTT broker here or set empty string if no password is used.
char MQTTTopicPrefix[65] = "bsb-lan"; 	        // Optional: Choose the "topic" for MQTT messages here. In case of empty string, default topic name will be used

// Optional: Define a device name to use as header in json payload. In case of empty string, "BSB-LAN" will be used.
// This value is also used as a client ID towards the MQTT broker, change it if you have more than one BSB-LAN on your broker.
char MQTTDeviceID[32] = "BSB-LAN";

// Logging mode: 0 - disabled, 1 - write values to SD card, 2 - write 24h averages to SD card, 4 - send values to MQTT, 8 -  send values to UDP. Can be any sum of these values.
byte LoggingMode = 4; //CF_LOGMODE_SD_CARD | CF_LOGMODE_24AVG | CF_LOGMODE_MQTT | CF_LOGMODE_UDP


// Compile IPWE extension
#define IPWE
bool enable_ipwe = false;  // true - activate IPWE extension (http://xxx.xxx.xxx.xxx/ipwe.cgi)
// Parameters to be displayed in IPWE extension
parameter ipwe_parameters[40] = {
// parameter, destination (as in dest_address below, -1 means "default (dest_address) address")
  {8700, -1},                  // Außentemperatur
  {8743, -1},                  // Vorlauftemperatur
  {8314, -1}                   // Rücklauftemperatur
};

// If you prefer to use the log file plotting (/DG) used in BSB-LAN 2.1.3, disable the following #define.
// Otherwise a newer implementation will be used that does require (automated) loading of an additional
// Javascript library off the internet (currently 204 KB), but offers the following improvements:
// - better legibility for value numbers with plot lines close to each other (mouseover on plot)
// - user can interactively highlight plot lines for improved overview (mouseover on legend entries)
// - user can interactively disable plot lines for improved overview and vertical scaling (click on legend entries)
// - added zoom (mousewheel/pinch on plot) and pan capability (drag zoomed-in plot)
// - selective plotting of data from big logs
#define USE_ADVANCED_PLOT_LOG_FILE
#define DEFAULT_DAYS_TO_PLOT "3" // must be a positive integer value in double quotes!
// Log file plotting uses two JavaScript libraries, which are usually loaded from the web.
// Should you prefer to use local copies instead, put the files from the data subdirectory onto your
// bsb-lan unit's SD card (*), and provide their path names here. For browsers that support gzip
// compression (what browser doesn't?), it is sufficient to copy the *.gz file versions to your
// bsb-lan unit, but omit the ".gz" from the path names you put into the following lines!
// (*) In case you're using an ESP32's internal memory instead of an SD card, use
// https://github.com/lorol/arduino-esp32littlefs-plugin to upload the files.
// For local use of these libraries to work, "#define WEBSERVER" is also required!
//#define D3_LIBRARY_PATH "/d3.js"
//#define C3_LIBRARY_PATH "/c3.js"

// Compile CUNO/CUNX/modified MAX!Cube extension.
#define MAX_CUL
bool enable_max_cul = false;                // enable or disable connection to CUNO/CUNX/modified MAX!Cube;
byte max_cul_ip_addr[4] = {192,168,178,5};     // IP of CUNO/CUNX/modified MAX!Cube. Please use commas instead of dots!!!
char max_device_list[20][11] = {               // list of MAX! wall/heating thermostats that should be polled
  "KEQ0502326",                                // use MAX! serial numbers here which have to have exactly 10 characters
  "KEQ0505080"
};


// defines the number of retries for the query command
#define QUERY_RETRIES  3

// Setting bus pins and bus type

byte bus_pins[2] = {19,18}; //First value - RX pin, second value - TX pin. 0,0 - auto select (19,18 for Due, 16,17 for NodeMCU, 36,17 for Olimex EVB, 36,5 for Olimex POE and 68,69 for Mega).
uint8_t bus_type = 0;  // set bus system at boot: 0 = BSB, 1 = LPB, 2 = PPS
// BSB:
// - 'own_address' sets own address, defaults to 0x42 (LAN in serial monitor)
// - 'dest_address' sets destination address, defaults to 0 for heating system.
// LPB:
// - 'own_address and 'dest_address' set own and destination address (high nibble = segment, low nibble = device minus 1)
// - defaults to 0x42 for own address and 0x00 for destination address, i.e. segment 4, device 3 for Arduino/BSB-LAN and segment 0, device 1 for heating system
byte own_address = 0x42;
byte dest_address = 0x00;
// PPS:
// - set 'pps_write' to "1" to enable writing to heater - only use this if there is no other room controller (such as QAA50/QAA70) active.
bool pps_write = 0;
byte QAA_TYPE = 0x53;  // 0x53 = QAA70, 0x52 = QAA50, 0x5A = QAA10, 0x37 = QAA95, 0x66 = BMU, 0xEA = MCBA/REA70/DC225

/* Set the device family and device variant of your heating system. Only change this if you _really_ know what you are doing!
 * Set fixed_device_family and fixed_device_variant to your device family and variant (parameters 6225 and 6226) here
 * if heating system is not running when Arduino is powered on.
*/
uint16_t fixed_device_family = 97;
uint16_t fixed_device_variant = 100;

// defines default flag for parameters
// use "#define DEFAULT_FLAG FL_SW_CTL_RONLY" to control read/write functionality via configuration in web interface.
// use "#define DEFAULT_FLAG 0" to make (almost) all parameters writeable
// use #define DEFAULT_FLAG FL_RONLY to run the program always in read-only mode.
#define DEFAULT_FLAG 0

// include commands from BSB_lan_custom.h to be executed at the end of each main loop
//#define CUSTOM_COMMANDS

/*
 * Check for new versions when accessing BSB-LAN's main page.
 * Doing so will poll the most recent version number from the BSB-LAN server.
 * In this process, it is unavoidable that your IP address will be transferred to the server, obviously.
 * We nevertheless mention this here because this constitutes as 'personal data' and this feature is therefore disabled by default.
 * Activating this feature means you are consenting to transmitting your IP address to the BSB-LAN server where it will be stored
 * for up to two weeks in the server's log files to allow for technical as well as abuse analaysis.
 * No other data (such as anything related to your heating system) is transmitted in this process, as you can see in the source code.
*/
#define VERSION_CHECK
bool enable_version_check = true;

#define ENABLE_ESP32_OTA
boolean enable_ota_update = false;

// Reduce clock speed of ESP32 from 240 MHz to 80MHz, saving ca. 20% energy.
// Works well when connecting via LAN, but since it reduces WiFi range and log file display times when using WiFi, it is disabled by default.
boolean esp32_save_energy = false;

// "External" web server. Read files from SD-card. Only static content: html, js, css, jpg, etc.
//#define WEBSERVER

// Configuration will be stored in EEPROM
#define CONFIG_IN_EEPROM

// Compile web-based configuration and EEPROM config store module extension.
#define WEBCONFIG

// Compile JSON-based configuration and EEPROM config store module extension.
#define JSONCONFIG

#define RGT_EMULATOR
parameter rgte_sensorid[3][5] = {{{0, -1}, {0, -1}, {0, -1}, {0, -1}, {0, -1}}, {{0, -1}, {0, -1}, {0, -1}, {0, -1}, {0, -1}}, {{0, -1}, {0, -1}, {0, -1}, {0, -1}, {0, -1}}}; //Temperature sensor program IDs for RGT1/PPS - RGT3. If zero then RGT will not be emulated. If more than one program set per RGT then average will be calculated and used.

//Enable presence buttons and TWW/DHW push on selected pins.
// Pins on Mega can be (Digital) 2, 3, 18, 19, 20, 21
// On Due any Digital pins can be selected excluding 12, 16-21, 31, 33, 53.
// Make sure you aren't using pins which are already in use for sensors (default: 2, 3, 7) or change them accordingly.
#define BUTTONS
uint8_t button_on_pin[4] = {0, 0, 0, 0}; //Order: TWW push, presence ROOM1, presence ROOM2, presence ROOM3

// Variables for future use:
// Compile room unit replacement extension
#define ROOM_UNIT
byte UdpIP[4] = {0,0,0,0}; // (destination IP address for sending UDP packets to, the room unit replacement of FHEM user @fabulous uses that)
uint16_t UdpDelay = 15;    // (interval in seconds to send UDP packets)

// Compile off-site logger extension
#define OFF_SITE_LOGGER
byte destinationServer[128] = "";   // URL string to periodically send values to an off-site logger
uint16_t destinationPort = 80;      // port number for abovementioned server
uint32_t destinationDelay = 84600;  // interval in seconds to send values

#define CONFIG_VERSION 35

/************************************************************************************/
/************************************************************************************/
/* Settings -   END                                                                 */
/************************************************************************************/
/************************************************************************************/

The text was updated successfully, but these errors were encountered:

fredlcore · 2024-02-29T15:37:50Z

It could be that the Arduino Due library does not handle the DNS call for IP addresses in the same way that the ESP32 libraries do. I read that you would have to call the connect function differently when using an IP address compared to a hostname, but during my testings, both worked fine when using the one with a hostname, even if I used an IP address.
Can you please try an use a hostname instead? Every computer in a local network should have one, you may find out about it in your router if you are unsure.

fredlcore · 2024-02-29T16:05:43Z

Update: On my system, I get this, if I use an IP address that is not in use:

00:03:21.201 > Client ID: BSB-LAN
00:03:21.201 > Will topic: BSB-LAN/status
00:03:21.229 > [ 10995][E][WiFiGeneric.cpp:1578] hostByName(): DNS Failed for ��A?
00:03:22.231 > Failed to connect to MQTT broker, retrying...
00:03:22.259 > [ 12026][E][WiFiGeneric.cpp:1578] hostByName(): DNS Failed for ��A?
00:03:23.263 > Failed to connect to MQTT broker, retrying...
00:03:23.294 > [ 13060][E][WiFiGeneric.cpp:1578] hostByName(): DNS Failed for ��A?
00:03:24.295 > Failed to connect to MQTT broker, retrying...

Once I switch back to the correct server's IP address, I get this:

00:04:48.774 > Virtual parameter 20000 queried. Table line 1067
00:04:48.779 > Publishing to topic: BSB-LAN/20000
00:04:48.781 > Payload: 0
00:04:48.784 > Successfully published...

tiger42 · 2024-02-29T16:06:08Z

Using the hostname did not work either. My local DNS server is configured correctly and does resolve the name (which I've verified using another system).

fredlcore · 2024-02-29T16:12:28Z

Can you try the following please:
Change this line in include/mqtt_handler.h:

    MQTTPubSubClient->setServer(mqtt_host, mqtt_port);

to these lines

    Serial.println(mqtt_host);
    Serial.println(mqtt_port);
    byte mqtt_broker_ip_addr[4] = {192,168,0,1};
    mqtt_port = 1883;
    MQTTPubSubClient->setServer(mqtt_broker_ip_addr, mqtt_port);

and see if that works and what the serial monitor outputs as mqtt_host and mqtt_port?

fredlcore · 2024-02-29T17:37:46Z

I have now uploaded a version that calls setServer with an IPAddress object, so if that was the problem, it should work now.

tiger42 · 2024-02-29T18:16:12Z

It's working now when using an IP address. 👍🏻
It does not work yet when using a host name. I haven't looked into the code if it should though. For me that's fine, but if you wish, I would try to debug that tomorrow. But don't expect too much, C is not my main language 😉

fredlcore · 2024-03-01T01:49:42Z

Good to know it works - and yes, if you have the time, please feel free to figure out what's going (wr)on(g). My changes simply converted the IP address from a string to actual numerical values which the PubSubClient instance accepts as one way of the setServer() function. This is then passed on to the Client object, which in BSB-LAN varies depending on whether you are using an ESP32 or an Arduino. On an ESP32, the WifiClient object is used and it provides a connect function which accepts a host as a char array (i.e. as text) and then uses the DNSClient object's getHostByName function on that char array. On the ESP32, the getHostByName function tolerates an IP address and just returns as it is. So that's why there were no problems for me before. Apparently, this is handled differently on the Arduino side. Not only does the getHostByName function not tolerate an IP as a string, it also does not seem to support DNS resolution in the first place. The place to look for would therefore be the client function of the EthernetClient class as well as something similar as the getHostByName function and see if these support host name resolution, and if yes, in which way.

EDIT: I just had a look myself and figured out that at least on my machine, the code in Dns.h for the getHostByName function is exactly the same, including accepting an IP address as a string instead of its numerical values (for this, it doesn't even call a DNS server). Therefore, it should also have no problem resolving a hostname. So the problem must lie somewhere else.

tiger42 · 2024-03-02T18:05:45Z

After fiddling around with the Arduino Dns.h library and dig/nslookup from another Linux machine, I've found out that my DNS server was in fact configured wrong and could not resolve some local addresses. Everything is fine now! The issue can be closed.

fredlcore · 2024-03-03T01:27:21Z

Grrr ;)... Ok, in order not to keep useless code in the scarce memory that we have, can you please test again by replacing these lines:

    int addr[4];
    int result = sscanf(mqtt_host, "%d.%d.%d.%d", &addr[0], &addr[1], &addr[2], &addr[3]);
    if (result == 4) {
      IPAddress MQTT_IP(addr[0],addr[1],addr[2],addr[3]);
      MQTTPubSubClient->setServer(MQTT_IP, mqtt_port);
    } else {
      MQTTPubSubClient->setServer(mqtt_host, mqtt_port);
    }

with this line:

      MQTTPubSubClient->setServer(mqtt_host, mqtt_port);

If this works (as it obviously should, going by the code), I can revert everything to the state before...

tiger42 · 2024-03-03T14:05:06Z

Damn, I was wrong about the DNS resolving on the Arduino... My DNS server really does work now, but the Arduino's name resolution still does not. I had overlooked some left over debug code I wrote, which had an IP address hard coded... 😞
That means after reverting the MQTT connection code as you asked for, it still does not work. Neither with an IP address given nor with a server name...
But I came up with a solution that does work in any case, with given IP address and with DNS resolving. I don't know if this will work on the ESP though:

    IPAddress MQTT_IP;
    int addr[4];
    int result = sscanf(mqtt_host, "%d.%d.%d.%d", &addr[0], &addr[1], &addr[2], &addr[3]);
    if (result == 4) {
      MQTT_IP.fromString(mqtt_host);
    } else {
      DNSClient dnsClient;
      dnsClient.begin(dns_addr);
      dnsClient.getHostByName(mqtt_host, MQTT_IP);
    }
    printFmtToDebug(PSTR("Using MQTT broker %d.%d.%d.%d:%d\r\n"), MQTT_IP[0], MQTT_IP[1], MQTT_IP[2], MQTT_IP[3], mqtt_port);
    MQTTPubSubClient->setServer(MQTT_IP, mqtt_port);

The test for a valid IP could be done more elegant using a RegExp or so as the address components (addr[n]) are not used afterwards. Maybe you have an idea for a better solution. As I said, I am not a C developer...

Of course Dns.h needs to be included. I did it in line 615 of BSB_LAN.ino, as it only seems to be needed when MQTT is used:

#ifdef MQTT
  #include "src/PubSubClient/src/PubSubClient.h"
  #include <Dns.h>
#endif

After all I think there must be a difference between some network libraries used for Arduino and for ESP. Otherwise I could not explain the different behaviour.

fredlcore · 2024-03-03T14:36:12Z

Ok, thanks for testing, but I'm going to wait now until some other Arduino Due user confirms this problem. Because since the libraries seem to be identical to the point that I could follow up with it, the whole DNS resolution would have to have a problem on the Due then. The code you posted is almost identical to what is already in the EthernetClient.cpp for the Due:

int EthernetClient::connect(const char * host, uint16_t port)
{
	DNSClient dns; // Look up the host first
	IPAddress remote_addr;

	if (_sockindex < MAX_SOCK_NUM) {
		if (Ethernet.socketStatus(_sockindex) != SnSR::CLOSED) {
			Ethernet.socketDisconnect(_sockindex); // TODO: should we call stop()?
		}
		_sockindex = MAX_SOCK_NUM;
	}
	dns.begin(Ethernet.dnsServerIP());
	if (!dns.getHostByName(host, remote_addr)) return 0; // TODO: use _timeout
	return connect(remote_addr, port);
}

Maybe there is a problem with your library inclusion or something like that. And/or you could add debug messages in this section to figure out when it's called and what paths are taken. But since the Client object of PubSubClient is based on EthernetClient, it should work right out of the box, as it does on the ESP32.

drfjordan · 2024-03-31T20:09:21Z

I have updated today to BSB-LAN, Version 3.4.4-20240331020948 and for some so far unknown reason the MQTT broker is not showing the expected data content.
(I had been using 3.2.2-20230606224119 on Arduino DUE board with the Ethernet shield (since some years) and it did work good so far.)

I have received some indication from my DNS server that there had been a lot of suspicious requests to "BSB-LAN/status":
Mar 31 19:46:36: query[A] BSB-LAN/status from 192.168.1.221
Mar 31 19:46:36: config BSB-LAN/status is NXDOMAIN

This is the content of my current file "BSB_LAN_config.h":

/************************************************************************************/
/**********************************************<**************************************/
/* Settings -   BEGIN                                                               */
/************************************************************************************/
/************************************************************************************/

// Upon first installation, rename this file from BSB_lan_config.h.default to BSB_lan_config.h and adjust settings accordingly

/* Select language; so far German is the most complete, with English following.
 * Available languages are: Czech (CS), German (DE), Danish (DA), English (EN), Spanish (ES), Finnish (FI),
 * French (FR), Greek (EL), Hungarian (HU), Italian (IT), Dutch (NL), Polish (PL), Russian (RU), Swedish (SV),
 * Slovenian (SI) and Turkish (TR).
 * Incomplete languages will automatically be filled up with English translations first, and if no English translation
 * is available, fallback will take place to German.
*/
#define LANG DE

/*
Allow to initialize program configuration by reading settings from EEPROM
byte UseEEPROM = 0; // Configuration is read from this config file.
                    // Configuration can be stored in EEPROM but will not be used while UseEEPROM is zero.
                    // Set zero for fallback startup in case EEPROM configuration is broken or you cannot access BSB-LAN anymore.
byte UseEEPROM = 1; // Configuration will be read from EEPROM. This is the default.
*/
byte UseEEPROM = 0;

/*
 * Configuration of the network settings
*/
uint8_t network_type = LAN;             // Set to LAN (0) when using Ethernet connection. Set to WLAN (1) when using WiFi. Arduino users using WiFiSpi have to activate the definement below as well.
uint16_t HTTPPort = 80;
bool useDHCP = true;                    // Set to false if you want to use a fixed IP.
byte ip_addr[4] = {192,168,1,221};     // Please note the commas instead of dots!!!  Set useDHCP (above) to false if you want to use a fixed IP.
byte gateway_addr[4] = {192,168,1,253}; // Gateway address. This is usually your router's IP address. Please note the commas instead of dots!!! Ignored if first value is 0.
byte dns_addr[4] = {192,168,1,2};     // DNS server. Please note the commas instead of dots!!! Ignored if first value is 0.
byte subnet_addr[4] = {255,255,255,0};  // Subnet address. Please use commas instead of dots!!! Ignored if first value is 0.

char wifi_ssid[32] = "YourWiFiNetwork"; // enter your WiFi network name (SSID) here
char wifi_pass[64] = "YourWiFiPassword";// enter your WiFi password here
uint8_t bssid[6] = {0x00, 0x00, 0x00, 0x00, 0x00, 0x00};  // enter specific BSSID address here to ensure connecting to a specific router. Leave all zeros in normal circumstances.

//#define WIFISPI                         // Activate this on the Arduino to enable WiFi via WiFiSpi. DO NOT enable this on an ESP32.
#define WIFI_SPI_SS_PIN 12              // defines SPI-SS pin for Arduino-ESP8266 connection

char mDNS_hostname[32] = "BSB-LAN";     // Advertises the hostname in the local network. Set this to an empty string if you don't want your device to be found under this name in your network.

/* NTP settings to acquire exact date and time via network.
 * Attention: This only works with ESP32 microcontrollers so far!
 * Use pool.ntp.org if your BSB-LAN installation can access the internet.
 * Otherwise you may also use your router's address if it can act as a NTP server.
 * The default timezone "CET-1CEST,M3.5.0,M10.5.0/3" covers most Central European countries (GMT+1) and takes care of daylight saving.
 * Use "EET-2EEST,M3.5.0/3,M10.5.0/4" for Eastern European countries (GMT+2), or 
 * use "WET0WEST,M3.5.0/1,M10.5.0" for Western European countries (GMT+0).
 * See here for a full list of timezones for places all over the world:
 * https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv
*/
const char ntp_server[20] = "de.pool.ntp.org"; // Set to empty string in case you don't want to use NTP
const char local_timezone[30] = "CET-1CEST,M3.5.0,M10.5.0/3";

/* SECURITY OPTIONS
 * There are several options to control and protect access to your heating system. However, keep
 * in mind, that even activating all three options are no guarantee that a versatile intruder with
 * access to your (W)LAN won't be able to gain access. In any case, no encryption of data streams
 * is provided from the Arduino itself. Use VPN or a SSL proxy if that is a must for you and connect
 * the Arduino wired to the VPN server or SSL proxy. On the other hand, someone with this amount
 * of criminal activity will probably have it easier just to access your heating system face-to-face ;)
*/

/*
 * if PASSKEY length is non-zero, the URL has to contain the defined passkey as first element
 * e.g.
 * char PASSKEY[64] = "1234";
 * http://192.168.178.88/1234/                - to view the main website (don't forget the trailing slash!)
 * http://192.168.178.88/1234/K               - to list all categories
 * http://192.168.178.88/1234/8700/8740/8741  - to list parameters 8700, 8740 and 8741 in one request
*/
char PASSKEY[64] = "";

/* activate IP-address-based access.
You can set any ip address as trusted.
Not used if first byte is 0
*/
byte trusted_ip_addr[4] = {0,0,0,0};
byte trusted_ip_addr2[4] = {0,0,0,0};

/* Activate HTTP-Auth authentication to provide username/password based access. No encryption!
 * Credentials have to be entered in the form
 * User:Password
 */
//char USER_PASS[64] = "User:Password";
char USER_PASS[64] = "";      // HTTP-Auth will be disabled if USER_PASS string length is zero

/*
 *  Enter a MAC address, found either on the EthernetShield or use the one below.
 *  Change this if you have more than one BSB-LAN adapter in your LAN, so that there aren't any address conflicts.
 *  MAC address here only affects the LAN shield, it doesn't apply to the WiFi-ESP-solution.
*/
byte mac[6] = { 0xA8, 0x61, 0x0A, 0xAE, 0x2F, 0x56 };

// Setting bus type
uint8_t bus_type = 0;  // set bus system at boot: 0 = BSB, 1 = LPB, 2 = PPS
// BSB:
// - 'own_address' sets own address, defaults to 0x42 (LAN in serial monitor)
// - 'dest_address' sets destination address, defaults to 0 for heating system.
// LPB:
// - 'own_address and 'dest_address' set own and destination address (high nibble = segment, low nibble = device minus 1)
// - defaults to 0x42 for own address and 0x00 for destination address, i.e. segment 4, device 3 for BSB-LAN and segment 0, device 1 for heating system
byte own_address = 0x42;
byte dest_address = 0x00;
// PPS:
// - set 'pps_write' to "1" to enable writing to heater - only use this if there is no other room controller (such as QAA50/QAA70) active.
bool pps_write = 0;
byte QAA_TYPE = 0x53;  // 0x53 = QAA70, 0x52 = QAA50, 0x5A = QAA10, 0x37 = QAA95, 0x66 = BMU, 0xEA = MCBA/REA70/DC225
// Setting bus pins
byte bus_pins[2] = {0,0}; //First value - RX pin, second value - TX pin. 0,0 - auto select (19,18 for Due, 16,17 for NodeMCU, 36,17 for Olimex EVB, 36,5 for Olimex POE and 68,69 for Mega).

/* Set the device family and device variant of your heating system. Only change this if you _really_ know what you are doing!
 * Set fixed_device_family and fixed_device_variant to your device family and variant (parameters 6225 and 6226) here
 * if heating system is not running when BSB-LAN is powered on.
*/
uint16_t fixed_device_family = 96;
uint16_t fixed_device_variant = 100;

// defines default flag for parameters
// use "#define DEFAULT_FLAG FL_SW_CTL_RONLY" to control read/write functionality via configuration in web interface.
// use "#define DEFAULT_FLAG 0" to make (almost) all parameters writeable
// use #define DEFAULT_FLAG FL_RONLY to run the program always in read-only mode.
#define DEFAULT_FLAG FL_SW_CTL_RONLY

// Setting to determine on ESP32 whether to use SD card adapter (if present) or ESP32's internal flash.
// SD card should always be preferred to protect the ESP32's flash from wearing down.
uint8_t LogDestination = SDCARD;  // Possible logging devices: SDCARD (0) = SD card, FLASH (1) = ESP32's flash memory
// If you use an SD card reader on the Joy-It ESP32 NodeMCU, you can configure the SPI pins here:
#define SD_SCK 18
#define SD_MISO 19
#define SD_MOSI 23
#define SD_CS 5
//#define FORCE_SD_MMC_ON_NODEMCU   // enable this definement if you want to force using the SD_MMC library instead of SPI on an ESP32 NodeMCU


// Log "raw" bus telegrams. Data saved in journal.txt on SD-card.
// Telegrams logged upon boot:
// int logTelegram = LOGTELEGRAM_OFF;                                       // nothing to log,
// int logTelegram = LOGTELEGRAM_ON;                                        // log all telegrams,
// int logTelegram = LOGTELEGRAM_UNKNOWN_ONLY;                              // log unknown telegrams,
// int logTelegram = LOGTELEGRAM_BROADCAST_ONLY;                            // log broadcast telegrams,
// int logTelegram = LOGTELEGRAM_UNKNOWN_ONLY + LOGTELEGRAM_BROADCAST_ONLY; // log unknown broadcast telegrams only;
int logTelegram = LOGTELEGRAM_OFF;

// Logging data from parameters
// Interval and list of parameters can be redefined through /L command during runtime
// Data will be written to "datalog.txt"
unsigned long log_interval = 90;  // Logging interval (to SD card, UDP and MQTT broker) in seconds
parameter log_parameters[40] = {
// parameter, destination (as in dest_address below, -1 means "default (dest_address) address")
  {700, -1},                    // Heizkreis 1 Betriebsart
  {1600, -1},                   // Trinkwasser Betriebsart
  {8700, -1},                   // Außentemperatur
  {8740, -1},                   // Raumtemperatur
  {8310, -1},                   // Kesseltemperatur
  {8830, -1},                   // Trinkwassertemperatur
  {8950, -1},                   // Schienenvorlauftemperatur
  {8980, -1},                   // Pufferspeichertemperatur
  {8300, -1},                   // 1. Brennerstufe
  {8301, -1},                   // 2. Brennerstufe
  {8331, -1},                   // Startzähler 1.Stufe
  {8333, -1},                   // Startzähler 2.Stufe
  {8330, -1},                   // Betriebsstunden 1.Stufe
  {8332, -1},                   // Betriebsstunden 2.Stufe
  {9031, -1},                   // Relaisausgang QX1
  {8730, -1},                   // Heizkreispumpe Q2
  {8820, -1},                   // Trinkwasserpumpe Q3
  {20000, -1},                  // Spezialparameter: Brenner-Laufzeit Stufe 1
  {20001, -1},                  // Spezialparameter: Brenner-Takte Stufe 1
  {20002, -1},                  // Spezialparameter: Brenner-Laufzeit Stufe 2
  {20003, -1},                  // Spezialparameter: Brenner-Takte Stufe 2
  {20004, -1},                  // Spezialparameter: TWW-Laufzeit
  {20005, -1}                   // Spezialparameter: TWW-Takte
};

// Compile MQTT extension: activate sending log_parameters to MQTT broker every log_interval seconds
char mqtt_broker_addr[33] = "192.168.1.2:1883";    // MQTT broker. Adjust LoggingMode to activate. Can be IP address or hostname. Optional port can be specified after trailing colon. If omitted, port defaults to 1883.
char MQTTUsername[65] = "";                 // Set username for MQTT broker here or set empty string if no username/password is used.
char MQTTPassword[65] = "";                 // Set password for MQTT broker here or set empty string if no password is used.
char MQTTTopicPrefix[65] = "BSB-LAN"; 	        // Optional: Choose the "topic" for MQTT messages here. In case of empty string, default topic name will be used
byte mqtt_mode = 1; // MQTT: 1 - send messages in plain text format, 2 - send messages in JSON format, 3 - send messages in rich JSON format. Use this if you want a json package of your logging information printed to the mqtt topic
// JSON payload will be of the structure: {"MQTTDeviceID": {"status":{"log_param1":"value1"}}}
// rich JSON payload will be of the structure: {"MQTTDeviceID": {"id": "parmeter number from log values", "name": "parameter name from logvalues", "value": "query result", "desc": "enum value description", "unit": "unit of measurement", "error", error code}}

// Optional: Define a device name to use as header in json payload. In case of empty string, "BSB-LAN" will be used.
// This value is also used as a client ID towards the MQTT broker, change it if you have more than one BSB-LAN on your broker.
char MQTTDeviceID[32] = "MyHeater";

// Logging mode: 0 - disabled, 1 - write values to SD card, 2 - write 24h averages to SD card, 4 - send values to MQTT, 8 -  send values to UDP. Can be any sum of these values.
byte LoggingMode = 4; //CF_LOGMODE_SD_CARD | CF_LOGMODE_24AVG | CF_LOGMODE_MQTT | CF_LOGMODE_UDP

// Create 24h averages from these parameters and save data into averages.txt on SD-card.
parameter avg_parameters[40] = {
// parameter, destination (as in dest_address below, -1 means "default (dest_address) address")
  {8700, -1},                         // Außentemperatur
  {8740, -1},                         // Raumtemperatur
  {8310, -1},                         // Kesseltemperatur
  {8830, -1},                         // Trinkwassertemperatur
  {8980, -1}                          // Pufferspeichertemperatur
};

// Define the pin for one wire temperature sensors. -1 = disable oneWire bus.
int8_t One_Wire_Pin = -1;

// Define the pins for DHT temperature/humidity sensors (Up to 10). -1 = disable DHT sensor.
int8_t DHT_Pins[10] =  {-1, -1, -1, -1, -1, -1, -1, -1, -1, -1};

// Define the number of BME280 temperature/humidity/pressure sensors on the I2C bus. 0 = disable BME280 sensors.
// Up to two sensors with 0x76 and 0x77 addresses, up to 16 when using a TCA9548A I2C mulitplexer.
byte BME_Sensors = 0;

bool enable_ipwe = false;  // true - activate IPWE extension (http://xxx.xxx.xxx.xxx/ipwe.cgi)
// Parameters to be displayed in IPWE extension
parameter ipwe_parameters[40] = {
// parameter, destination (as in dest_address below, -1 means "default (dest_address) address")
  {8700, -1},                  // Außentemperatur
  {8743, -1},                  // Vorlauftemperatur
  {8314, -1},                  // Rücklauftemperatur
//  {8750, -1},                  // Gebläsedrehzahl
//  {8830, -1},                  // Warmwassertemperatur
//  {8740, -1},                  // Raumtemperatur Ist
//  {8741, -1},                  // Raumtemperatur Soll
//  {8326, -1},                  // Brenner-Modulation
//  {8337, -1},                  // Startzähler Brenner
//  {8703, -1},                  // Aussentemperatur gedämpft
//  {8704, -1}                   // Aussentemperatur gemischt
};

// If you prefer to use the log file plotting (/DG) used in BSB-LAN 2.1.3, disable the following #define.
// Otherwise a newer implementation will be used that does require (automated) loading of an additional
// Javascript library off the internet (currently 204 KB), but offers the following improvements:
// - better legibility for value numbers with plot lines close to each other (mouseover on plot)
// - user can interactively highlight plot lines for improved overview (mouseover on legend entries)
// - user can interactively disable plot lines for improved overview and vertical scaling (click on legend entries)
// - added zoom (mousewheel/pinch on plot) and pan capability (drag zoomed-in plot)
// - selective plotting of data from big logs
#define USE_ADVANCED_PLOT_LOG_FILE
#define DEFAULT_DAYS_TO_PLOT "3" // must be a positive integer value in double quotes!
// Log file plotting uses two JavaScript libraries, which are usually loaded from the web.
// Should you prefer to use local copies instead, put the files from the data subdirectory onto your
// bsb-lan unit's SD card (*), and provide their path names here. For browsers that support gzip
// compression (what browser doesn't?), it is sufficient to copy the *.gz file versions to your
// bsb-lan unit, but omit the ".gz" from the path names you put into the following lines!
// (*) In case you're using an ESP32's internal memory instead of an SD card, use
// https://github.com/lorol/arduino-esp32littlefs-plugin to upload the files.
// For local use of these libraries to work, enabling webserver functionality is also required!
//#define D3_LIBRARY_PATH "/d3.js"
//#define C3_LIBRARY_PATH "/c3.js"

bool enable_max_cul = false;                // enable or disable connection to CUNO/CUNX/modified MAX!Cube;
byte max_cul_ip_addr[4] = {192,168,178,5};     // IP of CUNO/CUNX/modified MAX!Cube. Please use commas instead of dots!!!
char max_device_list[20][11] = {               // list of MAX! wall/heating thermostats that should be polled
  "KEQ0502326",                                // use MAX! serial numbers here which have to have exactly 10 characters
  "KEQ0505080"
};

// include commands from BSB_lan_custom.h to be executed at the end of each main loop
//#define CUSTOM_COMMANDS

/*
 * Check for new versions when accessing BSB-LAN's main page.
 * Doing so will poll the most recent version number from the BSB-LAN server.
 * In this process, it is unavoidable that your IP address will be transferred to the server, obviously.
 * We nevertheless mention this here because this constitutes as 'personal data' and this feature is therefore disabled by default.
 * Activating this feature means you are consenting to transmitting your IP address to the BSB-LAN server where it will be stored
 * for up to two weeks in the server's log files to allow for technical as well as abuse analaysis.
 * No other data (such as anything related to your heating system) is transmitted in this process, as you can see in the source code.
*/
bool enable_version_check = true;

// Enable ESP32 over-the-air updates. 
// Pro: You don't need to physically connect your PC to the ESP32 in order to update BSB-LAN.
// Contra: Someone who has access to your network and finds out about BSB-LAN's username and password can install their own software on the ESP32.
boolean enable_ota_update = false;

// Reduce clock speed of ESP32 from 240 MHz to 80MHz, saving ca. 20% energy.
// Works well when connecting via LAN, but since it reduces WiFi range and log file display times when using WiFi, it is disabled by default.
boolean esp32_save_energy = false;

// "External" web server. Read files from SD-card / flash memory. Only static content: html, js, css, jpg, etc.
boolean webserver = false;

// Debug options
byte debug_mode = 0;  // Debug mode: 0 - disabled, 1 - send debug messages to serial interface, 2 - send debug messages to telnet client
byte verbose = 0;     // If set to 1, all messages on the bus are printed to debug interface. Set to 2 only if you are asked by the developers as the extensive output will significantly slow down the microcontroller.
byte monitor = 0;     // Bus monitor mode. This is only necessary for in-depth debug sessions.
bool show_unknown = true; // true - show all parameters, false - hide unknown parameters from web display (parameters will still be queried and therefore take time!)

#define CONFIG_VERSION 37

/************************************************************************************/
/************************************************************************************/
/* Settings -   END                                                                 */
/************************************************************************************/
/************************************************************************************/

fredlcore · 2024-03-31T22:40:23Z

Is this related to the original bug report? Because you seem to be able to make a connection, you just mention the broker "not showing the expected data content" which could mean anything from no data to wrong data or correct data in the wrong places.
If it is not related to the original bug report, have the courtesy to open a separate bug report and add the required background information that is mentioned in the bug report. Based on the provided information, there is nothing I can do for you.

drfjordan · 2024-04-01T09:15:59Z

I guess it is related to the original bug report.
I have now enabled debugging and connected via telnet and see this output:

Version: 3.4.4-20240331020948
Client ID: MyHeater
Will topic: BSB-LAN/status
Failed to connect to MQTT broker, retrying...
Client ID: MyHeater
Will topic: BSB-LAN/status
Failed to connect to MQTT broker, retrying...
Client ID: MyHeater
Will topic: BSB-LAN/status
Failed to connect to MQTT broker, retrying...

Change this line in include/mqtt_handler.h:

    MQTTPubSubClient->setServer(mqtt_host, mqtt_port);

to

    byte mqtt_broker_ip_addr[4] = {192,168,1,2};
    mqtt_port = 1883;
    MQTTPubSubClient->setServer(mqtt_broker_ip_addr, mqtt_port);

succeeded to connect.

Therefore I think that some differences in DNS might exist which lead to the issue on the Arduino Due.

Interresting wise my PiHole (acting as DNS Server) did alert a "rate limit":

Client 192.168.1.221 has been rate-limited (current config allows up to 1000 queries in 60 seconds)

The dns query was not asking for the mqtt hostname, instead it was asking for the BSB-LAN/status
Mar 31 19:46:36: query[A] BSB-LAN/status from 192.168.1.221
Mar 31 19:46:36: config BSB-LAN/status is NXDOMAIN

tiger42 · 2024-04-01T12:40:11Z

@drfjordan Interesting find! I've looked into the dnsmasq logfiles on my server and found the same behavior (192.168.0.95 is BSB-LAN's IP):

dnsmasq: query[A] <name unprintable> from 192.168.0.95
dnsmasq: forwarded <name unprintable> to 192.168.0.100
dnsmasq: reply <name unprintable> is NXDOMAIN
dnsmasq: query[A] bsb-lan/status from 192.168.0.95
dnsmasq: cached bsb-lan/status is NXDOMAIN
dnsmasq: query[A] bsb-lan/status from 192.168.0.95
dnsmasq: cached bsb-lan/status is NXDOMAIN
dnsmasq: query[A] bsb-lan/status from 192.168.0.95
dnsmasq: cached bsb-lan/status is NXDOMAIN

"bsb-lan" is in lowercase letters on my system because I've changed the MQTTTopicPrefix in the config.
It looks like the DNS query does not send the MQTT server's name, but the MQTT "will" topic string instead. And it's very weird that the DNS server is queried at all, as I did not even specify a name but an IP address in my config!
It's also interesting that the very first query after starting BSB-LAN always fails because the queried name doesn't seem to be a valid string ("name unprintable").

Note that my workaround (#623 (comment)) is still working on my system (for both given IP and server name), maybe that helps when trying to find the actual problem.

fredlcore · 2024-04-01T13:06:07Z

Ok, thanks for the confirmation. I had just written to @drfjordan to open up a separate bug report for this, but it really seems that the root cause is the same for both, which is that for whatever reason the will topic is used as domain name and not the actual hostname entered in the configuration. This might explain why there is also no difference when you use an IP address or a FQDN because in both cases it uses a wrong memory address.

First I thought that this is a problem with a new EEPROM layout, but since @drfjordan has UseEEPROM set to 0, the EEPROM should not be used at all. Could you nevertheless both clear the EEPROM using /NE and then try again?
Also could you change the first lines of mqtt_connect() to this and send the SerMo output:

Serial.println(mqtt_broker_addr);
  char* tempstr = (char*)malloc(sizeof(mqtt_broker_addr));  // make a copy of mqtt_broker_addr for destructive strtok operation
Serial.println(tempstr);
  strcpy(tempstr, mqtt_broker_addr);
Serial.println(tempstr);
  uint16_t mqtt_port = 1883; 
  char* mqtt_host = strtok(tempstr,":");  // hostname is before an optional colon that separates the port
Serial.println(tempstr);
Serial.println(mqtt_host);
  char* token = strtok(NULL, ":");   // remaining part is the port number
Serial.println(token);
Serial.println(mqtt_port);
  if (token != 0) {
    mqtt_port = atoi(token);
  }
Serial.println(mqtt_port);
  free(tempstr);
Serial.println((mqtt_client[0]);

This really looks like some kind of memory leak, but if hard-coding the hostname works for both of you, it has to happen somewhere in those lines of code above. Let's hope we find it quickly :)...

tiger42 · 2024-04-01T14:31:26Z

Clearing the EEPROM did not change anything (I've actually disabled the EEPROM feature anyways).

The last Serial.println makes my Arduino crash for whatever reason (there's an additional opening bracket I had to remove btw.). The log output then looks like like this:

Waiting 3 seconds to give Ethernet shield time to get ready...
Device family: 97
Device variant: 100
Start network services
Starting MDNS service with hostname BSB-LAN
Setup com

Then the output stops and the Arduino is gone.

So I commented out the last Serial.println and got the following results:

With an IP given:

Setup complete
192.168.0.1:1883
y??o��? ?X;???sfFL$?*S'??`??Z?i"?��?
192.168.0.1:1883
192.168.0.1
192.168.0.1
1883
1883
1883
Client ID: BSB-LAN
Will topic: bsb-lan/status
Failed to connect to MQTT broker, retrying...
192.168.0.1:1883

192.168.0.1:1883
192.168.0.1
192.168.0.1
1883
1883
1883
MQTT connection lost
192.168.0.1:1883
192.168.0.1
192.168.0.1:1883
192.168.0.1
192.168.0.1
1883
1883
1883

With a server name ("bunny") given:

Setup complete
bunny:1883
y????�? ?X;???sfFL??*?'??`??Z?h"?��?
bunny:1883
bunny
bunny
1883
1883
1883
Client ID: BSB-LAN
Will topic: bsb-lan/status
Failed to connect to MQTT broker, retrying...
bunny:1883

bunny:1883
bunny
bunny
1883
1883
1883
MQTT connection lost
bunny:1883
^A
bunny:1883
bunny
bunny
1883
1883
1883

Note that the first two tries always give a different result in the second line than all subsequent tries.

fredlcore · 2024-04-01T15:28:37Z

Thanks for responding so quickly. The second line (with the "garbage") is expected, as it just prints out whatever there is in the reserved memory location. The fact that it contains the right content (i.e. hostname and port) shows that it works up to here. The rest is all correct as well: The string gets cut into the right components. And yet you still get the Will Topic as the domain name that your DNS server has to resolve?

Even with your workaround at the top, you should arrive at the same content until you do the DNS lookup where you query the host by name. Since both Arduino Due and ESP32 use the same PubSubClient library that we distribute with this project, any kind of overflow would have to happen on the ESP32 as well - and it works fine on my setup. PubSubClient even provides three different types of the setServer function, one which accepts an IP address based on the IPAddress data type (that's the one you're using), and two more which accept an int array and a char array, of which my code uses the latter one. So I'm really puzzled where this mixup actually takes place and why it doesn't take place on the ESP32.

To test this further, could you go to PubSubClient.cpp's setServer functions and change the const char * variant to this one:

PubSubClient& PubSubClient::setServer(const char * domain, uint16_t port) {
Serial.println(domain);
Serial.println(port); 
   this->domain = domain;
    this->port = port;
    return *this;
}

And then in boolean PubSubClient::connect(const char *id, const char *user, const char *pass, const char* willTopic, uint8_t willQos, boolean willRetain, const char* willMessage, boolean cleanSession) {
please add

Serial.println(this->domain);
Serial.println(this->port);

as the first lines at the beginning of the function. Take care that you pick the right connect() function, this is the last and longest one.

You can remove all the previous debug messages so that it's clear which one is the one you added just now.

fredlcore · 2024-04-01T15:33:34Z

One more test: Can you enable version check and check in your DNS that BSB-LAN tries to look up and connect to bsb-lan.de in order to get the current version available? This also uses the get host by name functionality that I have recently implemented with MQTT. So if this one works when looking up the version number, then there is no hardware or library reason why it shouldn't also work with MQTT...

drfjordan · 2024-04-01T16:34:39Z

I have enabled version check and it seems to work (see attachment) -> This means that DNS as such does resolve the corresponding host. Am Mo., 1. Apr. 2024 um 17:33 Uhr schrieb fredlcore < ***@***.***>:

…

One more test: Can you enable version check and check in your DNS that BSB-LAN tries to look up and connect to bsb-lan.de in order to get the current version available? This also uses the get host by name functionality that I have recently implemented with MQTT. So if this one works when looking up the version number, then there is no hardware or library reason why it shouldn't also work with MQTT... — Reply to this email directly, view it on GitHub <#623 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AS5ZPGUE72OUWDBLE6UJ4Y3Y3F46HAVCNFSM6AAAAABEADW2DKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZHE4DEMZSHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Mit freundlichem Gruß, Dr. Frank Jordan

fredlcore · 2024-04-01T16:47:27Z

Thanks for testing - so this confirms that the issue is not related to any incapabilities of the Arduino Due libraries, but must happen somewhere inbetween the parsing of the hostname and the connection within PubSubClient...

tiger42 · 2024-04-01T16:57:14Z

I've added the debug output as suggested (and added the name of the current method to not get confused by the output).
This is the result:

Failed to connect to MQTT broker, retrying...
PubSubClient::setServer
192.168.0.1
1883
Client ID: BSB-LAN
Will topic: bsb-lan/status
PubSubClient::connect
bsb-lan/status
1883

So the problem really lies somewhere between setting the hostname and connecting.

fredlcore · 2024-04-01T17:14:06Z

Thanks, that makes it even weirder. Because the setServer output shows that both domain and port have been stored as part of the PubSubClient object. It's basically encapsulated there and should remain as is until it is overwritten by another setServer function call. So maybe you can help me see something here that I don't see in this code in question:

    MQTTPubSubClient->setServer(mqtt_host, mqtt_port);
    String MQTTWillTopic = mqtt_get_will_topic();
    String MQTTRealClientId = mqtt_get_client_id();
    printFmtToDebug(PSTR("Client ID: %s\r\n"), MQTTRealClientId.c_str());
    printFmtToDebug(PSTR("Will topic: %s\r\n"), MQTTWillTopic.c_str());
    MQTTPubSubClient->connect(MQTTRealClientId.c_str(), MQTTUser, MQTTPass, MQTTWillTopic.c_str(), 0, true, PSTR("offline"));

The first line is the setServer call which internally does this:

    this->domain = domain;
    this->port = port;

@tiger42's outut has shown that this works as it should.
The subsequent lines only fill variables, they are not touching anything related to the PubSubClient:

    String MQTTWillTopic = mqtt_get_will_topic();
    String MQTTRealClientId = mqtt_get_client_id();
    printFmtToDebug(PSTR("Client ID: %s\r\n"), MQTTRealClientId.c_str());
    printFmtToDebug(PSTR("Will topic: %s\r\n"), MQTTWillTopic.c_str());

And if your output comes right after the call to PubSubClient's connect function, it also cannot be affected by anything that the PubSubClient library is doing here.

Can you please try and comment out the four lines above and instead fill the MQTTPubSubClient->connect call with hard-coded values? If this works, you can un-comment one line after the other and use that variable to see which variable introduces the error.

fredlcore · 2024-04-01T18:09:59Z

There was some unnecessary dealings with String instead of char arrays which I have just fixed in the most recent master version, it compiles, but I haven't tested whether it actually does what it should. But even if it doesn't, it might help to figure out if the problem lies in these String handlings...

drfjordan · 2024-04-01T18:28:21Z

I have tested the latest update and it seems to work now as expected. Used configuration as IP with port: "192.168.1.2:1883" Result: MQTT does connect and publish Used configuration with host name only "mqtt.jordan.home" Result: MQTT does connect and publish Many thanks for your help! Am Mo., 1. Apr. 2024 um 20:10 Uhr schrieb fredlcore < ***@***.***>:

…

There was some unnecessary dealings with String instead of char arrays which I have just fixed in the most recent master version, it compiles, but I haven't tested whether it actually does what it should. But even if it doesn't, it might help to figure out if the problem lies in these String handlings... — Reply to this email directly, view it on GitHub <#623 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AS5ZPGWHE32YMR5UCAIVH5LY3GPI3AVCNFSM6AAAAABEADW2DKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGI3TGMJVHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Mit freundlichem Gruß, Dr. Frank Jordan

tiger42 · 2024-04-01T18:33:34Z

I can also confirm, that the MQTT connection now works without problems.
I've also tested with IP/hostname and with/without the port.

fredlcore · 2024-04-02T00:34:16Z

Thanks to both of you for testing, it's really strange that this piece of code which is around for years, suddenly now makes a problem due to some other very small and rather unrelated code...

fredlcore closed this as completed Mar 31, 2024

fredlcore reopened this Apr 1, 2024

fredlcore closed this as completed Apr 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] MQTT connection fails #623

[BUG] MQTT connection fails #623

tiger42 commented Feb 29, 2024

fredlcore commented Feb 29, 2024

fredlcore commented Feb 29, 2024

tiger42 commented Feb 29, 2024 •

edited

fredlcore commented Feb 29, 2024 •

edited

fredlcore commented Feb 29, 2024

tiger42 commented Feb 29, 2024

fredlcore commented Mar 1, 2024 •

edited

tiger42 commented Mar 2, 2024

fredlcore commented Mar 3, 2024

tiger42 commented Mar 3, 2024

fredlcore commented Mar 3, 2024

drfjordan commented Mar 31, 2024 •

edited

fredlcore commented Mar 31, 2024 •

edited

drfjordan commented Apr 1, 2024 •

edited

tiger42 commented Apr 1, 2024

fredlcore commented Apr 1, 2024

tiger42 commented Apr 1, 2024

fredlcore commented Apr 1, 2024

fredlcore commented Apr 1, 2024

drfjordan commented Apr 1, 2024 via email

fredlcore commented Apr 1, 2024

tiger42 commented Apr 1, 2024

fredlcore commented Apr 1, 2024

fredlcore commented Apr 1, 2024

drfjordan commented Apr 1, 2024 via email

tiger42 commented Apr 1, 2024

fredlcore commented Apr 2, 2024

[BUG] MQTT connection fails #623

[BUG] MQTT connection fails #623

Comments

tiger42 commented Feb 29, 2024

fredlcore commented Feb 29, 2024

fredlcore commented Feb 29, 2024

tiger42 commented Feb 29, 2024 • edited

fredlcore commented Feb 29, 2024 • edited

fredlcore commented Feb 29, 2024

tiger42 commented Feb 29, 2024

fredlcore commented Mar 1, 2024 • edited

tiger42 commented Mar 2, 2024

fredlcore commented Mar 3, 2024

tiger42 commented Mar 3, 2024

fredlcore commented Mar 3, 2024

drfjordan commented Mar 31, 2024 • edited

fredlcore commented Mar 31, 2024 • edited

drfjordan commented Apr 1, 2024 • edited

tiger42 commented Apr 1, 2024

fredlcore commented Apr 1, 2024

tiger42 commented Apr 1, 2024

fredlcore commented Apr 1, 2024

fredlcore commented Apr 1, 2024

drfjordan commented Apr 1, 2024 via email

fredlcore commented Apr 1, 2024

tiger42 commented Apr 1, 2024

fredlcore commented Apr 1, 2024

fredlcore commented Apr 1, 2024

drfjordan commented Apr 1, 2024 via email

tiger42 commented Apr 1, 2024

fredlcore commented Apr 2, 2024

tiger42 commented Feb 29, 2024 •

edited

fredlcore commented Feb 29, 2024 •

edited

fredlcore commented Mar 1, 2024 •

edited

drfjordan commented Mar 31, 2024 •

edited

fredlcore commented Mar 31, 2024 •

edited

drfjordan commented Apr 1, 2024 •

edited