WifiClientSecure, PubSubClient and WifiManager dont work together

[arrowcircle]

Basic Infos

WifiClientSecure, PubSubClient and WifiManager dont play nice together.

Hardware

NodeMCU dev board.

Description

When I use WifiManager, I cant connect to Secure MQTT broker. When I remove WiFiManager and connect to wifi manually - it works great.

Settings in IDE

[env:native]
platform = native
test_filter = native
debug_tool = native
lib_ignore =
Adafruit BMP280 Library
Adafruit Unified Sensor
PubSubClient
ArduinoJson
WifiManager

[env:esp01]
platform = espressif8266
board = esp01
framework = arduino
upload_speed = 921600
monitor_speed = 115200
test_filter = esp01
build_flags = -DDEBUG_ESP_WIFI

lib_deps =
Wire
I2C
SPI
Adafruit BMP280 Library@1.0.2
Adafruit Unified Sensor@1.0.2
PubSubClient@2.6
ArduinoJson@5.13.2
WifiManager@0.14

Sketch

#include <FS.h>
#include <Wire.h>
#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>
#include <deque>
#include <ArduinoJson.h>
#include <PubSubClient.h>

#include <DNSServer.h>
#include <ESP8266WebServer.h>
#include <WiFiManager.h>

char *mqtt_server = "mqtt.xxx";
char *mqtt_port = "8883";
char *mqtt_user = "xxx";
char *mqtt_password = "xxx";
char *mqtt_id = "12345";

WiFiClientSecure wifiClient;
PubSubClient client(wifiClient);

DNSServer dns;

bool shouldSaveConfig = false;

void ensureFsMounted(){
  if (SPIFFS.begin()) {
    Serial.println("mounted file system");
    if (SPIFFS.exists("/config.json")) {
      //file exists, reading and loading
      Serial.println("reading config file");
      File configFile = SPIFFS.open("/config.json", "r");
      if (configFile) {
        Serial.println("opened config file");
        size_t size = configFile.size();
        // Allocate a buffer to store contents of the file.
        std::unique_ptr<char[]> buf(new char[size]);

        configFile.readBytes(buf.get(), size);
        DynamicJsonBuffer jsonBuffer;
        JsonObject& json = jsonBuffer.parseObject(buf.get());
        json.printTo(Serial);
        if (json.success()) {
          Serial.println("\nparsed json");

          strcpy(mqtt_server, json["mqtt_server"]);
          strcpy(mqtt_port, json["mqtt_port"]);
          strcpy(mqtt_user, json["mqtt_user"]);
          strcpy(mqtt_password, json["mqtt_password"]);
          strcpy(mqtt_id, json["mqtt_id"]);

        } else {
          Serial.println("failed to load json config");
        }
        configFile.close();
      }
    }
  } else {
    Serial.println("failed to mount FS");
  }
}

void configModeCallback (WiFiManager *myWiFiManager) {
  Serial.println("Entered config mode");
  Serial.println(WiFi.softAPIP());

  Serial.println(myWiFiManager->getConfigPortalSSID());
}

void saveConfigCallback () {
  Serial.println("Should save config");
  shouldSaveConfig = true;
}

void initWiFiManager() {
  // The extra parameters to be configured (can be either global or just in the setup)
  // After connecting, parameter.getValue() will get you the configured value
  // id/name placeholder/prompt default length
  WiFiManagerParameter custom_mqtt_server("server", "mqtt server", mqtt_server, 64);
  WiFiManagerParameter custom_mqtt_port("port", "mqtt port", mqtt_port, 6);
  WiFiManagerParameter custom_mqtt_user("user", "mqtt user", mqtt_user, 64);
  WiFiManagerParameter custom_mqtt_password("password", "mqtt password", mqtt_password, 64);
  WiFiManagerParameter custom_mqtt_id("id", "mqtt id", mqtt_id, 64);

  //WiFiManager
  //Local intialization. Once its business is done, there is no need to keep it around
  WiFiManager wifiManager;

  //set config save notify callback
  wifiManager.setSaveConfigCallback(saveConfigCallback);
  wifiManager.setAPCallback(configModeCallback);

  //set static ip
  // wifiManager.setSTAStaticIPConfig(IPAddress(10,0,1,99), IPAddress(10,0,1,1), IPAddress(255,255,255,0));
  
  //add all your parameters here
  wifiManager.addParameter(&custom_mqtt_server);
  wifiManager.addParameter(&custom_mqtt_port);
  wifiManager.addParameter(&custom_mqtt_user);
  wifiManager.addParameter(&custom_mqtt_password);
  wifiManager.addParameter(&custom_mqtt_id);

  //reset settings - for testing
  // wifiManager.resetSettings();
  // SPIFFS.format();
  // ESP.eraseConfig();
  // ESP.reset();

  //set minimu quality of signal so it ignores AP's under that quality
  //defaults to 8%
  wifiManager.setMinimumSignalQuality();
  
  //sets timeout until configuration portal gets turned off
  //useful to make it all retry or go to sleep
  //in seconds
  //wifiManager.setTimeout(120);

  //fetches ssid and pass and tries to connect
  //if it does not connect it starts an access point with the specified name
  //here  "AutoConnectAP"
  //and goes into a blocking loop awaiting configuration
  if (!wifiManager.autoConnect()) {
    Serial.println("failed to connect and hit timeout");
    delay(3000);
    //reset and try again, or maybe put it to deep sleep
    ESP.reset();
    delay(5000);
  }

  //if you get here you have connected to the WiFi
  Serial.println("connected...yeey :)");

  //read updated parameters
  strcpy(mqtt_server, custom_mqtt_server.getValue());
  strcpy(mqtt_port, custom_mqtt_port.getValue());
  strcpy(mqtt_user, custom_mqtt_user.getValue());
  strcpy(mqtt_password, custom_mqtt_password.getValue());
  strcpy(mqtt_id, custom_mqtt_id.getValue());

  //save the custom parameters to FS
  if (shouldSaveConfig) {
    Serial.println("saving config");
    DynamicJsonBuffer jsonBuffer;
    JsonObject& json = jsonBuffer.createObject();
    json["mqtt_server"] = mqtt_server;
    json["mqtt_port"] = mqtt_port;
    json["mqtt_user"] = mqtt_user;
    json["mqtt_password"] = mqtt_password;
    json["mqtt_id"] = mqtt_id;

    File configFile = SPIFFS.open("/config.json", "w");
    if (!configFile) {
      Serial.println("failed to open config file for writing");
    }

    json.printTo(Serial);
    json.printTo(configFile);
    configFile.close();
    //end save
  }

  Serial.println("local ip");
  Serial.println(WiFi.localIP());
}

void ensureMQTTConnection() {
  client.setServer(mqtt_server, int(mqtt_port));
  if (client.connect(mqtt_id)) {
    Serial.println("Connected to MQTT server");
  } else {
    Serial.println("Could not connect to MQTT server");
  }
}

void sendData() {
  StaticJsonBuffer<300> JSONbuffer;
  JsonObject& JSONencoder = JSONbuffer.createObject();
 
  JSONencoder["device"] = "ESP32";
  JSONencoder["Temperature"] = "23.5 C";
  JSONencoder["Pressure"] = "101300 Pa";
  JSONencoder["id"] = mqtt_id;

  char JSONmessageBuffer[100];
  JSONencoder.printTo(JSONmessageBuffer, sizeof(JSONmessageBuffer));
  // Serial.println(JSONmessageBuffer);

  client.publish("data/sensors", JSONmessageBuffer);
}

void setup() {
  //clean FS, for testing
  // SPIFFS.format();
  
  Serial.begin(9600);
  Serial.println(F("Starting sensor"));

  ensureFsMounted();
  initWiFiManager();
  ensureMQTTConnection();
}
  
void loop() {
  ensureMQTTConnection();
  sendData();
  client.loop();
  delay(5000);
}

Serial Output

*WM: port
*WM: Adding parameter
*WM: user
*WM: Adding parameter
*WM: password
*WM: Adding parameter
*WM: id
*WM:
*WM: AutoConnect
*WM: Connecting as wifi client...
*WM: Using last saved values, should be faster
*WM: Connection result:
*WM: 3
*WM: IP Address:
*WM: 192.168.1.62
connected...yeey :)
local ip
192.168.1.62
*WM: freeing allocated params!
Could not connect to MQTT server
Could not connect to MQTT server
Could not connect to MQTT server
Could not connect to MQTT server

[uched41]

I had this problem a few days ago. Declaring the wifiManager in global scope, just before the WiFiClient works for me, no real reason here.
I also noticed that if I use: wifiManager.setSTAStaticIPConfig the mqtt does not work. These are random solutions but might we worth a try.

[tablatronix]

hmm did you test this in development branch?

[bkrajendra]

Yes @uched41 I also observed similar issue with MQTT.
With me actually MQTT doe not connect when we do first time WiFi configuration. But we need to reset ESP to connect MQTT. After reset MQTT works properly. This might be related to how WiFi is connected. and how MQTT is initialised or connected to brocker. Something is happening in sequence. As you need to have WiFi connection before you try mqtt.begin(), but you dont have it while WiFi config.
But I tried putting mqtt.begin in after wifi ip is printed but does not help me.
I used Dev branch.

[tablatronix]

There seems to be some kind of esp bug, where it wont reconnect properly if it thinks is already is.

[tablatronix]

are you using set static config?

[bkrajendra]

Im using static IP for STA.

[tablatronix]

#720

[shahabmusic]

I am having the same problem. When I use ESP8266WebServer, esp8266 crash when connecting to secure mqtt. If I don'tdeclare websaver and don't use webserver, esp8266 does not crash and secure mqtt connects with no issue.

here is the error
Attempting MQTT connection...Fatal exception 28(LoadProhibitedCause):
epc1=0x402159d4, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

Exception (28):
epc1=0x402159d4 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

here is the stack:

Decoding stack results
0x40228cf4: hash_dn at src/x509/x509_minimal.c line 363
0x40229487: br_x509_minimal_run at src/x509/x509_minimal.c line 1208
0x40229d8a: xm_append at src/x509/x509_minimal.c line 285
0x40223d70: br_ssl_hs_client_run at src/ssl/ssl_hs_client.c line 1863
0x40221d98: jump_handshake at src/ssl/ssl_engine.c line 1081
0x4022227d: br_ssl_engine_recvrec_ack at src/ssl/ssl_engine.c line 1206
0x40205f21: BearSSL::WiFiClientSecure::run_until(unsigned int, bool) at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 497
0x402228ec: br_ssl_hs_client_run at src/ssl/ssl_hs_client.c line 958
0x40222894: br_ssl_hs_client_init_main at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/xtensa-lx106-elf/include/sys/pgmspace.h line 67
0x402223a2: br_ssl_engine_hs_reset at src/ssl/ssl_engine.c line 1305
0x40206130: BearSSL::WiFiClientSecure::wait_for_handshake() at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 521
0x402062c3: BearSSL::WiFiClientSecure::connectSSL(char const*) at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 930
0x4020888c: esp_yield() at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\core_esp8266_main.cpp line 91
0x402014eb: delay at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\core_esp8266_wiring.c line 51
0x40204f2a: WiFiClient::connect(IPAddress const&, unsigned short) at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\libraries\ESP8266WiFi\src/include/ClientContext.h line 136
0x40100114: millis at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\core_esp8266_wiring.c line 183
0x40206386: BearSSL::WiFiClientSecure::connect(char const*, unsigned short) at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 218
0x40208fdc: PubSubClient::connected() at C:\Users\light\OneDrive\Documents\Arduino\libraries\PubSubClient\src\PubSubClient.cpp line 606
0x4020688b: PubSubClient::connect(char const*, char const*, char const*, char const*, unsigned char, unsigned char, char const*, unsigned char) at C:\Users\light\OneDrive\Documents\Arduino\libraries\PubSubClient\src\PubSubClient.cpp line 129
0x4010038c: umm_free at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\umm_malloc\umm_malloc.c line 1300
0x40206a74: PubSubClient::connect(char const*) at C:\Users\light\OneDrive\Documents\Arduino\libraries\PubSubClient\src\PubSubClient.cpp line 106
0x40208198: String::~String() at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\WString.cpp line 125
0x40202c08: reconnect() at C:\Users\light\OneDrive\Documents\Arduino\BearSSL_Validation_test_mqtt/BearSSL_Validation_test_mqtt.ino line 271
0x40206e0d: std::function ::operator= &>(std::function &) at c:\users\light\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\2.5.0-3-20ed2b9\xtensa-lx106-elf\include\c++\4.8.2/functional line 2331
0x4020555f: BearSSL::WiFiClientSecure::connected() at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 246
0x40202cf4: loop() at C:\Users\light\OneDrive\Documents\Arduino\BearSSL_Validation_test_mqtt/BearSSL_Validation_test_mqtt.ino line 292
0x40208938: loop_wrapper() at C:\Users\light\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\core_esp8266_main.cpp line 125

[shahabmusic]

I am having the same issue. when I use client.setInsecure(), it connects with no problems. however it crashes when I use client.setTrustAnchors(&cert).
I used to solve this issue by not declaring webserver. but now it does not work even if I don't declare webserver

[dontsovcmc]

Maybe it's out of memory? TLS connection needs about 25kb. I have a crash in TLS new operator after ConfigPortal: dontsovcmc/waterius#82 but I don't have a loop.

[chaseTfreeman]

Hello I'm having similar issues and hoping someone has a working example. My issue is that ".subscribe" is not calling the callback. I am seeing a "1" returned when I print the result of the subscribe.

I am using pubsubclient, wificlientsecure, and wifimanager. I'm using AWS as my MQTT server. I have no problem publishing but cannot subscribe successfully.

#include <ESP8266WiFi.h>
#include <PubSubClient.h>
#include <NTPClient.h>
#include <WiFiUdp.h>
#include <WiFiManager.h> //https://github.com/tzapu/WiFiManager

WiFiUDP ntpUDP; 
NTPClient timeClient(ntpUDP, "pool.ntp.org");

// Update these with values suitable for your network.


const char* AWS_endpoint = "[aws_string_censored]-ats.iot.us-east-1.amazonaws.com";

WiFiClientSecure espClient;
PubSubClient client(espClient);
long lastMsg = 0;
char msg[50];
int value = 0;

void setup_wifi() {

delay(10);

//Local intialization. Once its business is done, there is no need to keep it around
WiFiManager wifiManager;

//fetches ssid and pass from eeprom and tries to connect
//if it does not connect it starts an access point with the specified name
//here  "AutoConnectAP"
//and goes into a blocking loop awaiting configuration
//or use this for auto generated name ESP + ChipID
wifiManager.autoConnect();


//if you get here you have connected to the WiFi
Serial.println("connected...yeey :)");
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");

}

randomSeed(micros());

Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
}

void callback(char* topic, byte* payload, unsigned int length) {
// Switch on the LED if an 1 was received as first character
if ((char)payload[0] == '1') {
    digitalWrite(BUILTIN_LED, LOW);   // Turn the LED on (Note that LOW is the voltage level
    // but actually the LED is on; this is because
    // it is active low on the ESP)
    client.publish("outTopic", "1");
} else {
    digitalWrite(BUILTIN_LED, HIGH);  // Turn the LED off by making the voltage HIGH
    client.publish("outTopic", "0");
}

}

void reconnect() {
// Loop until we're reconnected
while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Create a random client ID
    String clientId = "ESP8266Client-";
    clientId += String(random(0xffff), HEX);
    // Attempt to connect
    if (client.connect(clientId.c_str(),"username","password")) {
    Serial.println("connected");
    //   Once connected, publish an announcement...
    client.publish("outTopic", "hello world");
    ... and resubscribe
    //   print 1 if client is subscribes succesfully
    Serial.println(client.subscribe("outTopic"));
    client.subscribe("outTopic"); //<<<<<<<<<<<< This is where it fails to call the callback.
    } else {
    Serial.print("failed, rc=");
    Serial.print(client.state());
    Serial.println(" try again in 5 seconds");
    // Wait 5 seconds before retrying
    delay(5000);
    }
}
}

void setup() {
pinMode(BUILTIN_LED, OUTPUT);     // Initialize the BUILTIN_LED pin as an output
Serial.begin(115200);

setup_wifi();

timeClient.begin();
while(!timeClient.update()){
    timeClient.forceUpdate();
}

espClient.setX509Time(timeClient.getEpochTime());


delay(1000);
if (!SPIFFS.begin()) {
    Serial.println("Failed to mount file system");
    return;
}
Serial.print("Heap: "); Serial.println(ESP.getFreeHeap());

// Load certificate file
File cert = SPIFFS.open("/cert.der", "r"); //replace cert.crt eith your uploaded file name
if (!cert) {
    Serial.println("Failed to open cert file");
}
else
    Serial.println("Success to open cert file");

delay(1000);

if (espClient.loadCertificate(cert))
    Serial.println("cert loaded");
else
    Serial.println("cert not loaded");

// Load private key file
File private_key = SPIFFS.open("/private.der", "r"); //replace private eith your uploaded file name
if (!private_key) {
    Serial.println("Failed to open private cert file");
}
else
    Serial.println("Success to open private cert file");

delay(1000);

if (espClient.loadPrivateKey(private_key))
    Serial.println("private key loaded");
else
    Serial.println("private key not loaded");



    // Load CA file
    File ca = SPIFFS.open("/ca.der", "r"); //replace ca eith your uploaded file name
    if (!ca) {
    Serial.println("Failed to open ca ");
    }
    else
    Serial.println("Success to open ca");

    delay(1000);

    if(espClient.loadCACert(ca))
    Serial.println("ca loaded");
    else
    Serial.println("ca failed");

Serial.print("Heap: "); Serial.println(ESP.getFreeHeap());

client.setServer(AWS_endpoint, 8883);
client.setCallback(callback);
}

void loop() {

if (!client.connected()) {
    reconnect();
}
client.loop();

}

[adeltc]

Any progress on this issue. I'm still seeing with the last versions of the wifimanager and PubSubClient. The symptoms I'm seeing the WifiManager Auto connect that stop broadcasting SSID in AP mode. As soon as I remove the TLS code (replace "WiFiClientSecure" with "WiFiClient" ), It works perfectly. When I put back "WiFiClientSecure", the AP mode stop working correctly, SSID is not visible anymore.

In the beginning, I though it was an out of memory issue then I moved the code that instantiates "WiFiClientSecure" after the Wifi Manager AutoConnect... so "WiFiClientSecure" is not instantiated before the WifiManager Autoconnect complete. Even then I still see the same issue.

Long story short, as soon as the code of "WiFiClientSecure" compiles, the WifiManager Autoconnect stops working correctly... even if we don't instantiate the variables.

[tablatronix]

Shrug no one has posted logs from dev

[chaseTfreeman]

@adeltc - My configuration works, except for the .subscribe from the pub/sub client. Perhaps you can share your code?

[adeltc]

I got my problem solved by Selecting "Basic SSL ciphers(lower ROM use)" in th Arduino Menu "Tools"-->"SSL Support".
If I select back "All ciphers" then my problem comes back: the SSID of of the Autoconnect in AP doesn't show up. Can't see neither with my phone or my laptop. So it's impossible to connect to the esp