Failure of MQTT PubSubClient : Asyncronous operations not supported?

[northwestfrog]

I attempted to use the standard MQTT Client with your WIFI library and subscription related incomming information was confused with responses from AT+CISTATUS.
It sounds like the incomming information from the Serial port should be classified as response from incomming traffic (+IPD...) vs other responses from outgoing requests such AT+CISTATUS.

Your library is compatible enough with PubSubClient to compile, but some work has to be done to make the processing of the incomming serial traffic asynchronous (SerialEvent) so responses to outgoing request are kept distinct from incoming requests from the different connected clients.

I have spent a lot of time trying to figure out why the MQTT subscriptions were not working, Unless you already have an Idea to fix your library, I may be able to provide some paths to fix the issue, this will require some work.

[canguy247]

I can confirm this behaviour.

[brettanomyces]

This is exactly the use case I have, gutted it wont work.

[bportaluri]

There are many MQTT libraries around.
Can you please share the sketch you are using or send me a link to it?

[northwestfrog]

The PubSubClient described earlier is the following. Examples are included
https://github.com/knolleary/pubsubclient

[vannoo67]

I just had some success using WiFiEsp with Adafruit_MQTT_Library ( https://github.com/adafruit/Adafruit_MQTT_Library )

I modified their ESP8266 example (https://github.com/adafruit/Adafruit_MQTT_Library/blob/master/examples/mqtt_esp8266/mqtt_esp8266.ino) for use with the ESP8266 as an adapter rather than the host

by including < WiFiEsp.h > & < WiFiEspClient.h >
instead of < ESP8266WiFi.h >

Specifing my local Mosquitto server;
#define AIO_SERVER "10.1.1.2"
#define AIO_SERVERPORT 1883
#define AIO_USERNAME ""
#define AIO_KEY ""

Use WiFiEspClient instead of WiFiClient
//WiFiClient client;
WiFiEspClient client;

Init the serial port that the ESP8266 is connected to, (note: I have WIFI_SERIAL #defined as Serial3 on my Mega, but you could probably use SoftSerial)
WIFI_SERIAL.begin(115200);

and init WiFi with the Serial Stream
WiFi.init(&WIFI_SERIAL);

just before
WiFi.begin(WLAN_SSID, WLAN_PASS);

and away she goes.

I'm looking forward to using knolleary/pubsubclient, but for the time being, this may well be a workable solution.

[vannoo67]

While I got further with Adafruit_MQTT_Library than knolleary/pubsubclient, it turns out that subscribing to more than one topic is unreliable. (Sometimes works, mostly MQTT_connect() fails )

[diegommarino]

Hey, don't know if it may help, but I made a MQTT example and post it in this issue #37 .

[thinkingste]

Hi diegommarino,
Your example is similar to mine but it's only reliable for the publishing part. The problem discussed here is the subscription part. Some of the subscribed messages are missing randomly.

northwestfrog,
Could you please share your idea of how to fix this issue? I can see on ExpDrv class that +IPD tag is already checked on availData function and it's used by available function on WiFiEspClient class.

[northwestfrog]

Hi, you want to completely separate the reception of the messages and the
messages sent.

incomming traffic has a certain pattern. +IPD is an unsollicited incomming
message, it should go in its own incomming higher level queue (one per
channel?). As for other incomming such as +CISTATUS, they should go in a
different queue, one that is related to responses from outgoing requests.

Triggers for reading the incomming serial information should be often, but
should only be for populating the different incomming queues. Once done,
the process expecting a response from AT+CISTATUS, would query the higher
level queue to see if the response has arrived (non blocking call).

Jean-Francois Gagnon
Phone: +1-418-843-8835
Mobile: +1-418-953-3172
Email: northwestfrog@gmail.com
Address: 7790 rue de Barcelone, Quebec QC G2C 2G6
http://www.facebook.com/northwestfrog
http://ca.linkedin.com/in/northwestfrog
http://plus.google.com/+JeanFrancoisGagnon
Get a signature like this: Click here!
http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=999850926168115

2016-07-09 8:13 GMT-04:00 thinkingste notifications@github.com:

Hi diegommarino,
Your example is similar to mine but it's only reliable for the publishing
part. The problem discussed here is the subscription part. Some of the
subscribed messages are missing randomly.

northwestfrog,
Could you please share your idea of how to fix this issue? I can see on
ExpDrv class that +IPD tag is already checked on availData function and
it's used by available function on WiFiEspClient class.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#40 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ARhyJ0-rJt56FDF0PwbPoRlSy6OfXvSuks5qT5BhgaJpZM4IOfIP
.

[jieweiyang]

I have the same environment, but little bit strange.
While use PubSubClient->WifiESP->ESP, subscribe to topic, then publish to same topic, i can see packets between MQTT broker and ESP, and ESP->pubsubclient also received the message.
Maybe because the "[mqtt].loop()" is right after the command of publish?

But use other MQTT client to publish in same channel, while MQTT broker got message, the PubSubClient never show received.

I did sniffer on IP package, and confirmed the communication between MQTT and ESP are ACK.
Then monitor the COM output of ESP, it did has +IPD received.
But in the PubSubClient, even "[wifi].available();" never shows data.

More digging in the COM output package use "Serial Port Monitor".
The difference between "received" and "not received" is:
Received: Message at the end of COM package.
Not Received: Message in mid of COM package.

Obviously, the COM package are flooded by CISTATUS and corrupted.

Maybe the problem is how WifiESP communicate with ESP via UART?
Or maybe the ESP UART output issue?

[jieweiyang]

Believe most of you use the sample code from PubSubClient, the reason of CISTATUS flooded is the interval of "[mqtt].loop()", by default, it's following the main loop(), that means it will run on every clock,

I just make a interval around 100ms use mills() (I always try avoid use delay()), then everything working "perfect" except a delay between every send/receive.

I guess this maybe is the hardware limitation, the baudrate of COM is far slower then the Arduino clock.

I tried on ESP8266/Arduion + NodeMCU, there is no interval required.

[eumb]

jieweiyang , I did went for the solution (add 100ms ) as you provided but I am not sure where to add the delay.
I did as below but still cannot get any messages from the broker.

void loop() {

unsigned long currentMillis = millis();
if (!client.connected()) {
reconnect();
}

if (client.connected())

  client.loop();

if (currentMillis - previousMillis > interval) {
// save the last time you blinked the LED
previousMillis = currentMillis;
client.loop();
}
}

Thanks for any help

[jieweiyang]

basically, anywhere you can, just make sure there is some kind of delay between each "client.loop()".

Your codes, the "if (client.connected()) client.loop();" will run on every main loop while MQTT is connected, so, the result still flooding.
Actually, i think you can remove this, as your interval is controlling by the “if(current****”

[Muplex]

Hei guys,
If I well undestand your posts, I have the exact same problem of you!

I'm trying to build a little sensors lan in my home between Arduino uno nodes and ESP8266 connected to the arduino (with some sensor each board).
As you know very well, these is the only arduino library (correct me if I'm wrong) that support SoftSerial and also MQTT (pubsubclient by knolleary) . (ESPduino is similar but don't work with software serial, right?)

All seems to work very fine but some times, randomly, I got delay in esp incoming message and subsequent [WiFiEsp] TIMEOUT: X. OR ESP reading incorrect message eg, "MyTopic" -> "+I" as message, but the real message sended is totaly different!!!!!! Some other times, the topic seems to be truncate in the last letters.

I confirm having a little (100ms) delay before the client.loop() helps a lot but not work perfectly.

Have you got discovered something about How To Solve these issue ???
MQTT_Adafruit works better than knolleary's pubsubclient???

[antonyjt]

Hi, I've also had very similar problems with the incoming data being truncated, and getting TIMEOUT messages. Have spend ages trying to debug/fix this.

Eventually, what worked for me has been to change the serial baud rate to 9600, while also making sure the loop() function is called frequently (every 30ms if possible).

Unfortunately, the loop() function is quite slow when running at 9600, so I also edited PubSubClient, and remove the connected() check at the beginning of the loop() function.

boolean PubSubClient::loop() {
//AJT: Removed the check for connected, as this was making this loop very slow.
//if (connected()) {
unsigned long t = millis();.......etc

This seems to work 95% of the time, although I still get a TIMEOUT every now and then.

[ongvanor]

i do have same problem, sometimes i dont receive the subscribed msgs. sometimes i receive sometimes not. do we have fix for this? cause this is the only library i know that supports software serial

[zhoufusong]

i do have same problem. there are any solution now?

[Muplex]

I discover that total message sended (topic + message) works very good if it is short messages.
Try to reduce the path length of the topic with a number code instead of long words.

Home/1/2/1
ON

works better than

Home/Living_room/Light/RGB_light
Turn it ON

I hope to be helpfull

[Hami24]

Not sure if it's helpful or not, but I made an example where you handle the MQTT communication with ESP8266 and send commands over serial connection to Arduino Uno. I used Adafruit MQTT library, but it will work PubSub too since PubSub doesn't have any problems on the ESP8266. (For people who want to use MQTT with Arduino).

Basically, in my example you upload one code to the ESP8266 and the other to the Arduino development board, link to the code:
https://github.com/Hami24/ArduinoIoT

P.S. I tried to do the same with this library, but didn't have any luck doing that.

[haoxiangli6]

hey, I think you guys could try to change _SS_MAX_RX_BUFF in SoftwareSerial.h.

[everardogentil]

Hi, there! I am trying to get updates from MQTT broker in my arduino/esp8266 circuit with WiFiEsp but even if I subscribe to the right topics I get the "[WiFiEsp] TIMEOUT: X" message. Is there any update for this problem?

[alanzhaonys]

Confirmed this issue still exists. I still couldn't get WiFiEsp to work with PubSubClient, particularly the receiving of message.

[JAndrassy]

Confirmed this issue still exists. I still couldn't get WiFiEsp to work with PubSubClient, particularly the receiving of message.

try the new WiFiEspAT library
https://github.com/jandrassy/WiFiEspAT#Why-a-new-wifiesp-library

[alanzhaonys]

Confirmed this issue still exists. I still couldn't get WiFiEsp to work with PubSubClient, particularly the receiving of message.

try the new WiFiEspAT library
https://github.com/jandrassy/WiFiEspAT#Why-a-new-wifiesp-library

Hi,

First of all, thanks for the response.

As matter of fact, I just spent couple hours flashing the new AT firmware and switched to your library, the result was it wouldn't even detect the ESP8266.

LoBo's repository doesn't have the AT firmware 1.7.1 that is required by the library so I had to download it from: https://www.espressif.com/en/support/download/at - ESP8266 NonOS AT Bin V1.7.3 - I download and flashed the latest firmware. Correct me if this is the wrong AT firmware.

I then switched to the WiFiESPAt library, but it won't even get to the point where WIFI could connect.

Below are the commands I used to flash the ESP8266. I hope the problem was the firmware.

I would really like to get it working, at this point I am about to give up.

Thank for your help.

`
python3 esptool.py --port /dev/cu.usbserial-14220 erase_flash
Python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x00000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/boot_v1.7.bin
python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x3fe000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/blank.bin
python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x3fc000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/esp_init_data_default_v08.bin
python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x01000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/at/1024+1024/user1.2048.new.5.bin
python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x81000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/at/1024+1024/user2.2048.new.5.bin

`

[JAndrassy]

https://github.com/jandrassy/WiFiEspAT#at-firmware-versions
https://github.com/jandrassy/WiFiEspAT#getting-started

[alanzhaonys]

Confirmed this issue still exists. I still couldn't get WiFiEsp to work with PubSubClient, particularly the receiving of message.

try the new WiFiEspAT library
https://github.com/jandrassy/WiFiEspAT#Why-a-new-wifiesp-library

Hi,

First of all, thanks for the response.

As matter of fact, I just spent couple hours flashing the new AT firmware and switched to your library, the result was it wouldn't even detect the ESP8266.

LoBo's repository doesn't have the AT firmware 1.7.1 that is required by the library so I had to download it from: https://www.espressif.com/en/support/download/at - ESP8266 NonOS AT Bin V1.7.3 - I download and flashed the latest firmware. Correct me if this is the wrong AT firmware.

I then switched to the WiFiESPAt library, but it won't even get to the point where WIFI could connect.

Below are the commands I used to flash the ESP8266. I hope the problem was the firmware.

I would really like to get it working, at this point I am about to give up.

Thank for your help.

`
python3 esptool.py --port /dev/cu.usbserial-14220 erase_flash
Python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x00000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/boot_v1.7.bin
python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x3fe000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/blank.bin
python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x3fc000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/esp_init_data_default_v08.bin
python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x01000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/at/1024+1024/user1.2048.new.5.bin
python3 esptool.py --port /dev/cu.usbserial-14220 write_flash 0x81000 ~/Desktop/arduino/At_firmware_bin1.54/ESP8266_NonOS_AT_Bin_V1.7.3\ 4/bin/at/1024+1024/user2.2048.new.5.bin

`

OMG......YOU'RE FREAKING AWESOME....I HAD SUCCESS!!!! Just when I am about to give up, I gave it a final look.

I went back to the instructions and realized I might have flashed the firmware incorrectly.

This is the correct command that I used:
python3 esptool.py --port /dev/cu.usbserial-14220 write_flash --flash_size 2MB-c1 0x0 boot_v1.7.bin 0x01000 at/1024+1024/user1.2048.new.5.bin 0x1fb000 blank.bin 0x1fc000 esp_init_data_default_v08.bin 0xfe000 blank.bin 0x1fe000 blank.bin

Again, I downloaded the firmware from here: https://www.espressif.com/en/support/download/at - ESP8266 NonOS AT Bin V1.7.3

(I know this is off topic, I still couldn't find any info on what flash address to use for different ESP8266, if you can shed some lights, that would be great).

Everything works now, publishing and receiving of MQTT data. I lost almost 3 days on this exploring different options, YOU SAVED MY INSANITY!

AND I know this have been bugged so many people based on my research, I've tried other MQTT libraries and the well know delay(100) after the MQTT client loop, none of it worked.

I can now confirmed that ESP8266 AT 1.7.3 firmware + PubSubClient + WiFiEspAT will definitely work in publishing and receiving MQTT!

[alanzhaonys]

https://github.com/jandrassy/WiFiEspAT#at-firmware-versions
https://github.com/jandrassy/WiFiEspAT#getting-started

See my response above. I had success, that was exactly what I read, then gave it another try and had success!

Thank you so much sir!

[JAndrassy]

https://github.com/jandrassy/WiFiEspAT#at-firmware-versions
https://github.com/jandrassy/WiFiEspAT#getting-started

See my response above. I had success, that was exactly what I read, then gave it a another and had success!

Thank you so much sir!

it is the same addresses for all sizes larger then 1 MB. so the esptool command line from README should be used.
for 0.5 and 1 MB the Lobo build must be used.

[aryvini]

I had success upgrading the AT firmware and using the WiFiEspAT library to connect to a MQTT server, provided by losant platform. My current hardware is made of an Arduino mega 2560 and I had trouble when using the softwareserial library to connect to the ESP01 module. However, when utilizing the native Serial1 port, everything works like a charm. It was the only solution I've found so far to get a positive connection to the Losant's server. In the next few days, I'll be testing more the communication and utilization of the library. So far so good. Great job.

[mm0ck3r]

Thanks to vannoo67