UART rx timeout issue!

[hitecSmartHome]

Board

ESP32-Wrover

Device Description

Hardware Configuration

Version

v2.0.14

IDE Name

PlatformIO

Operating System

Windows10

Flash frequency

80

PSRAM enabled

yes

Upload speed

115200

Description

I'm using onReceive callback for Serial1.
Sometimes the packet is truncated and I got the last one or two bytes in the next packet.
No matter the packet timeout it seems.

Sketch

void setup(){
    Serial1.begin(MBUS_BAUD, SERIAL_8N1, MBUS_RX, MBUS_TX);
    Serial1.setPins(-1, -1, -1, MBUS_RTS);
    Serial1.setMode(MODE_RS485_HALF_DUPLEX);
    Serial1.setTxBufferSize(RESPONSE_BUFFER_SIZE);
    Serial1.setRxBufferSize(RESPONSE_BUFFER_SIZE);
    Serial1.setRxTimeout(RX_TIMEOUT);
    Serial1.onReceive([this](void) { handleRawPacket(); },true);
    Serial1.onReceiveError([this](hardwareSerial_error_t error) { handlePacketError(error); });
}

void handleRawPacket() {
    packetByteCount = 0;
    while (Serial1.available()) {
        respBuffer[packetByteCount] = Serial1.read();
        packetByteCount++;
    }
    #if MODBUS_DEBUG
    printf("[Modbus] - Got packet: ");
    for (size_t i = 0; i < packetByteCount; i++){
        printf("0x%02x ", respBuffer[i]);
    }
    printf("\n");
    vTaskDelay(DEBUG_DELAY_MS);
    #endif
    gotPacket = true;
}

void handlePacketError(hardwareSerial_error_t error){
    // Ignore the UART_BREAK_ERROR because it is expected.
    if( error == UART_BREAK_ERROR ){ return; }
    #if MODBUS_DEBUG
        printf("\n[Modbus] - Packet error code: %d\n", error);
        printf("[Modbus] - Packet error description: %s\n", uartErrorStrings[error]);
    #endif
}

Debug Message

Just one packet in two ( usually the last two bytes is trucated and comes as a second packet ).
I have tried RX_TIMEOUT from 3 to 20. Does not matter.

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[hitecSmartHome]

My basic idea:

// RX_TIMEOUT / (MBUS_BAUD / 10) = millisec
// 4 / (115200 / 10) = 3.4 ms
// 5 / (115200 / 10) = 4.3 ms
// 6 / (115200 / 10) = 5.2 ms
#define RX_TIMEOUT 4 // Have tried even RX_TIMEOUT 20 / No effect
#define RESPONSE_BUFFER_SIZE 2048

#define MBUS_BAUD 115200
#define MBUS_RX 35
#define MBUS_TX 32
#define MBUS_RTS 33

void setup(){
    Serial1.begin(MBUS_BAUD, SERIAL_8N1, MBUS_RX, MBUS_TX);
    Serial1.setPins(-1, -1, -1, MBUS_RTS);
    Serial1.setMode(MODE_RS485_HALF_DUPLEX);
    Serial1.setTxBufferSize(RESPONSE_BUFFER_SIZE);
    Serial1.setRxBufferSize(RESPONSE_BUFFER_SIZE);
    Serial1.setRxTimeout(RX_TIMEOUT);
    Serial1.onReceive([this](void) {
        // In theory, we got a whole packet.
        packetByteCount = 0;
        while (Serial1.available()) {
            respBuffer[packetByteCount] = Serial1.read();
            packetByteCount++;
        }
        gotPacket = true;
    },true);
    Serial1.onReceiveError([this](hardwareSerial_error_t error) { handlePacketError(error); });
}

void loop(){
    if( gotPacket ){
        processPacket();
        packetByteCount = 0;
        gotPacket = false;
    }
}

Sometimes the onReceive cb triggers before I got a whole packet. Packets are standard modbus packets. A lot of times It got cut on the last one or two ( sometimes even three ) bytes.

[hitecSmartHome]

We measured the byte times with an oscilloscope and they all come at the exact same time. No delay between bytes in a frame.
It seems to me when the onReceive cb is called, I can't read the whole uart data. Will test it further soon.

Meanwhile I print out the byte count in the cb

Serial1.onReceive([this](void) {
    packetByteCount = 0;
    int available = Serial1.available();
    printf("GOT PACKET COUNT: %d\n",available);
    while(available--){
        respBuffer[packetByteCount] = Serial1.read();
        packetByteCount++;
    }
    gotPacket = true;
},true);

The problem is that it does not happen every time, just sometimes.
Here is a more realistic approach of mine.

MODBUS Snippet

/* add a new modbus address to a given device */
boolean Modbus::writeNewAddress(uint16_t id, uint16_t serial, uint16_t newAddress){
    const byte writeBuffSize = 13;
    uint8_t buffer[writeBuffSize];

    buffer[0] = BROADCAST_ADDRESS;
    buffer[1] = WRITE_FUNC_CODE;

    buffer[2] = highByte(WRITE_NEW_ADDRESS_REG_ADDRESS);
    buffer[3] = lowByte(WRITE_NEW_ADDRESS_REG_ADDRESS);

    buffer[4] = highByte(WRITE_NEW_ADDRESS_DATA_QUANTITY);
    buffer[5] = lowByte(WRITE_NEW_ADDRESS_DATA_QUANTITY);

    buffer[6] = uint8_t(WRITE_NEW_ADDRESS_DATA_QUANTITY*2);

    buffer[7] = highByte(id);
    buffer[8] = lowByte(id);

    buffer[9] = highByte(serial);
    buffer[10] = lowByte(serial);

    buffer[11] = highByte(newAddress);
    buffer[12] = lowByte(newAddress);

    rawWrite(buffer,writeBuffSize);
    return waitResponse();
}

/* write the raw buffer to the serial */
void Modbus::rawWrite(uint8_t *data, int length) {
    int crc16 = getCRC_16(data, length);
    uint8_t buffer[length + 2];

    memcpy(buffer, data, length);
    buffer[length] = lowByte(crc16);
    buffer[length + 1] = highByte(crc16);

    lastFuncCode = buffer[1];
    Serial1.write(buffer, length + 2);
}

/* Check if the packet is valid or not */
boolean Modbus::isValidResponse() {
    boolean crcIsOk = isValidCRC();
    boolean hasError = hasErrorByte();
    boolean hasAnyData = packetByteCount > 0;
    boolean isValid = crcIsOk && hasAnyData && !hasError;
    #if MODBUS_DEBUG
      printf("[Modbus] - Is CRC valid: %s\n", crcIsOk?"true":"false");
      printf("[Modbus] - Has error byte: %s\n", hasError?"true":"false");
      printf("[Modbus] - Has any data: %s\n", hasAnyData?"true":"false");
    #endif
    return isValid;
}

/* Wait for a response */
boolean Modbus::waitResponse() {
    while (!gotPacket) { vTaskDelay(1); }
    gotPacket = false;
    return isValidResponse();
}

MODBUS usage

/* Add new addresses to the modules on the bus */
for(initialModule module : initialModules) {
  int newAddress = getNewAddress();
  if( newAddress == -1 ){continue;}
  boolean result = modbus.writeNewAddress(module.id, module.serial, newAddress);
  if(!result){ continue; }
  addFreshModule(module.id, module.serial, newAddress);
}

Main app flow

write something -> wait for a response -> process response -> write something...

[SuGlider]

Sometimes the onReceive cb triggers before I got a whole packet. Packets are standard modbus packets. A lot of times It got cut on the last one or two ( sometimes even three ) bytes.

ESP32 Arduino UART is based on the UART IDF implementation. The way it works follows these "rules":

• the UART driver will trigger the OnReceive() callback whenever the internal SoC FIFO reaches a threshold from 5 to 120 bytes. This threshold can be set using bool HardwareSerial::setRxFIFOFull(uint8_t fifoBytes).
• It will also trigger the callback whenever an RX timeout (calculated based on 1 bit at the current baud rate) happens using bool HardwareSerial::setRxTimeout(uint8_t symbols_timeout).

There is further information in the header file at https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/HardwareSerial.h#L215-L240

Therefore, if the MODBUS frame has, for instance, 50 to 100 bytes, it is possible to set the RxFIFOFull to 110 bytes and the callback will be able to read it all, as in the code, you have presented, would do.

But if the frame has, for instance, 300 bytes, the callback will be called for the first 120 received bytes (default value) and available() will inform that there are, at that time, 120 bytes in the buffer when it is executed.
The next 120 bytes (FIFO Full) will trigger the callback again, in a next time, and the remaining 60 bytes will be triggered by RX Timeout (callback). It means that for receiving 300 bytes, the callback function would be called 3 times. The callback routine would need to assemble the MODBUS packet along the time and sequencial calls.

Let's say the a MODBUS packet would timeout a received packet if no UART communication happens after 100 ms from the last received byte. This timeout can be used to check if the complete packet has been received in a sigle execution of the onReceive() callback. Another possibility could be that the MODBUS frame has an specific field to mark the end of the packet, it could also work for waiting until the complete frame is received.

This timeout can be as high as necessary (30 seconds, for instance) or as low as the double of RX Timeout value.
It runs on a separated task, therefore it doesn't delay the main sketch execution (setup()/loop()), which runs on its own task.

In order to retrieve all the, for instance, 300 bytes, it would be necessary to run a code like this:

Serial1.onReceive([this](void) {
    uint32_t packetByteCount = 0;
    const unsigned long packetTimeout_ms = 100;  // set it to whatever is necessary
    
    unsigned long lastByteReceivedTime = millis();
    // loop will only exit on "Packet timeout"
    while(millis() < lastByteReceivedTime + packetTimeout_ms){
        if (Serial1.available()) {
            respBuffer[packetByteCount] = Serial1.read();
            packetByteCount++;
            lastByteReceivedTime = millis();
        }
    }
    printf("GOT PACKET COUNT: %d\n", packetByteCount);
    gotPacket = true;
},true);

Please check/run some examples here:
https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/OnReceive_Demo/OnReceive_Demo.ino
https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/RxFIFOFull_Demo/RxFIFOFull_Demo.ino
https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/RxTimeout_Demo/RxTimeout_Demo.ino
https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/onReceiveExample/onReceiveExample.ino
https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/RS485_Echo_Demo/RS485_Echo_Demo.ino

[hitecSmartHome]

Thank you for the detailed answer.
The problem is that I got only 8 bytes at once where it fails.
I have several hardwares on the bus which will send back 8 bytes each.
It never reaches the 120 byte treshold.

Can it be that the fifo is not emptied when I read the serial data and the packets are left in there?
I have three scenarios in my app.

1. Master scans the bus ( writes a broadcast packet ) -> slaves are answering one by one ( 11 byte / answer )
2. Master starts to give new addresses to the slaves -> ( 8 byte / answer )
3. Normal communication ( answer can be a really big frame )

In the first scenario, there is no problem at all ever. I did hundreads of tests, never fails.
In the second scenario, it fails a lot of times and random packets but all 8 bytes long
In the third scenario, never fails.

[hitecSmartHome]

If the fifo full triggers the cb i need a way to reset the fifo buffer when I got a packet.

[hitecSmartHome]

For example now i have 7 external hardware on my bus.
If each sends back a packet thats 8*7 bytes which is 56 bytes max.
If i understand correctly, the default fifo buffer is 120 bytes.
Even if the Serial1.read() does not reset the fifo buffer I should have no problem reading all of the packets.

I have copyed your example to my sketch. It just frozen and the gotPacket variable never triggers.

Serial1.onReceive([this](void) {
    uint32_t packetByteCount = 0;
    const unsigned long packetTimeout_ms = 200;  // set it to whatever is necessary
    
    unsigned long lastByteReceivedTime = millis();
    // loop will only exit on "Packet timeout"
    while(millis() < lastByteReceivedTime + packetTimeout_ms){
        if (Serial1.available()) {
            respBuffer[packetByteCount] = Serial1.read();
            packetByteCount++;
            lastByteReceivedTime = millis();
        }
    }
    if(isNewAddr){
        // Write my buffer to serial to see what we got.
        printf("GOT PACKET COUNT: %d\n",packetByteCount);
        for (size_t i = 0; i < packetByteCount; i++){
            printf("0x%02x ", respBuffer[i]);
        }
        printf("\n");
    }
    gotPacket = true;
},true);

[hitecSmartHome]

I modified this to this

Serial1.onReceive([this](void) {
    packetByteCount = 0;
    unsigned long lastByteReceivedTime = millis();
    while(millis() < lastByteReceivedTime + PACKET_TIMEOUT_MS_TEST){
        if (Serial1.available()) {
            respBuffer[packetByteCount] = Serial1.read();
            packetByteCount++;
            lastByteReceivedTime = millis();
        }
        if( millis() - lastByteReceivedTime > 150 ){
            break;
        }
    }
    if(isNewAddr){
        printf("GOT PACKET COUNT: %d\n",packetByteCount);
        for (size_t i = 0; i < packetByteCount; i++){
            printf("0x%02x ", respBuffer[i]);
        }
        printf("\n");
    }
    gotPacket = true;
},true);

I still got dropped bytes on the bus
Tried with onlyOnTimeout set to false, to true, doesn't work.

[SuGlider]

Could the necessary timeout for modbus bytes to arrive be higher than 200ms?
It sounds like the bytes are not arriving within the "timeout" interval.

Could you please try checkin what would be the highest timeout for your modbus?
Maybe longer, like a couple seconds?

[TD-er]

Just to comment on the timeout....
There are quite a few Modbus devices which need about 200 msec to reply.
For example SenseAir S8 may sometimes take > 200 msec to reply.

So I would suggest to start experiment with higher values first to see what your device may need.

[hitecSmartHome]

We had an issue with our hardwares. ( we have custom hardwares on the modbus )
Indeed there was a software bug which caused the packets to cut in half. Hovewer this break was only 1.2ms, max 3.5ms.
A whole packet comes throught in max 3ms. It is not a standard modbus, we made it a little bit faster.

Here is a good request and a good response on oscilloscope

Here is a good request and a bad response on oscilloscope

This was our bad, ofc but when I changed the rxTimeout to 5 or 6 or even to 10 this gap should have been a no problem scenario.

I have tried the rxTimeout from 4 to 10