Feature Request: Callback for "Buffer near full" for HardwareSerial class or HW-Flow-Control implementation

[Apollon77]

I currently try to integrate an ESP8266 together with Arduino MEga2560 in a project and the most common problem with ESP8266 (especially for Non-Mega devices) is that it sends data when they are available - and the TX buffer in ESP8266 is bigger then the RX buffer in the standard HardwareSerial class.
Some ideas in the net are to change the buffer size on Arduino and so modifying the classes. This is an option but when the device has more to do in the loop it may be not enough to read the data before ESP has send so many data that RX buffer in Arduino is overload.

ESP8266 supports hardware flow control in general so it should be able to control the data that will be sended ... but we need a trigger.

My idea was to add some kind of optional callback call/interrupt as soon as the RX buffer is filled to to (e.g.) 80% or such, so that the device can react on it and use the flow control to stop data sending before buffer overflows in a first step till it's avble to handle that ...

What do you think of such an idea?!

The alternative (or real solution) would be to support HW Flow Control (CTS/RTS) in HW-Serial, so no buffer overflow ever can happen ...

Ingo

[matthijskooijman]

I currently try to integrate an ESP8266 together with Arduino MEga2560 in a project and the most common problem with ESP8266 (especially for Non-Mega devices) is that it sends data when they are available - and the TX buffer in ESP8266 is bigger then the RX buffer in the standard HardwareSerial class.

I'm not exactly sure if the buffer sizes are really relevant here? That the TX buffer is bigger, only means that your sending code will start blocking later, but as long as you keep writing new bytes before the TX buffer is empty, the transmitter will continuously be transmitting bytes regardless of the buffer size. E.g., if I do Serial.write(buf, 128), that will write 128 bytes in direct succession, regardless of the buffer size.

My idea was to add some kind of optional callback call/interrupt as soon as the RX buffer is filled to to (e.g.) 80% or such, so that the device can react on it and use the flow control to stop data sending before buffer overflows in a first step till it's avble to handle that ...

A callback could work, though that means running code in interrupt context, which is generally tricky. I can see two other ways to support this:

1. Add direct support for flow control to HardwareSerial. e.g. do something like Serial.setCTS(4) to configure the pin number to use for RX flow control (note that CTS/RTS/RTR naming is a bit vague in this regard, so perhaps setFlowControlPins(x, y) might be better?) The advantage of this approach is that implementing TX flow control is a lot easier, if you would need some way to pause TX buffer processing externally (which would be needed for TX flow control), I'm afraid that all kinds of race conditions and deadlocks will ensue.
2. Expose the number of free bytes in the RX buffer to the sketch, so loop() can regularly check that and set an output pin accordingly. If you hardcode the buffer size in your loop, you can even do this now: digitalWrite(FLOW_PIN, Serial.available() > (int)(64 * 0.8));

[Apollon77]

Adding flow control in general (for HW-Serial and maybe also New-Soft-Serial) would be awesome because it will be the "real" solution in my eyes. I have found that for Teensy something like This was added to code by PaulStoffregen (https://forum.pjrc.com/threads/29446-Teensy-Hardware-Flow-Control-RTS-CTS --> PaulStoffregen/cores@c2f550d), but only the possibility to set the pins and to allowit in general I saw there. So there is no automatic buffer-fill-status depending logic what would be the real solution in my eyes ...

The second option would not help me in my special case because I can not guarantee that the check in "loop" will be fast enough to detect it before the buffer overflows. That's why the "callback/interrupt" idea ... the interrupt logic would be very simple: set the flow control pin to HIGH and set a boolean flag for "data available" so that the next loop run can handle the data and reset the control pin to get all the other data ...

[cmaglie]

Does the AVR support flow control in hardware? I see that SAM and SAMD have hardware support for that, but implementing this feature on AVR will probably require a lot more work.

[matthijskooijman]

@cmaglie, some AVR chips support flow control in hardware, like the 32u4 (but the Leonardo does not have those pins broken out, IIRC). The 328p doesn't support it.