Problems with Software Flow Control

[TG9541]

I'm reporting on neundorf/CuteCom 7da86b3 compiled on Ubuntu 14.04 using up-to date qt5-default and libqt5serialport5-dev.

In my use case, I need software flow control with XON/XOFF (DC1 0x11, DC3 0x13). After receiving CR the embedded device sends XOFF (0x13), does some processing, sends some text followed by a LF, and then sends XON (0x11) to indicate that the next line of text can be received and processed.

My expectation is that CuteCom stops sending characters in the interval between XOFF and XON, and that it hides the control characters, just like CR and LF, unless "Settings/Display Control Characters" is checked.

However, although I didn't check "Settings/Display Control Characters", CuteCom displays the hex values for DC1 and DC3:

HEX 80 1 DUMP␍<0x13>␊
  80   0  0  4 44 55 4D 50  0  0  0  0  0  0  0  0  0  ___DUMP_________ ok␊
<0x11>

Flow control as such doesn't appear to be working:

When I send a text file using the "script" feature, in the combined rendering of sent and received characters below, CuteCom sends the sequence ": dec -1" even after receiving a , causing an overrun:

<0x11>: inc 1 0 +! ;<0x13>: dec -1 
<0x11>0 +! ;<0x13> :0?

My expectation is that CuteCom honors software flow control, and after receiving waits with sending new characters until has been received.

Please note that other communication programs, like MiniCom, show a similar behavior regarding the visibility of and . It's possible that "I'm doing it wrong" or that something in underlying communication layers is broken.

[TG9541]

I went through the CuteCom source code, and apparently it's relying on flow control on the serial device layer. It's likely that the root cause is related to a bug in Debian, and the only thing CuteCom could do is work around the issue.

Edit1: the problem might be rather intricate: http://linux-serial.vger.kernel.narkive.com/ufjVkd8J/serial-flow-control-appears-broken , and in this case the bottom line is that assumptions about flow control on the driver level have been invalid for a very long time.

Edit2: here is something very similar: http://qubanshi.cc/questions/1900501/pyserial-xon-off-fails-on-linux-but-works-on-windows

Question: did you test flow control (software or hardware), or did you rely on the device driver?

[cyc1ingsir]

Hi thanks for reporting this rather complex bug and your thorough investigation.
CuteCom v0.3x and above relies on the serial port module provided by Qt (http://doc.qt.io/qt-5/qtserialport-index.html)
It would be interesting to check out the sample implementation provided with the examples for this module (http://doc.qt.io/qt-5/qtserialport-terminal-example.html). Is this able to handle software flow control?
Unfortunately I have no means of testing flow control (neither soft- nor hardware controlled).
I'd also be interested to know if version 0.22 handled it as expected.

[TG9541]

Hi, thanks for the quick reply!

In the meantime, something in my system has changed - it's a bit like chasing a phantom, but there seems to be something that can be addressed.

I installed Qt Creator, and checked the behavior of the Terminal Example:

• with "Flow control: None" I see non-ASCII characters where the XOFF and XON are to be expected
• with "Flow control: XON/XOFF" no traces of XOFF or XON are visible
• when I type on the keyboard during a transmission from my embedded device, characters are being sent to the embedded device although XOFF is active.

With stty I get the following termios status for the Terminal Example:

$ stty -a -F /dev/ttyUSB0 
speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>;
swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; flush = ^O;
min = 0; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl ixon ixoff -iuclc -ixany
-imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke

This was much more what I expected, except for the undesired result of the typing during transmission test. The termios flags ixon and ixoff are consistent with "Flow control: XON/XOFF". Prior tests (before rebooting my PC) showed them consistently as -ixon and -ixoff (that is inactive).

Now I repeated the test with CuteCom 0.22 and 0.3x with the same result! I also tested minicom and miniterm.py - again the same!

However, in CuteCom (0.22 and 0.3x), miniterm.py, and in minicom, sending files still failed (which is consistent with the finding that all programs still send keys when XOFF is active.

The conclusion is that on my machine termios (or a another device layer) doesn't honor XON/XOFF in "cooked mode" in the expected way (it just hides the characters). I'd recommend that you put the issue on hold for now until the test result has been confirmed on a different machine. In that case processing XON/XOFF directly, instead with termios might be a way to proceed.

Edit:
I reckon that the problem is that the termios isn't in "cooked mode" (-icanon in the stty output above). The problem is indeed more intricate: there is no bug in anything, serial terminal programs that rely on tty just make invalid assumptions. Windows has a completely different approach to character-IO, and the abstraction in libraries like Qt or PySerial isn't sufficient for something like synchronous communication to work (except in the form of time-compensation, like new line delay).

Currently I'm looking in the direction of synchronous (or blocking) communication. Qt provides blocking master and blocking slave examples. The UNIX tty concept with "master-termios-slave" is explained here.

[TG9541]

After doing some experiments with picoterm (which, by the way, is great!) I think I've found the root cause of the problem:

• Qt Serial Port relies on termios which might use cooked mode to for dealing with blocking XON/XOFF
• termios relies on the line discipline, implemented by n_tty.c for handling XON/XOFF through tty_io.c
• tty_io.c uses the "driver method" tty_stop(tty) registered by tty_operations through tty_struct if it is defined at all!
• tty_operations is the central lookup table for the implementation of operations in "tty drivers" like serial_core.c, or usb-serial.c
• serial-core implements XON/XOFF in drivers like 8250.c through weaving control flow into the data flow
• USB-Serial devices generally don't define the operation "stop". In this respect they're in good company (e.g. sdio_uart.c, cdc-acm.c, bluetooth/rfcomm/tty.c ...). IR-DA with ircomm_tty.c is an exception to the rule
• it doesn't matter if you select "Flow control: Hardware" (e.g. RTS/CTS) or "Flow control: Software" (i.e. XON/XOFF) in a termios based serial terminal program if the operations "stop" and "start" are not supported
• I'm using a CH341 which is based on usb_serial_driver. And that's why XON/XOFF will always fail with any serial terminal program that relies on termios low control

The bottom line is this: today a serial terminal program can either rely on on legacy hardware (e.g. 8250 or 16450 UART chips), SoCs with built-in UARTs, or else recognize that low level flow control now falls in the domain of low level communication tools (e.g. CuteCom ;-) ).

@cyc1ingsir I'd be really interested in your thoughts!

[cyc1ingsir]

Issue moved to https://gitlab.com/cutecom/cutecom/issues/22

[sbridger]

A related note. When using xon/xoff with small linux routers, I found the latency was up to as 20 chars (at 57k). This means your embedded device fifo must have 20 bytes free after the trigger point. I changed the rx fifo to 48bytes to be reliable.

[cyc1ingsir]

The project and hence the issue has moved.

[TG9541]

@cyc1ingsir it's really nice that you took the issue to the new platform but it's also a good thing that you left it here for reference.