Support framing error/break detection

[onitake]

In DMX-512, a break condition is used to signal the start of a data frame. Unfortunately, using POSIX or even W32 APIs, it seems to be quite hard to detect when a break occurs - in contrast with sending a break.

Nevertheless, would it be possible to implement support for detecting breaks, or at least allow configuring serial ports to send special codes when a break is detected?

On POSIX, this could be achieved by clearing IGNPAR and setting PARMRK, while for W32, I found this: https://social.msdn.microsoft.com/Forums/en-US/b503e85f-4efc-4fe5-9cb9-89ec93d64430/detect-a-break-as-the-begining-of-a-serial-data-frame-windows-iot-serial-device?forum=WindowsIoT

First, send IOCTL_SERIAL_SET_CHARS, with SERIAL_CHARS::BreakChar set to the character you want to represent the break signal.

Send IOCTL_SERIAL_SET_HANDFLOW, with the SERIAL_BREAK_CHAR flag set in the SERIAL_HANDFLOW::FlowReplace field.

Another option for POSIX would be to clear IGNBRK and set BRKINT - the Linux man page states:

If IGNBRK is set, a BREAK is ignored. If it is not set but BRKINT is set, then a BREAK causes the input and output queues to be flushed, and if the terminal is the controlling terminal of a foreground process group, it will cause a SIGINT to be sent to this foreground process group.

If the SIGINT can be prevented somehow, flusing the input queue might be exactly what's needed for this special case.

I saw that there was a proposal for something similar (parity error detection) in #154 , but it got closed without being merged.

[EternityForest]

I would vote for parmrk. Clearing the queue would lose data, and would not give the user program any indication the break happened. Plus, adding parmrk would be almost trivial to do.

[smarek]

@onitake not sure if there is already available solution, so I made a simple (linux/posix only) utility, to set port attributes and detect sync in the stream here https://github.com/smarek/dmx-python-client with particular emphasis on how i use the existing file-descriptor and modify the attributes here https://github.com/smarek/dmx-python-client/blob/master/roh/dmx/client/dmx_client.py#L47

Or is there any easier solution to this? Thank you!

[onitake]

@smarek I tried to solve it like this:

def initdmx(port):
    with serial.Serial(port=port, baudrate=250000, bytesize=8, parity='N', stopbits=2, timeout=1.0) as ser:
        fd = ser.fileno()
        attr = termios.tcgetattr(fd)
        # enable breaks: set PARMRK and INPCK, clear IGNPAR and ISTRIP in iflag
        attr[0] = attr[0] | termios.PARMRK | termios.INPCK & ~termios.IGNPAR & ~termios.ISTRIP
        termios.tcsetattr(fd, termios.TCSANOW, attr)

def readdmx(ser):
    mark = ser.read()
    if len(mark) < 1:
        return NO_DATA
    if mark[0] == 255:
        mark = ser.read()
        if len(mark) < 1:
            return NO_DATA
        if mark[0] == 255:
            return FRAME_ERROR
    return mark[0]

It didn't work reliably, however.

[smarek]

@onitake thank you for sharing your code

I developed https://github.com/smarek/dmx-python-client as a client to uDMX USB dongle and RS485 to RS232 USB converter

With master software being QLC+ or Q Light Controller, i found that the transmission is really unstable and thus I cannot rely on single bytestream sync and I have to verify each frame and re-sync each time the frame does not end with expected BREAK sequence (FF 00 00)

Also the rate at which data come in is almost the same as speed i can read the data out from pyserial buffer, so i've never experienced empty buffer, you on the other hand, have the NO_DATA as something that happened to you quite regularly

So my reading dmx data here: https://github.com/smarek/dmx-python-client/blob/master/roh/dmx/client/dmx_client.py#L67
always does check for correct frame signature, and if it's not present (because BREAK was indicated somewhere within the frame as an error of expectations from laggy master (both master software and hardware seemed to be unstable when sending the data, especially at high rates) and get_sync routine could be improved greatly as documented here smarek/dmx-python-client#1

[onitake]

I have tested a bunch of RS485-to-USB converters and observed the same problem: The frame sync goes all over the place practically instantly, and I can't manage to synchronize to the frame start.

As can be seen in initdmx(), the idea was to set two stop bits and a 1-second timeout, then react to frame errors, which should occur whenever a new frame starts. When there is "silence" on wire, the timeout should ensure that the read() waits at least one frame period (1s), then fires and I get NO_DATA, which resets the sync counter. But this doesn't seem to work.

Here's the input loop:

def loop(ser, present, refresh, report):
    # frame type (=start code)
    frametype = 0
    # frame
    frame = bytearray(512)
    # frame index counter
    index = 0

    # break only on ctrl+c
    while True:
        value = readdmx(ser)
        received = False
        if value < 0:
            # break or read error, reset frame counter
            index = 0
            report(value)
        else:
            # not out of bounds?
            if index < 512:
                # nope, process
                if index == 0:
                    # set the frame type
                    frametype = value
                else:
                    # store next byte (we store the start code separately)
                    frame[index-1] = value
                # present
                received = True
        # did we receive a valid value?
        if received:
            # yes, was it something else than the start code?
            if index > 0:
                # are we in a default frame? (i.e. start code = 0)
                if frametype == 0:
                    # data value, present it
                    present(index, value)
            # and increment the counter
            index += 1
            # and present (probably not the best idea to do here though)
            refresh()

[smarek]

Well with more further testing, obviously, having any DMX address set to value 0xFF will pass it encoded to application as 0xFF0000 making decoding (and reading) each frame difficult, not modifying the DMX values and clearing the buffer seems like better option to me now, that i've spent several hours with it, but as @onitake said in first post, maybe only for this special case

[smarek]

So i got it figured out, for anybody interested, tag v0.3 works so far with all usb/gpio rs485 to serial (baud=250000), i got to test with
https://github.com/smarek/dmx-python-client

Trick is

1. Turn on ISTRIP, this will make the only 0xFF present in data, the BREAK (per PARMRK)
2. Find 0xFF 0x00 0x00 sequence, which marks break (follows directly after data, preceeds the zero-dmx-address byte and 512 values)
3. Turn off ISTRIP, once you find the break/sync
4. read data (with PARMRK turning each valid data byte 0xFF into 0xFF 0xFF, so you have to turn those 2-bytes into 1-byte value as you fill the DMX512 frame buffer), until you read out 516 bytes (1 zero byte, 512 data bytes, 3 BREAK bytes)
5. repeat step 4 for more valid DMX512 frames

Also while testing buffers add up and over time data are processed with more delay (in my trivial/naive scripts), so currently there is buffer reset every 50 valid DMX512 frames received, see https://github.com/smarek/dmx-python-client/blob/master/roh/dmx/client/dmx_client.py#L117

Usual time to obtain sync, on rpi zero w, is 0.7 seconds, and valid frame update (DmxClientCallback) comes 3-5 times per second, which was good enough for me

I'm not sure pyserial can solve this without having dmx-specific protocol plugin/handler