TX stops after a few bytes - unless port is being read elsewhere

[AndersBNielsen]

I'm using a common FT232 USB -> UART dongle on macOS Sonoma.

I have a script that first sends a 0x01 (SOH) bytes, waits a bit, then sends a binary file. Serial communication works fine with GNU Screen.

Observations:
If I connect to a logic analyzer and send the 0x01 bytes, delay, and do a ser.write(file_data) I observe the 0x01 byte, the delay and then only 4 bytes afterwards. Then TX just stops (line high)

If I send out the bytes one at a time and do a read in between, about 10 bytes are sent +- 2 bytes.

try:
            for byte in file_data:
                data_received = ser.read(ser.in_waiting)
                ser.write(byte.to_bytes(1, 'big'))

The file I'm trying to send is about 100 bytes but only 4-10 are sent unless I use the workaround below.

If I open the port, in a separate Terminal window, with the command cat -v < $serial (where $serial is the port FD) and leave it open while running the python script - it transmits all bytes in the file, without issue.
I'm guessing it might be something with the buffers on the FT232 but I have no idea how to debug this further.
I've also noticed I need inter_byte_timeout=0 when opening - I'm assuming this is because that var forces reconfigure of the open port.

Here's the full code:

import sys
import time
import serial

def send_file(serial_port, baud_rate, file_path):
    try:
        # Open serial port
        ser = serial.Serial(serial_port, baud_rate, timeout=0, inter_byte_timeout=0)
        # Send Start of Header (SOH) byte (hex 0x01)
        ser.write(b'\x01')
        data_received = ser.read(ser.in_waiting)
        # Send binary file
        with open(file_path, 'rb') as file:
            file_data = file.read()

        #time.sleep(0.500)
        time.sleep(0.2)

        try:
            for byte in file_data:
            #    data_received = ser.read(ser.in_waiting)
                ser.write(byte.to_bytes(1, 'big'))

        except serial.SerialException as e:
            print(f"SerialException: {e}")
        except serial.SerialTimeoutException as e:
            print(f"SerialTimeoutException: {e}")
        except serial.SerialWriteTimeoutException as e:
            print(f"SerialWriteTimeoutException: {e}")

        print("File sent successfully.")

    except Exception as e:
        print(f"Error: {e}")

    finally:
        # Close the serial port
        ser.close()

if __name__ == "__main__":
    # Check if correct number of command line arguments is provided
    if len(sys.argv) != 4:
        print("Usage: python script.py <serial_port> <baud_rate> <file_path>")
        sys.exit(1)

    # Extract command line arguments
    serial_port = sys.argv[1]
    baud_rate = int(sys.argv[2])
    file_path = sys.argv[3]

    # Call the function to send the file
    send_file(serial_port, baud_rate, file_path)

[Nielotz]

Check whether OS autosuspends USB.

[AndersBNielsen]

Check whether OS autosuspends USB.

Does not.
The thought did make me consider if the macOS console spits out anything useful, but doesn't seem to be the case.
I might try to downgrade a Mac to see if something went sideways with Sonoma. Already tried FTDI VCP drivers to no avail.
Validating script and dongle on a linux box tomorrow.

[AndersBNielsen]

Mmmkaay. It doesn't seem to be limited to FT232 chips or Sonoma.

I tried the script above with a Raspberry Pi and the same FT232 - worked flawlessly.
Then I downgraded a Mac to Ventura - same issue.
Then I tried an Arduino Pro Micro with USB passthrough firmware - it got more bytes through (like 60 instead of 10) but transfer was still cut short(at seemingly random lenghts) unless I had a cat -v < $serialport running in another window.

... and then I figured it out.

The connection is closed before the last bytes are actually sent - if they're in the buffer when closing I assume they're flushed and that's what's going on here.
At higher speeds I guess it's not a problem but running at 9600 baud like I do here it seems to be.

Considering this is also a problem upstream - even running echo $longstring > /dev/cu.usbserialx has this problem - I'm not sure this is a pyserial bug...

But inserting a 100ms delay before closing seems to be enough of a workaround in my case - dunno if there's a good way to make sure TX buffer is empty before closing on macOS that can or should be implemented.