open() returns before port is ready

[jimbo1969]

When opening a pyserial port and immediately reading from it or flushing it or whatever, the immediate action fails silently. To reproduce (note mine is Windows 10, python 3.5, but I've seen reports of the same behavior on Linux):

import unittest
import serial
import time

"""
1) create a virtual or real connection between COM12 and COM13
2) in a terminal connected to COM12 (at 9600, N81), enter some junk text (e.g.'sdgfdsgasdg')
3) then execute this unit test
"""

class Test_test1(unittest.TestCase):
    def test_A(self):
        with serial.Serial(port='COM13', baudrate=9600) as s:   # open serial port
            print("Read ASAP:  {}".format(s.read(s.in_waiting)))
            time.sleep(0.1)     # wait for 100 ms for pyserial port to actually be ready
            print( "Read after delay:  {}".format(s.read(s.in_waiting)) )

if __name__ == '__main__':
    unittest.main()

Output will be:

Read ASAP:  b''
Read after delay:  b'sdgfdsgasdg'
[...]
Ran 1 test in 0.101s

Desired Output:

Read ASAP:  b'sdgfdsgasdg'
Read after delay:  b''.
[...]
Ran 1 test in 0.101s

[guerriky]

Consider this (over simplified) code:

import serial

def f():
    s=serial.Serial("/dev/ttyUSB1")
    s.write(b"aaa")

Running this in a terminal while having another terminal open in read produces the expected result, the write operation is performed immediately.

So, the open() method actually returns when the port is ready.

Now, I'm no expert, but I think the issue that you are having has nothing to do with how the pyserial library reports port readiness, but rather with how serial buffers are managed along the transmission line.

A (rudimental) solution to your situation might be to just wait for the first byte and then attempt to read, like so:

import unittest
import serial
import time

"""
1) create a virtual or real connection between COM12 and COM13
2) in a terminal connected to COM12 (at 9600, N81), enter some junk text (e.g.'sdgfdsgasdg')
3) then execute this unit test
"""

class Test_test1(unittest.TestCase):
    def test_A(self):
        with serial.Serial(port='COM13', baudrate=9600) as s:   # open serial port
            while not s.in_waiting:
                time.sleep(0.01)
            print("Read ASAP:  {}".format(s.read(s.in_waiting)))
            time.sleep(0.1)     # wait for 100 ms for pyserial port to actually be ready
            print( "Read after delay:  {}".format(s.read(s.in_waiting)) )

if __name__ == '__main__':
    unittest.main()

[jimbo1969]

Thanks for the response Riccardo. Your example (the immediate write of b'aaa') works because write() blocks until the port is ready. Try it with some non-blocking function, such as flushInput() or (non-blocking read()) when you know there is something in the input buffer. It will fail to empty (or read) it unless you wait for a while after opening.

I'd be curious to know if the behavior is the same in *nix as it is in Windows.

[guerriky]

Hello @jimbo1969,
I had a few go's with the pyserial library and I honestly don't know how to interpret my results .-.

Have a look at the following code:

import serial,time
def f():
	s0=serial.Serial("/dev/ttyUSB0")
	time.sleep(0.1)
	s1=serial.Serial("/dev/ttyUSB1")
	s1.write(b"UUUUUUUUUUUUUUUUUUUU")
	print(s0.in_waiting)
	r=s0.read(s0.in_waiting)
	print(r)
f()

This produces the following outcome:

0
b''

However, if you move the sleep statement, like this:

import serial,time
def f():
	s0=serial.Serial("/dev/ttyUSB0")
	s1=serial.Serial("/dev/ttyUSB1")
	s1.write(b"UUUUUUUUUUUUUUUUUUUU")
	time.sleep(0.1)
	print(s0.in_waiting)
	r=s0.read(s0.in_waiting)
	print(r)
f()

It outputs:

20
b'UUUUUUUUUUUUUUUUUUUU'

Of course, I'm going at default baud (9600?) on a Linux PC with two real serial ports cross-connected, so it's probably attempting to read too soon.

It doesn't look like your specific case though.

I don't have a windows machine to try it myself; just out of curiosity, would you mind running something like that on your machine?

Thank you,
Riccardo

[jimbo1969]

Riccardo,
You already answered your own question (in your first post). You have to allow time for the message to be transmitted & buffered and made available to be read by pyserial. It is unrelated to the issue at hand, which is data that has been available in the buffer for tens of seconds or minutes not being read because open() returns before the port can be read from. You should start a separate thread if your unrelated sample it is not the expected behavior.

Thanks,
--Jim

[guerriky]

Hi Jim,
yeah, sorry, I digressed.

For now, I can only confirm that this also happens to me under Linux, regardless of whether I use the in_waiting value or a fixed value (provided I set the timeout to zero).

I'll update when I know more.

Bye,
Riccardo

[jimbo1969]

Thanks Riccardo, but what you did doesn't actually confirm that the issue at hand happens in Linux. Your sample writes to one serial port, and then immediately tries to read that data from another, connected, serial port, without any delay between the write and the read. I'm not sure that is unexpected behavior.

The issue discussed here involves writing to a serial port that is connected to another serial port, like in your case, but then going and having a cup of coffee, and then coming back and reading from the connected port immediately after opening it, but long after we know it has had ample opportunity to receive & buffer what we sent to it (much) earlier.

To confirm, I opened a TeraTerm serial port on COM12 (with no flow control), which is connected to COM13. Using TeraTerm, I send a handful of random characters, which I record for posterity. Then I close TeraTerm, to ensure that it isn't simply buggy and waiting for some RTS signal or something from the other end. Then I answer an email to kill some time. Then I run my Python test script that I posted in the first post here, on COM13, which is attached to COM12. The result:
'''
Read ASAP: b''
Read after delay: b'sd'
[...]
Ran 1 test in 0.102s
'''
It only gets the first two bytes of what I had sent to it earlier, but only upon the second read() attempt. Such a scenario is actually what put me onto this problem. I was attempting to read the response from a particular command, but it was often prepended with junk left-over from a previous session that didn't get cleared out. So I started with a flushInput() immediately after opening, which silently fails to actually flush the input buffer. In the process of creating a simple test-case, I discovered that the read() also does not work immediately after opening, and read() seemed like a test-case with a better signal-to-noise ratio (and/or wider understanding/adoption) for discussion than flushInput() -- thus making it a better candidate for illustrating the issue.

I suspect there is double-buffering occurring. Is, perhaps, the OS buffering serial input on the port, followed by pyserial opening it at that level and returning control to Python, while the OS then asynchronously starts dumping the contents of its own buffer into pyserial's input buffer? If so, can anyone explain, for curiosity's sake why we get just the oldest two bytes from that buffer when the other end has long disconnected, but a larger number of bytes when the other end is still present, even though there is no flow control turned on? Or more importantly, how to avoid getting leftover garbage in the response when I reopen a closed pyserial connection, write to it, and then read the response from the remote device? (my sleep() workaround is a cheesy and ultimately unreliable non-solution, it seems)

Thanks,
--Jim

[0x3333]

I'm having the same issue. If I try to write to the serial port right after open, it will fail silently.
I had to add time.sleep(2) before writing to the port.

    ser = serial.Serial(port, 57600, timeout=7)
    time.sleep(2)
    ser.write("\nR{}\n".format(pulses).encode("ascii", "ignore"))

[petercorke]

Also having this issue, with MacOS Mojava and a RaspPi. A delay of around 2 seconds is needed in order for comms (write fixed length message, get fixed length reply) to happen with an embedded device.
As reported by others the port is opened in non-blocking mode which is an advised good practice. However it's hard to find definitive information about what a non-blocking open means, is the file handle usable immediately? select/poll don't offer any options to wait for an open to complete.
I'm reluctant to hard code a fixed delay, my current preferred solution is to keep throwing packets at the embedded device until it replies, but even that feels very hacky. What's going on?

I should add that my serial device is an FT232R USB UART, is this a common factor with other experiencing this delay?

[axel-kah]

Same issue here. Fixed delay is the only workaround.

My setup:

• Windows 10
• pyserial 3.4 from pypi
• python 3.6.5
• serial device is a FTDI FT232RL

Tried software flow control, hardware flow control, setting read/write-timeout. In addition I put isOpen() right after serial open and it always returns True, despite silently failing to write (with timeout of 1 second). The read after the write runs into the timeout and does not return the expected repsonse but an empty string.

As a workaround I need to wait for 2 seconds (1 is not enough) and then I can query with the device with read delays of about 30 ms no problem.

If you need to have something tested on my setup, just let me know!

[carlcrott]

Chiming in on this as well.

This code works:

import serial
import time
arduino_pump = serial.Serial('/dev/cu.wchusbserial14510', 115200, timeout=1)
time.sleep(1.5)
arduino_pump.write(b'A')
time.sleep(0.5)
confirmation = arduino_pump.readline().strip()
print(confirmation)

and appropriately returns:

b'65'

Where this code does not work:

import serial
import time
arduino_pump = serial.Serial('/dev/cu.wchusbserial14510', 115200, timeout=1)
# No sleep delay here
arduino_pump.write(b'A')
time.sleep(0.5)
confirmation = arduino_pump.readline().strip()
print(confirmation)

and returns:

b''

Arduino Code for reference:

byte incomingByte;

void setup() {
    Serial.begin(115200);
}

void loop() {
    if (Serial.available() > 0) {
        incomingByte = Serial.read();
        Serial.println(incomingByte);
    }
    delay(1); // stability
}

What seems counter-intuitive is that the Serial object is returned before its ready to be written to.

Is there a way to block the process until its completely instantiated?

[harryberlin]

try arduino_pump.write('A') without b

[petercorke]

Seems to be some common experience. It'd be great to hear from the developer, some activity from a couple of months back, but the releases are now quite old. Did anybody dive into the code and have a look, I haven't yet.

[carlcrott]

@harryberlin the b isn't a typo and instead it controls python sending the data as byte-encoded.

EX: If I leave it out, I get the error message:

TypeError: unicode strings are not supported, please encode to bytes: 'A'

[der-thomas]

I'd like to share some additional insight on this issue:

I'm using Python 3.7 on Windows 10. I found that if I don't wait long enough, the connection also fails for the following transfers.

With PySerial 3.4:

So, I connected an oscilloscope to the TX and RX pins of an Arduino Mega to see what is going on. The following pictures all show the activity on these pins over a period of a few seconds. The yellow curve is the signal transmitted to the Arduino, the blue curve is the signal from the Arduino. The vertical spikes in both lines are the actual data transfers.

When the port is opened, the signal line to the Arduino (yellow curve) is pulled low for a short time. This is the first spike in the graph. Interestingly, there are small voltage dips on the line from the Arduino (blue curve). The firs on appears when the port is opened. Note the second voltage dip approx. 0.9 s after the first one. I found that the delay has to be long enough for this dip to appear. From then on, it works. If there is a transfer before this second dip occurs, the communication won't work at all (in my case).

Working communication with a delay of 1 s:

The graph above is for this code:

import serial
import time

UnknownDevice = serial.Serial()
UnknownDevice.timeout = 1
UnknownDevice.baudrate = 115200
UnknownDevice.bytesize = serial.EIGHTBITS
UnknownDevice.parity = serial.PARITY_NONE
UnknownDevice.stopbits = serial.STOPBITS_ONE
UnknownDevice.xonxoff = False
UnknownDevice.port = COMPort.device
UnknownDevice.open()

time.sleep(1)
    
UnknownDevice.write(b"request")

This doesn't work (delay of 0.5 s):

import serial
import time

UnknownDevice = serial.Serial()
UnknownDevice.timeout = 1
UnknownDevice.baudrate = 115200
UnknownDevice.bytesize = serial.EIGHTBITS
UnknownDevice.parity = serial.PARITY_NONE
UnknownDevice.stopbits = serial.STOPBITS_ONE
UnknownDevice.xonxoff = False
UnknownDevice.port = COMPort.device
UnknownDevice.open()

time.sleep(0.5)
    
UnknownDevice.write(b"request")

This behaviour is not specific to PySerial. The same delay is necessary when using PyVISA 1.9.1.

Working communication

import visa
import time

rm = visa.ResourceManager()
UnknownDevice = rm.open_resource(COMPort)

UnknownDevice.timeout = 1000
UnknownDevice.baud_rate = 115200
UnknownDevice.data_bits = 8
UnknownDevice.parity.none
UnknownDevice.stop_bits.one
UnknownDevice.flow_control = 0

time.sleep(1)

UnknownDeviceAnswer = UnknownDevice.query("ident")

And this doesn't work (delay of 0.5s):

import visa
import time

rm = visa.ResourceManager()
UnknownDevice = rm.open_resource(COMPort)

UnknownDevice.timeout = 1000
UnknownDevice.baud_rate = 115200
UnknownDevice.data_bits = 8
UnknownDevice.parity.none
UnknownDevice.stop_bits.one
UnknownDevice.flow_control = 0

time.sleep(0.5)

UnknownDeviceAnswer = UnknownDevice.query("ident")

I have no idea what is taking approx. 1 s before the communication works. But maybe someone can work it out from here.

[dmweir]

For anybody who comes across this thread and is working with an arduino type microcontroller, the issue is not with pyserial. The arduino will auto reset when the serial port connection is opened. I believe that this is the cause of the delay. If you watch the onboard leds when you connect, you should be able to confirm that is the case. Experimenting a little bit, setting dsrdtr=True when creating the connection seems to stop the auto resetting on an arduino uno (windows 10) and may be a work around.

[jherrerob]

You can stop Arduino from resetting by using a 1uF -10uF capacitor between ground and reset. You will need to remove it in order to upload new code into the Arduino.

[yurenchen000]

Use pure C code to test, also have same problem, I use HL-340 usb-serial.
seems it's usb-serial problem on linux:
https://stackoverflow.com/a/38352262/4896468