telnetlib Telnet.expect fails with timeout=0

[JoelLovinger]

BPO	14556
Nosy	@jackdied, @bitdancer, @ZackerySpytz, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-11-30.10:07:00.749>
created_at = <Date 2012-04-12.01:20:16.981>
labels = ['type-bug', 'library']
title = 'telnetlib Telnet.expect fails with timeout=0'
updated_at = <Date 2020-11-30.10:07:00.451>
user = 'https://bugs.python.org/JoelLovinger'

bugs.python.org fields:

activity = <Date 2020-11-30.10:07:00.451>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-30.10:07:00.749>
closer = 'iritkatriel'
components = ['Library (Lib)']
creation = <Date 2012-04-12.01:20:16.981>
creator = 'Joel.Lovinger'
dependencies = []
files = []
hgrepos = []
issue_num = 14556
keywords = []
message_count = 7.0
messages = ['158096', '158098', '158102', '158126', '158161', '379563', '382135']
nosy_count = 5.0
nosy_names = ['jackdied', 'r.david.murray', 'Joel.Lovinger', 'ZackerySpytz', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14556'
versions = ['Python 2.7']

[JoelLovinger]

In Python 2.4.3 a Telnet.expect with timeout=0 would always make at least one call to Telnet.fill_rawq if a match couldn't be found and the connection was open.

In Python 2.7.1/2.7.3 Telnet.expect with timeout=0 breaks before any call to Telnet.fill_rawq if a match isn't found in already read raw/cooked data. Expected behavior is that on timeout=0 at least one non-blocking attempt should be made to read from the connection when necessary to fulfill a match.

From code inspection (including Python 2.7.3) timeout behavior was modified to provide an overall elapsed timeout instead of passing unmodified timeout to each select resulting in the failure.

Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from telnetlib import Telnet
>>> import time
>>> tn = Telnet("scn.org", 23)
>>> time.sleep(5) # short wait for data to be available
>>> tn.expect(['.{16}'], 0)
(-1, None, '')
>>>

Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from telnetlib import Telnet
>>> import time
>>> tn = Telnet("scn.org", 23)
>>> time.sleep(5) # short wait for data to be available
>>> tn.expect(['.{16}'], 0)
(0, <_sre.SRE_Match object at 0x01752410>, '\r\nSeattle Communit')
>>>

[bitdancer]

Can you point to the changes you think are at issue? That might help us track down why the change was made. This isn't necessarily a bug, but even if it isn't, the behavior should probably be explicitly documented.

[JoelLovinger]

Quick response!

Based on review of Telnet.expect in Python 2.4.3 and Python 2.7.1/2.7.3.

In Python 2.4.3 the timeout is passed unmodified on each loop iteration to the underlying select to get more data for a potential match. Iteration only ends on EOF, select timeout with no rx, or match. A timeout=0 received data, without blocking, until no more data or a match was found.

Unfortunately this implementation can extend the timeout indeterminately if the connection consistently has data for each select but no match is made.

Imagine with timeout=10 and a byte coming in every 5 seconds. Select will always succeed and continue to iterate.

Could even happen with timeout=0 if match processing takes long enough for each iteration (high system load, long input buffer, complicated reg ex, etc).

Python 2.7.1 keeps track of overall elapsed time in Telnet.expect itself and explicitly drops out when it exceeds the timeout. Further passes (timeout-elapsed) as the timeout to select.

Works to ensure an expect never exceeds its timeout, but breaks the case where timeout=0 can read non-blocking to find a match.

[bitdancer]

Ah, so there are actually two timeouts of interest. One is "time out if there is no more data for X seconds", and the other is "time out if there is no match for X seconds". It used to do the former, now it does the latter.

I think you get the former by calling socket.settimeout() and then using a blocking call for the expect.

[JoelLovinger]

2.4 behavior, "time out if there is no more data for X seconds", only worked as expected in the case of timeout=0. Any other timeout could result in indefinite extension and needed fixing.

2.7 behavior, "time out if there is no match for X seconds" fixes timeout!=0 cases while breaking for timeout=0.

Can't get former behavior on timeout=0 in 2.7 using socket.settimeout(). Call to socket.recv() then throws a timeout exception which isn't caught by Telnet.expect.

I think fixing timeout=0 will require a patch to Telnet.expect. The simplest would likely be to insert a "very eager" type read to grab all data without blocking before entering the match loop. Won't need to modify existing timeout logic and sidesteps any new corner cases on select/recv iteration.

The only functional side effect (other than moving storage in some cases from the socket to the Telnet object) is on greedy RE that will have more data to match. Documentation already warns against greedy RE as non-deterministic so hopefully not an issue.

[ZackerySpytz]

Python 2.7 is no longer supported, so I think this issue should be closed.

[iritkatriel]

This has been fixed by now:

Running Release|x64 interpreter...
Python 3.10.0a2+ (heads/bpo-42482:920f808f50, Nov 29 2020, 23:02:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from telnetlib import Telnet
>>> import time
>>> tn = Telnet("scn.org", 23)
>>> time.sleep(5) # short wait for data to be available
>>> tn.expect(['.{16}'], 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\User\src\cpython\lib\telnetlib.py", line 622, in expect
    m = list[i].search(self.cookedq)
TypeError: cannot use a string pattern on a bytes-like object
>>> tn.expect([b'.{16}'], 0)
(0, <re.Match object; span=(2, 18), match=b'Seattle Communit'>, b'\r\nSeattle Communit')
>>>