FTP modue functions are not re-entrant,give odd exceptions

[brucepeterson]

BPO	1469557
Nosy	@ronaldoussoren

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2006-07-28.17:02:41.000>
created_at = <Date 2006-04-13.00:13:56.000>
labels = ['invalid', 'library']
title = 'FTP modue functions are not re-entrant,give odd exceptions'
updated_at = <Date 2006-07-28.17:02:41.000>
user = 'https://bugs.python.org/brucepeterson'

bugs.python.org fields:

activity = <Date 2006-07-28.17:02:41.000>
actor = 'brucepeterson'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2006-04-13.00:13:56.000>
creator = 'brucepeterson'
dependencies = []
files = []
hgrepos = []
issue_num = 1469557
keywords = []
message_count = 5.0
messages = ['28212', '28213', '28214', '28215', '28216']
nosy_count = 2.0
nosy_names = ['ronaldoussoren', 'brucepeterson']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1469557'
versions = ['Python 2.4']

[brucepeterson]

If I define a class using the Thread and FTP moudles,
start a process which gathers FTP responses,
additional calls to the class may have the responses
of the thread instead of the main loop (or vice versa)
This causes weird and unexpected exceptions from the
ftplib.

For instance I get the following error when the
thread process does a pwd() function
error_reply: 213 34603008

The "213" reply is a response from the main process
size() function

---------
Code
---------

from time import sleep
from threading import Thread

class ftpMachine(Thread, ftplib.FTP):
    def __init__(self, svr, user, passwd):
        Thread.__init__(self)
        ftplib.FTP.__init__(self, svr, user, passwd)
        
    def run(self):
        for x in xrange(20):
            output="Thread -"+str(self.nlst())[:30]
            print output
            sleep (0.0100)

def main():
    aCon = ftpMachine("LocalFTP", "user", "")
    aCon.start()
    for x in xrange(20):
        output = "Main -- " + str(aCon.size("File"))
        print output  
        sleep (0.010)

Workround:
Rewrite code to create a third worker thread for
response isolation? Don't know that this would solve
the problem.

---------------
Exception example
---------------

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python24\lib\threading.py", line 442, in 
__bootstrap
    self.run()
  File "dualFTPIssue.py", line 17, in run
    output  = "Thread output " + str(self.nlst())[:30]
+" ..."
  File "C:\Python24\lib\ftplib.py", line 448, in nlst
    self.retrlines(cmd, files.append)
  File "C:\Python24\lib\ftplib.py", line 396, in 
retrlines
    conn = self.transfercmd(cmd)
  File "C:\Python24\lib\ftplib.py", line 345, in 
transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "C:\Python24\lib\ftplib.py", line 321, in 
ntransfercmd
    host, port = self.makepasv()
  File "C:\Python24\lib\ftplib.py", line 299, in 
makepasv
    host, port = parse227(self.sendcmd('PASV'))
  File "C:\Python24\lib\ftplib.py", line 566, in 
parse227
    raise error_reply, resp
error_reply: 213 34603008

[ronaldoussoren]

Logged In: YES
user_id=580910

IMHO this isn't a bug, your accessing a shared resource (the FTP connection)
from two threads without locking. Most of python's libraries aren't safe for this
kind of use.

BTW. The fact that you subclass form thread and ftplib.FTP is immaterial, you
will get the same effect if you create an ftplib.FTP and then use it in two threads.

[brucepeterson]

Logged In: YES
user_id=1500983

Unfortunately I can't start two instances of the FTP client
as the separate instances use two conneciton resources. I
was hoping for a non-exception when a unexpsected response
happens. That would require re-working the module.
For my workaround I set a time to determine when the
transfer should complete instead of a querry.

Thanks for your attention in this matter.

Closing this bug as my case is an unusual one.

[ronaldoussoren]

Logged In: YES
user_id=580910

I don't understand why you cannot use two connection resources. Are you
running in a severely resource constrained environment?

Anyway, what you're doing right now is undefined behaviour. Unless explicitly
stated otherwise classes in the stdlib aren't fully reentrant. You will therefore
have to arrange some kind of exclusion mechanism. One way of doing that is
to introduce a lock (probably wrapping the FTP client instead of using
multiple inheritance in the progress). Another way it to introduce a 3th thread
that performs the actual FTP operations and communicate with that using a
queue. Please ask around on python-list/comp.lang.python if you need help
with this.

BTW. I didn't close this bug, although I do understand why it was closed: the
behaviour you describe isn't a bug but expected behaviour.

[brucepeterson]

Logged In: YES
user_id=1500983

I am running on a unit that limits the FTP connections to 4
high bandwidth connections and normally uses a passive FTP
transfer.
The response from the passive FTP transfer could come at
any time. Also, by inspection, the FTP module throws an
exception when an unexpected reply happens. So when I'm
querrying the size of the clip on the destination and
instead get a transfer complete message from the passive
FTP session, that transfer complete message causes the
exception.
To "fix" this issue would requre that when the FTP
message queue is read, that pending requests (passive
transfers) would be considered before returning immediate
command responses (cwd, ls, size, etc.). If I get a couple
of days, I may rewrite the functions that read the FTP
responses and submit the changes for consideration.

For my current project, I have decided not to querry the
transfer to see if it completes, but instead to do a simple
wait before sending the next transfer.