smtpd.SMTPServer can cause asyncore.loop to enter infinite event loop

[cmcginty]

BPO	6589
Nosy	@giampaolo, @bitdancer
Files	smtp_test.py: Script to reproduce reported issue smtpd.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/giampaolo'
closed_at = <Date 2010-06-30.17:53:45.564>
created_at = <Date 2009-07-28.03:59:57.140>
labels = ['type-bug', 'library']
title = 'smtpd.SMTPServer can cause asyncore.loop to enter infinite event loop'
updated_at = <Date 2010-06-30.17:53:45.563>
user = 'https://bugs.python.org/cmcginty'

bugs.python.org fields:

activity = <Date 2010-06-30.17:53:45.563>
actor = 'giampaolo.rodola'
assignee = 'giampaolo.rodola'
closed = True
closed_date = <Date 2010-06-30.17:53:45.564>
closer = 'giampaolo.rodola'
components = ['Library (Lib)']
creation = <Date 2009-07-28.03:59:57.140>
creator = 'cmcginty'
dependencies = []
files = ['17807', '17819']
hgrepos = []
issue_num = 6589
keywords = ['patch']
message_count = 8.0
messages = ['91002', '106073', '108867', '108888', '108955', '108974', '108998', '109000']
nosy_count = 5.0
nosy_names = ['ggenellina', 'giampaolo.rodola', 'josiah.carlson', 'r.david.murray', 'cmcginty']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6589'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

[cmcginty]

When subclass of smtpd.SMTPServer, it is possible the get asyncore.loop
to enter an infinite loop where the following output is shown:

.....
warning: unhandled write event
warning: unhandled read event
warning: unhandled write event
warning: unhandled read event
warning: unhandled write event
warning: unhandled read event
warning: unhandled write event
warning: unhandled read event
warning: unhandled write event
warning: unhandled read event
warning: unhandled write event
warning: unhandled read event
....

To reproduce:

1. Init SMTPServer class instance inside of Thread class and call
   asyncore.loop()
2. Init second SMTPServer class instance from a second Thread class,
   binding to the same address and port. The SMTPServer instance will raise
   an exception that socket cannot bind to the port in use. The socket
   exception must be caught at some level, and the program execution
   allowed to continue.
3. First SMTPServer thread will enter infinite event loop.

Analysis:
When the exception is raised in the second SMTPServer instance, the new
instance has already registered itself with the asyncore library (ex:
'asyncore.dispatcher.__init__(self)'). There is no code in the
SMTPServere.__init__ method that catches the exception raised, caused by
the failed bind attempt. After the exception is caught, the first thread
continues to run, but asyncore is in an invalid state because it still
has the registration of the second SMTPServer instance that never completed.

Workaround and Proposed Fix:
A viable workaround seems to be catching the raised exception in
__init__ method of the SMTPServer subclass, and call self.close() before
re-raising the exception:

class MySmtpServer( smtpd.SMTPServer, object ):
   def __init__( self, **kwargs ):
      try:
         super( _SmtpSink, self).__init__(**kwargs)
      except Exception as e:
         self.close()   # cleanup asyncore after failure
         raise

For a long term fix, I would recommend performing the
asyncore.dispatcher.close() method call in the SMTPServer.__init__() method.

[giampaolo]

Could you provide an actual example code which reproduces this problem?
It's not clear to me how the dispatcher instance can end up in an "invalid state" since handle_error() should automatically remove the invalid dispatcher instance from the socket_map if anything unexpected happens.
If this doesn't happen it might be a problem related with what you've done in your subclass (e.g. you have overridden handle_error and avoided to call close() yourself).

Furthermore the use case you have described it's pretty uncommon as you shouldn't run SMTPServer in a thread in the first place.

If you intend to bind two servers simultaneously you just have to instantiate two STMPServer sub/classes and finally call asyncore.loop().
Both instances will automatically be served by asyncore itself.

[cmcginty]

This is how it gets in an "invalid state". Not sure why you can't look at the code and see the mistake.

"There is no code in the
SMTPServere.__init__ method that catches the exception raised, caused by
the failed bind attempt. After the exception is caught, the first thread
continues to run, but asyncore is in an invalid state because it still
has the registration of the second SMTPServer instance that never completed."

As far as running in a thread. I have a program that needs must start and stop the SMTPServer dynamically. I did this by putting SMTPServer in a thread. Maybe there is another way to do it, but if you are not going to support this, then it should be documented.

[bitdancer]

If you can provide a short example that reproduces the problem it will be much more likely to get fixed.

[cmcginty]

I attached a simple script that reproduces the report issue. I hope it helps.

[giampaolo]

Although the use case is pretty uncommon and somewhat twisted (take a look at Lib/test/test_ftplib.py for a nicer approach on wrapping asyncore.loop() in a thread) it is true that if SMTPServer class raise an exception at instantiation time, some garbage remains in asyncore.

To replicate this problem there's no need to involve threads:

>>> import asyncore, smtpd
>>> s = smtpd.SMTPServer(('127.0.0.1', "xxx"),None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.5/smtpd.py", line 280, in __init__
    self.bind(localaddr)
  File "/usr/local/lib/python2.5/asyncore.py", line 303, in bind
    return self.socket.bind(addr)
  File "<string>", line 1, in bind
TypeError: an integer is required
>>> asyncore.socket_map
{3: <smtpd.SMTPServer ('127.0.0.1', 'xxx') at 0xb783528c>}
>>> 

I think it's ok for SMTPServer.__init__ to cleanup asyncore and finally raise the exception, as you suggested in the first place.
I'll provide a patch later today.

[giampaolo]

Patch in attachment.

[giampaolo]

Fixed in r82404, r82406, r82407 and r82408.