test_xmlrpc is still flakey

[gvanrossum]

BPO	1503
Nosy	@birkenfeld

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-08-21.23:06:44.013>
created_at = <Date 2007-11-27.01:23:19.852>
labels = []
title = 'test_xmlrpc is still flakey'
updated_at = <Date 2010-10-13.00:09:22.219>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2010-10-13.00:09:22.219>
actor = 'schmir'
assignee = 'none'
closed = True
closed_date = <Date 2010-08-21.23:06:44.013>
closer = 'georg.brandl'
components = []
creation = <Date 2007-11-27.01:23:19.852>
creator = 'gvanrossum'
dependencies = []
files = []
hgrepos = []
issue_num = 1503
keywords = []
message_count = 14.0
messages = ['57864', '64566', '64567', '64568', '64569', '64570', '64571', '64616', '69720', '69724', '70750', '70759', '114599', '118490']
nosy_count = 4.0
nosy_names = ['nnorwitz', 'georg.brandl', 'alanmcintyre', 'schmir']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1503'
versions = ['Python 2.6', 'Python 3.0']

[gvanrossum]

See e.g.:
http://www.python.org/dev/buildbot/3.0/ppc%20Debian%20unstable%203.0/builds/303/step-test/0

Note how it fails the first time and passes on the re-run. I've seen
this before (just not on any of my own systems). I've also seen it fail
twice (on buildbot machines).

[schmir]

The current buildbot has errors similar to this one (I assume):

Exception happened during processing of request from ('127.0.0.1', 53126)
Traceback (most recent call last):
  File "/Users/ralf/trunk/Lib/SocketServer.py", line 281, in
_handle_request_noblock
    self.process_request(request, client_address)
  File "/Users/ralf/trunk/Lib/SocketServer.py", line 307, in process_request
    self.finish_request(request, client_address)
  File "/Users/ralf/trunk/Lib/SocketServer.py", line 320, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/Users/ralf/trunk/Lib/SocketServer.py", line 615, in __init__
    self.handle()
  File "/Users/ralf/trunk/Lib/BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "/Users/ralf/trunk/Lib/BaseHTTPServer.py", line 301, in
handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "/Users/ralf/trunk/Lib/socket.py", line 369, in readline
    data = self._sock.recv(self._rbufsize)
error: [Errno 35] Resource temporarily unavailable

---

The problem is that the test calls

serv.socket.settimeout(3)

in the http_server function. This implicitly sets the server socket to
nonblocking state.
The accept call then returns a socket object, which

• is blocking on OS X 10.4 ppc
• is nonblocking on linux

I can easily reproduce that on my mac mini g4 with python 2.6.

[schmir]

I just double checked with the following program:
#! /usr/bin/env python

import os
import fcntl
import socket


def isnonblocking(fd):
    
    return bool(fcntl.fcntl(fd, fcntl.F_GETFL, 0) & os.O_NONBLOCK)

def main():
    s=socket.socket()
    
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind(("0.0.0.0", 8000))
    s.listen(5)

    s.settimeout(30)
    
    
    print "server isnonblocking:", isnonblocking(s.fileno())
    client, addr = s.accept()
    print "client isnonblocking:", isnonblocking(client.fileno())
    
    
        
if __name__=="__main__":
    main()

on my g4 mac it prints:
~/ python serv.py
ralf@mini ok
server isnonblocking: True
client isnonblocking: True

on linux:
~/ python mini/serv.py
ralf@rat64 ok
server isnonblocking: True
client isnonblocking: False

[schmir]

now that I see that the buildbot was running on ppc *Debian* I'm not
quite sure if we're talking about the same issue.

[alanmcintyre]

It's my fault the xmlrpc tests try to use non-blocking sockets. That
got added because sometimes failing tests would just sit there with the
server blocking until the entire test process got killed for running too
long.

There are some tests that are skipped in test_xmlrpc because of
(apparent) Windows socket quirks; should they also be skipped for OS X
PPC, or should the flaky tests just be scrapped?

When I was last working on this I couldn't come up with a better way to
run these tests, so unless somebody can suggest a new approach I'm just
left with recommending the skip & scrap options as the only way to stop
the flakiness.

[schmir]

No, please do not disable them. I'm not quite sure what to do, but
apparently these sockets returned from accept should be turned into
blocking sockets.
I just do not know, where this should happen. I think that this could
even be done inside the accept call (which then might break some code).
At least it should be done in the BaseHTTPServer code, which apparently
cannot handle nonblocking sockets.

[schmir]

With the following diff, test_xmlrpc.py passes without problems.

Like I said, someone else should decide where to turn on blocking mode.
I now think that it should be in the socket.accept (i.e. in the C code)
at least for unix platforms.
Those who really want a nonblocking socket, will most probably call that
setblocking(0) anyway (or their program is broken on linux, which
returns blocking sockets by default).

--- a/Lib/SocketServer.py	Wed Mar 26 22:41:36 2008 +0100
+++ b/Lib/SocketServer.py	Wed Mar 26 23:48:13 2008 +0100
@@ -441,8 +441,10 @@
         May be overridden.
 
         """
-        return self.socket.accept()
-
+        r= self.socket.accept()
+        r[0].setblocking(1)
+        return r
+    
     def close_request(self, request):
         """Called to clean up an individual request."""
         request.close()

[nnorwitz]

Ugh. The manpage for accept on Ubuntu 6.10 says:

"""
On Linux, the new socket returned by accept() does not inherit file
status flags such as O_NONBLOCK and O_ASYNC from the listening socket.
This behaviour differs from the canonical BSD sockets implementation.
Portable programs should not rely on inheritance or non-inheritance of
file status flags and always explicitly set all required flags on the
socket returned from accept().
"""

http://msdn2.microsoft.com/en-us/library/aa450277.aspx says that Windows
(CE, but I assume all variants) are like BSD in that they inherit
attributes.

"""The newly created socket is the socket that will handle the actual
connection and has the same properties as socket s, including the
asynchronous events registered with the WSAEventSelect function."""

I assume that means blocking behavior.

I checked in r61993 which should fix the immediate problem with
test_xmlrpc. I wonder if we should change socket to do the same thing
for all platforms.

[gvanrossum]

Does anybody still care about this?

[schmir]

I still think that blocking mode should be turned on in socket.accept.
settimeout implicitly sets non-blocking mode (which might be a bit
surprising).
The sockets returned from accept() being in non-blocking mode is
surprising (as this bug shows).

[gvanrossum]

Without a patch this will not improve.

[schmir]

I would say without a decision on what to do, there will be no patch :)

I can write a patch, which unconditionally turns on blocking mode for
sockets returned from accept (on unix-platforms). Would that be fine?

[birkenfeld]

I've not seen special failures of test_xmlrpc lately.

[schmir]

The tests now pass, but the underlying issue hasn't been fixed!