Can't import xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer when zlib is not available

[ezio-melotti]

BPO	6499
Nosy	@pitrou, @kristjanvalur, @ezio-melotti, @bitdancer
Files	test_docxmlrpc.py: modified file with better error handling

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/kristjanvalur'
closed_at = <Date 2009-12-13.17:24:26.648>
created_at = <Date 2009-07-17.02:20:04.905>
labels = ['type-bug']
title = "Can't import xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer when zlib is not available"
updated_at = <Date 2009-12-13.17:24:26.609>
user = 'https://github.com/ezio-melotti'

bugs.python.org fields:

activity = <Date 2009-12-13.17:24:26.609>
actor = 'r.david.murray'
assignee = 'kristjan.jonsson'
closed = True
closed_date = <Date 2009-12-13.17:24:26.648>
closer = 'r.david.murray'
components = []
creation = <Date 2009-07-17.02:20:04.905>
creator = 'ezio.melotti'
dependencies = []
files = ['14573']
hgrepos = []
issue_num = 6499
keywords = []
message_count = 20.0
messages = ['90599', '90668', '90678', '90709', '90714', '90715', '90751', '90782', '90784', '90788', '90808', '90834', '90835', '90958', '90960', '91916', '92009', '92560', '92567', '96338']
nosy_count = 4.0
nosy_names = ['pitrou', 'kristjan.jonsson', 'ezio.melotti', 'r.david.murray']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6499'
versions = ['Python 2.7', 'Python 3.2']

[ezio-melotti]

I'm working on bpo-6026 and I noticed that the patch for bpo-6267 introduced
an "import gzip" in Lib/xmlrpclib.py in r73638.
gzip tries to import zlib, and if it's not available the import fails.
This led to 3 new tests failures in the trunk: test_xmlrpc,
test_docxmlrpc and test_multiprocessing.
This also mean that the modules xmlrpclib, DocXMLRPCServer and
SimpleXMLRPCServer (and possibly others) cannot be imported anymore if
zlib is not available (they used to work on 2.6).

xmlrpclib should check if the "import gzip" fails and disable the new
gzip-related features (raising an error only when someone tries to use
them).

I don't know if this check can be moved directly on gzip but it seems
unlikely.

[kristjanvalur]

I thought zlib was a builtin module? Why isn't it available?

[ezio-melotti]

I don't know why it's not installed here, but I didn't remove it
intentionally.
"here" is a Linux machine with Ubuntu 8.04.3 (hardy) LTS.
One reason might be that I just have a limited number of program
installed and zlib is probably a dependency of some popular packages and
hence it's found in most of the machines.

When I compiled Python I got this message, so I think that Python
expects zlib to be already there:
Python build finished, but the necessary bits to build these modules
were not found: _dbm _gdbm _hashlib _sqlite3 _ssl _tkinter bz2 zlib

[pitrou]

Kristján, zlib is only built when the required development headers (.h
files for the zlib library) are available.

[kristjanvalur]

submitted revision 74098 and revision 74099 which should fix your issue.
Oh, and revision 74100 and revision 74101 as well (insufficient testing
on my part, tsk. tsk.)

[ezio-melotti]

Thanks, now all the tests pass.
However I had intermittent failures (with a really useful error message)
on test_docxmlrpc:

$ ./python Lib/test/regrtest.py -v test_docxmlrpc
test_docxmlrpc
test_autolink_dotted_methods (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
Test that selfdot values are made strong automatically in the ... ok
test_autolinking (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
Test that the server correctly automatically wraps references to PEPS ... ok
test_invalid_get_response (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
... ok
test_lambda (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
Test that lambda functionality stays the same.  The output produced ... ok
test_system_methods (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
Test the precense of three consecutive system.* methods. ... ok
test_valid_get_response (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) ... ok

---

Ran 6 tests in 0.392s

OK
1 test OK.
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:

[kristjanvalur]

I found one problem in test_docxmlrpc.py, a missing "import socket" at
the top (for the socket.timeout exception handler).
Please add that, and see what happens.

But there seems to be a problem with the error handing in your build.
probably, sys.stderr is messed up somehow. You could try putting a
breakpoint in line 444 of threadmodule.c and get to the bottom of it.

But to short circuit that,
try adding an "import sys" and then in the finally clause (around line
54) add "sys.stderr = sys.__stderr__" and see if you get better error
output.

[ezio-melotti]

sys.stderr looks ok, I tried what you said and I also tried to put some
"assert sys.stderr == sys.__stderr__" here and there and nothing changed.
I commented out the tests one by one and I found out that the first test
alone (test_valid_get_response) is enough to have that error message
(the second alone doesn't seem to show any error).
There's also a comment that says that response.read() is necessary but
nothing changes if I remove that line.
I also noticed that if I put a print at the end of the finally block the
error message doesn't appear anymore.
I didn't tried to debug threadmodule.c.

[kristjanvalur]

Well, I think the only way to move forward with this is to try to
figure out what the actual error is. Something is wrong in printing it
out to stderr, and it would be helpful to understand why it is not
being output. Perhaps this is some buffering issue. You could try to
redirect stderr to a file, using something like this i the finally
clause:
sys.stderr = open("/tmp/err.txt", "w", 0)

[ezio-melotti]

Same message in the terminal and nothing in the file.

Can you reproduce it if you run the test several times? Some times I
have to run "./python Lib/test/regrtest.py -v test_docxmlrpc" 15-20 or
even more times before the message appears.

[kristjanvalur]

I can't reproduce this, no. I only have access to a windows machine
and you appear to have a custom build (no zlib). I need your help to
get the error message, so you need to try harder. (there are probably
at least two problems, one causing an exeption and one causing the
error output to fail)

Try adding something like this into the file, after the except
socket.timeout clause:
except:
import traceback
traceback.print_exc(100, open(r"c:/tmp/err.txt", "a", 0))
raise

[ezio-melotti]

I tried that too, with no results.

However using advanced debugging techniques I found something new. I
added 3 prints in the finally, one before serv.server_close(), one
after, and the last at the end, after evt.set().
The first two works, the third one sometimes doesn't (i.e. nothing is
printed), even if no errors are displayed.

I'm not 100% sure but at some point I think I saw the third message
being printed at the beginning, before the first. But now I can't
reproduce it anymore.

[ezio-melotti]

I tried to remove the first two prints (I reverted the file and the only
difference with the original was the print after evt.set() (that here
didn't print anything)) and I got something better:

$ ./python Lib/test/regrtest.py -v test_docxmlrpc
test_docxmlrpc
test_valid_get_response (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) ... ok

---

Ran 1 test in 0.063s

OK
1 test OK.
Exception in thread Thread-1 (most likely raised during interpreter
shutdown):
Traceback (most recent call last):
  File "/home/wolf/python-trunk/Lib/threading.py", line 524, in
__bootstrap_inner
  File "/home/wolf/python-trunk/Lib/threading.py", line 477, in run
  File "/home/wolf/python-trunk/Lib/test/test_docxmlrpc.py", line 55, in
server
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute
'write'
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:

[kristjanvalur]

Ah, so this is an interpreter shutdown issue, it seems.
Something is causing this thread to not exit until the application exits
and that can cause all sorts of weird race conditions. I wonder why that
is happening. There must be an issue with test_valid_get_response

[kristjanvalur]

No, sorry. You have just introduced the race condition by putting a print
after the evt.set(). Setting the event causes the main thread to exit
python and so anything after that line will likely not work.

In fact, this is probably the reason why we didn't get any sensible error
output, because error of the evt.set in the finally.
Please try the modified version attatched.

[kristjanvalur]

Any news on this?

[ezio-melotti]

Not yet, the machine I was using to work on this is currently broken and
I couldn't test the new test_docxmlrpc yet. Once I've fixed the machine
and tried it I'll let you know.

[ezio-melotti]

I fixed the machine and tried the file that you uploaded, but nothing
changed. This is what I used (traceback.print_exc() prints to stderr,
so the traceback should be visible):
$ while ./python Lib/test/regrtest.py -v test_docxmlrpc.py; do :; done

/dev/null
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:
...

[kristjanvalur]

Ok, this means that the exception is raised after the finally, when the
thread is exiting.
Now, at this point the process is exiting and therefore we have trouble
printing the exception. (this is probably also the cause of the
exception in the first place, some cleanup that fails because the
process is exiting).
Now, the traceback printing code is complex and it could fail in a
multitude of places, but from the stuff that is output, it would appear
that printing to sys.stderr is failing. To find out the error, we need
to goad python into displaying the exception, even though the process is
exiting.
On a windows machiine, I would put a DebugBreak() call in t_bootstrap()
in threadmodule_t and try to find out where the error output code is
failing. I don't know if you can do just-in-time debugging on your
machine. It would be best, of course, to singlestep through the code at
line 452 in threadmodule.c. Then we could pinpoint where the output is
failing and fix python to make it more resilient at process exit.

Now, lines 452-455 don't produce any output. For starters, try to
insert "file=0" before line 452 in threadmodule.c, so that
PyObject_Print is called instead of PyFile_WriteObject() (which I think
is failing.)
If that produces output, then we can proceed to hardwire "stderr" into
the traceback stuff.

[bitdancer]

Ezio, the original problem this ticket was opened for appears to be
solved, so I'm going to close it. If you still want to work on the
thread exception issue, please open a new ticket referencing this one.