Changes for OS X (tested with OS X 10.9/Mavericks only) #3

razzmatazz · 2014-04-25T10:22:50Z

use LOCAL_PEERCRED on OS X for checking credentials;
don't do setsockopt(SOL_SOCKET, SO_RCVBUF | SO_SNDBUF, 0) on OS X;– things are working properly still anyway, no apparent buffering is present

ionelmc · 2014-04-25T11:55:20Z

Hmmm, not sure about the buffering settings, I can't recall why I did it, gitblame is no use

You have run the whole test suite on osx right?

razzmatazz · 2014-04-25T14:02:13Z

Sorry, the only testing I did run was:

bash$ python; >>> import manhole; >>> manhole.install()
nc -U /tmp/manhole-$pid

tox fails on me on OS X because python-signalfd dependency doesn't compile (seems to be a linux-specific package):

...
cc -arch x86_64 -O2 -fPIC -Wimplicit -I/opt/X11/include:/usr/local/include: -I/opt/X11/include:/usr/local/include: -I/Users/bob/src/python-manhole/.tox/pypy/include -c signalfd/_signalfd.c -o build/temp.macosx-10.9-x86_64-2.7/signalfd/_signalfd.o

signalfd/_signalfd.c:11:10: fatal error: 'sys/signalfd.h' file not found

#include <sys/signalfd.h>

         ^

1 error generated.

error: command 'cc' failed with exit status 1
...

ionelmc · 2014-04-25T14:04:17Z

Can you try to remove this line https://github.com/ionelmc/python-manhole/blob/master/tox.ini#L22 and try to run tox -e py2.7 again (or whatever python you use) ?

razzmatazz · 2014-04-25T14:13:17Z

A lot of tests failed;
Most of them replied with something in tune of

AssertionError: Waited 10.00secs but ('Not patching os.fork and os.forkpty. Activation is done by signal 12',) did not appear in output in the given order !

Could this be related to disabling calls to setsockopt()? I.e. buffering stops the response from getting other to another process?

Does it fail the same for you when you comment out the setsockopt() calls?

bob@saulius-175 python-manhole (macosx-support)$ tox -e py2.7
GLOB sdist-make: /Users/bob/src/python-manhole/setup.py
py2.7 create: /Users/bob/src/python-manhole/.tox/py2.7
py2.7 installdeps: coverage, process-tests, nose
py2.7 inst: /Users/bob/src/python-manhole/.tox/dist/manhole-0.6.0.zip
py2.7 runtests: PYTHONHASHSEED='3301218095'
py2.7 runtests: commands[0] | nosetests test_manhole --with-coverage --cover-inclusive --cover-erase
Killing <process_tests.TestProcess object at 0x100dd0f10> !
F..Killing <process_tests.TestProcess object at 0x100de2b50> !
F.Killing <process_tests.TestProcess object at 0x100de2d10> !
FKilling <process_tests.TestProcess object at 0x100deb8d0> !
FKilling <process_tests.TestProcess object at 0x100deb9d0> !
FKilling <process_tests.TestProcess object at 0x100deb890> !
FKilling <process_tests.TestProcess object at 0x100df7250> !
FKilling <process_tests.TestProcess object at 0x100df72d0> !
FKilling <process_tests.TestProcess object at 0x100df73d0> !
FKilling <process_tests.TestProcess object at 0x100dffa10> !
FKilling <process_tests.TestProcess object at 0x100dff910> !
FKilling <process_tests.TestProcess object at 0x100dff850> !
FKilling <process_tests.TestProcess object at 0x100dff990> !
FKilling <process_tests.TestProcess object at 0x100e08310> !
FKilling <process_tests.TestProcess object at 0x100e083d0> !
FKilling <process_tests.TestProcess object at 0x100e08450> !
FKilling <process_tests.TestProcess object at 0x100e084d0> !
F
======================================================================
FAIL: test_activate_on_usr2 (test_manhole.ManholeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bob/src/python-manhole/tests/test_manhole.py", line 211, in test_activate_on_usr2
    self.wait_for_strings(proc.read, TIMEOUT, 'Not patching os.fork and os.forkpty. Activation is done by signal 12')
  File "/Users/bob/src/python-manhole/.tox/py2.7/lib/python2.7/site-packages/process_tests.py", line 215, in wait_for_strings
    seconds, strings
AssertionError: Waited 10.00secs but ('Not patching os.fork and os.forkpty. Activation is done by signal 12',) did not appear in output in the given order !
-------------------- >> begin captured stdout << ---------------------
*********** OUTPUT ***********
Manhole: Not patching os.fork and os.forkpty. Activation is done by signal 31

******************************

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: test_exit_with_grace (test_manhole.ManholeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bob/src/python-manhole/tests/test_manhole.py", line 108, in test_exit_with_grace
    'from PID:%s UID:%s' % (os.getpid(), os.getuid()),
  File "/Users/bob/src/python-manhole/.tox/py2.7/lib/python2.7/site-packages/process_tests.py", line 215, in wait_for_strings
    seconds, strings
AssertionError: Waited 10.00secs but ('from PID:5956 UID:501',) did not appear in output in the given order !
-------------------- >> begin captured stdout << ---------------------
*********** OUTPUT ***********

######### ProcessID=5960, ThreadID=4514992128 #########
File: "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783, in __bootstrap
  self.__bootstrap_inner()
File: "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  self.run()
File: "/Users/bob/src/python-manhole/.tox/py2.7/lib/python2.7/site-packages/manhole.py", line 187, in run
  self.handle(self.client)
File: "/Users/bob/src/python-manhole/.tox/py2.7/lib/python2.7/site-packages/manhole.py", line 231, in handle
  run_repl()
File: "/Users/bob/src/python-manhole/.tox/py2.7/lib/python2.7/site-packages/manhole.py", line 260, in run_repl
  dump_stacktraces()
File: "/Users/bob/src/python-manhole/.tox/py2.7/lib/python2.7/site-packages/manhole.py", line 379, in dump_stacktraces
  for filename, lineno, name, line in traceback.extract_stack(stack):

######### ProcessID=5960, ThreadID=4510785536 #########
File: "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783, in __bootstrap
  self.__bootstrap_inner()
File: "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  self.run()
File: "/Users/bob/src/python-manhole/.tox/py2.7/lib/python2.7/site-packages/manhole.py", line 162, in run
  client.join()
File: "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 949, in join
  self.__block.wait()
File: "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 339, in wait
  waiter.acquire()

######### ProcessID=5960, ThreadID=140735310521104 #########
File: "/Users/bob/src/python-manhole/tests/test_manhole.py", line 379, in <module>
  time.sleep(TIMEOUT * 10)
#############################################


Python 2.7.6 (default, Apr  9 2014, 11:48:52)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> FOOBAR
>>>
******************************
*********** OUTPUT ***********
Manhole: Patched <built-in function fork> and <built-in function forkpty>.
Manhole: Manhole UDS path: /tmp/manhole-5960
Manhole: Waiting for new connection (in pid:5960) ...
Manhole: Started ManholeConnection thread. Checking credentials ...
Manhole: Accepted connection <socket._socketobject object at 0x10c5e3c90> from PID:0 UID:501 GID:12

******************************

ionelmc · 2014-04-25T14:23:08Z

Seems that there were some hardcoded signal numbers in the tests. I've fixed them - can you do a git pull --rebase https://github.com/ionelmc/python-manhole.git or something like that ?

Thanks for your patience in sorting this out.

razzmatazz · 2014-04-25T14:31:14Z

Still fails.. https://gist.github.com/razzmatazz/11291569

ionelmc · 2014-04-25T14:40:19Z

Ok, it appears that using LOCAL_PEERCRED doesn't return the client pid in the resulting struct. But it does return what is important (the uid). I have fixed the tests to assert only this.

Can you do the rebase/test dance again?

razzmatazz · 2014-04-25T14:45:37Z

OK, seems to have helped, but..
https://gist.github.com/razzmatazz/11292067

ionelmc · 2014-04-25T14:52:42Z

Can you rebuild the env and retry? (tox -r -e py2.7)

razzmatazz · 2014-04-25T14:56:06Z

https://gist.github.com/razzmatazz/11292420

ionelmc · 2014-04-25T14:58:09Z

What's the value of errno.errorcode[57] on OSX ?

ionelmc · 2014-04-25T22:04:43Z

O, it's seems it's ENOTCONN. Would you mind running it again ? (tox -r)

razzmatazz · 2014-04-27T11:56:40Z

Hi, are there any new changes on your master branch related to this test dump I posted?

https://gist.github.com/razzmatazz/11292420

There seems to be no new commits on ionelmc/python-manhole master branch.

I.e. tests still fail for me with pretty much the same trace for each failed test:

Traceback (most recent call last):
  File "/Users/bob/src/python-manhole/tests/test_manhole.py", line 114, in test_exit_with_grace
    sock.close()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/contextlib.py", line 34, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/bob/src/python-manhole/.tox/py2.6/lib/python2.6/site-packages/process_tests.py", line 223, in dump_on_error
    yield
  File "/Users/bob/src/python-manhole/tests/test_manhole.py", line 113, in test_exit_with_grace
    sock.shutdown(socket.SHUT_RD)
  File "<string>", line 1, in shutdown
error: [Errno 57] Socket is not connected

ionelmc · 2014-04-27T13:48:54Z

Ah sorry, I forgot to mention, the fixes are in https://github.com/ionelmc/python-process-tests - it's used in the test suite - it seems that there were some issues with it on osx.

It has been fixed and uploaded on pypi - if you recreate the env (tox -r) you shouldn't get that error.

razzmatazz · 2014-04-27T15:21:19Z

Hi,
It still doesn't work for me:
https://gist.github.com/razzmatazz/11348270

ionelmc · 2014-04-27T15:28:10Z

We're getting very close - only 1 test failed. I've made a change so that error is captured in case you want run the tests again.

razzmatazz · 2014-04-27T15:35:41Z

Nice!
Should I try it again? Or will you notice me when you've fixed the latest issue?

ionelmc · 2014-04-27T15:36:42Z

It's fixed in the master. Can you do another rebase/merge and run tests again ?

- use LOCAL_PEERCRED on OS X for checking credentials; - don't do setsockopt(SOL_SOCKET, SO_RCVBUF|SO_SNDBUF, 0) on OS X;– things are working properly still anyway, no apparent buffering is present

razzmatazz · 2014-04-27T15:42:20Z

Finally a success!
https://gist.github.com/razzmatazz/11348727

I rebased on your latest master and added a commit to remove python-signalfd dep; - not sure what to do about it?

ionelmc · 2014-04-27T15:43:55Z

Don't worry about that :)

Changes for OS X (tested with OS X 10.9/Mavericks only)

ionelmc · 2014-04-27T15:46:19Z

Thanks for the fix

razzmatazz · 2014-04-27T15:52:11Z

No problem, the process of making it work was time well spent :)

razzmatazz added 2 commits April 27, 2014 18:41

Changes for OS X (tested with OS X 10.9/Mavericks only)

e4c1dfb

- use LOCAL_PEERCRED on OS X for checking credentials; - don't do setsockopt(SOL_SOCKET, SO_RCVBUF|SO_SNDBUF, 0) on OS X;– things are working properly still anyway, no apparent buffering is present

tox.ini: remove python-signalfd dep

1827ad8

ionelmc added a commit that referenced this pull request Apr 27, 2014

Merge pull request #3 from razzmatazz/macosx-support

6971055

Changes for OS X (tested with OS X 10.9/Mavericks only)

ionelmc merged commit 6971055 into ionelmc:master Apr 27, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes for OS X (tested with OS X 10.9/Mavericks only) #3

Changes for OS X (tested with OS X 10.9/Mavericks only) #3

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 27, 2014

ionelmc commented Apr 27, 2014

razzmatazz commented Apr 27, 2014

ionelmc commented Apr 27, 2014

razzmatazz commented Apr 27, 2014

ionelmc commented Apr 27, 2014

razzmatazz commented Apr 27, 2014

ionelmc commented Apr 27, 2014

ionelmc commented Apr 27, 2014

razzmatazz commented Apr 27, 2014

Changes for OS X (tested with OS X 10.9/Mavericks only) #3

Changes for OS X (tested with OS X 10.9/Mavericks only) #3

Conversation

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 25, 2014

ionelmc commented Apr 25, 2014

ionelmc commented Apr 25, 2014

razzmatazz commented Apr 27, 2014

ionelmc commented Apr 27, 2014

razzmatazz commented Apr 27, 2014

ionelmc commented Apr 27, 2014

razzmatazz commented Apr 27, 2014

ionelmc commented Apr 27, 2014

razzmatazz commented Apr 27, 2014

ionelmc commented Apr 27, 2014

ionelmc commented Apr 27, 2014

razzmatazz commented Apr 27, 2014