Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

call to nt.assert_true(path._writable_dir(home)) returns false in test_path.py #1998

Closed
idella opened this issue Jun 21, 2012 · 18 comments · Fixed by #2022
Closed

call to nt.assert_true(path._writable_dir(home)) returns false in test_path.py #1998

idella opened this issue Jun 21, 2012 · 18 comments · Fixed by #2022
Milestone

Comments

@idella
Copy link

idella commented Jun 21, 2012

test_nocase_showall (IPython.utils.tests.test_wildcard.Tests) ... ok

======================================================================
ERROR: get_home_dir() still works if $HOME is not set
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython/utils/tests/test_path.py", line 162, in test_get_home_dir_4
    home = path.get_home_dir(True)
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython/utils/path.py", line 222, in get_home_dir
    'set $HOME environment variable to override' % homedir)
HomeDirError: /var/tmp/portage is not a writable dir, set $HOME environment variable to override

Run as root, it claims /var/tmp/portage is not writable.  wrong.
It appears there is much uncertainty in the correct outcome of this tests.

    if 'HOME' in env: del env['HOME']
    # this should still succeed, but we don't know what the answer should be

and

Doctest: IPython.parallel.client.view.DirectView ... ok
Doctest: IPython.parallel.controller.dependency.require ... ok
ERROR

======================================================================
ERROR: test suite for <module 'IPython.parallel.tests' from '/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython/parallel/tests/__init__.py'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/suite.py", line 208, in run
    self.setUp()
  File "/usr/lib64/python2.7/site-packages/nose/suite.py", line 291, in setUp
    self.setupContext(ancestor)
  File "/usr/lib64/python2.7/site-packages/nose/suite.py", line 314, in setupContext
    try_run(context, names)
  File "/usr/lib64/python2.7/site-packages/nose/util.py", line 478, in try_run
    return func()
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython/parallel/tests/__init__.py", line 69, in setup
    add_engines(1)
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython/parallel/tests/__init__.py", line 88, in add_engines
    rc.spin()
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython/parallel/client/client.py", line 802, in spin
    self._flush_iopub(self._iopub_socket)
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython/parallel/client/client.py", line 728, in _flush_iopub
    msg_id = parent['msg_id']
KeyError: 'msg_id'

----------------------------------------------------------------------
Ran 5 tests in 1.749s

FAILED (errors=1)

**********************************************************************
Test suite completed for system with the following information:
{'commit_hash': '<not found>',
 'commit_source': '(none found)',
 'ipython_path': '/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython',
 'ipython_version': '0.12',
 'os_name': 'posix',
 'platform': 'Linux-3.2.1-gentoo-r2-amd64-x86_64-AMD_Athlon-tm-_II_X4_645_Processor-with-gentoo-2.1',
 'sys_executable': '/usr/bin/python2.7',
 'sys_platform': 'linux2',
 'sys_version': '2.7.3 (default, Jun 18 2012, 20:52:42) \n[GCC 4.5.3]'}

Tools and libraries available at test time:
   curses matplotlib pexpect pymongo qt sqlite3 tornado wx wx.aui zmq

Ran 10 test groups in 34.731s

Status:
ERROR - 2 out of 10 test groups failed.
----------------------------------------
Runner failed: IPython.utils
@fperez
Copy link
Member

fperez commented Jun 21, 2012

mmh, could you explain a bit further what you were doing and under what conditions this failed? Without some more information, there's not enough here for us to know what to fix.

@bfroehle
Copy link
Contributor

Your first error appears to be an issue with Python itself, not IPython. I'm not sure how the title of this issue relates to the pasted output from iptest

Sorry. I misread the traceback. You are correct.

@fperez
Copy link
Member

fperez commented Jun 22, 2012

Unless we get actionable info for this within a few days, we'll have to close it; I don't know what we can do here.

@idella
Copy link
Author

idella commented Jun 22, 2012

well, I simply run the test suite of the current ipython; Sorry for not making it clear to you.
archtester ipython # ebuild ipython-0.12-r1.ebuild clean test

I've been badgered a number of times to not enter gentoo specific code or such, but this is quite simple.
As ebuild need be run as root. I've not attempted to run it as user since I have settings in gentoos background config file to validate it to be run as user anyway.
There are 2 separate fails, I simply added the second save making another issue. It seems that the copy paste I did has lost the second half of paths which were made very long by a copy to a gentoo dir {$T}, work/temp

======================================================================
ERROR: get_home_dir() still works if $HOME is not set
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/
IPython/utils/tests/test_path.py", line 162, in test_get_home_dir_4
    home = path.get_home_dir(True)
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/
IPython/utils/path.py", line 222, in get_home_dir
    'set $HOME environment variable to override' % homedir)
HomeDirError: /var/tmp/portage is not a writable dir, set $HOME environment variable to override

The first is from test_path.py
-------------------------------------------------------------------------------
@with_environment
def test_get_home_dir_4():
    """get_home_dir() still works if $HOME is not set"""

    if 'HOME' in env: del env['HOME']
    # this should still succeed, but we don't know what the answer should be
    home = path.get_home_dir(True)
    nt.assert_true(path._writable_dir(home))
------------------------------------------------------------------------------------
Perhaps it is a cause from Python itself.  If so, then that's fine, I shall treat it accordingly.

---------------------------------------------------------------------------------------------------------------------------------------------

The second is a doctest error from from IPython test group: IPython.parallel.

Doctest: IPython.parallel.client.view.DirectView ... ok
Doctest: IPython.parallel.controller.dependency.require ... ok
ERROR:
=======================================================================
ERROR: test suite for <module 'IPython.parallel.tests' from '/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/
temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython/parallel/tests/__init__.py'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/suite.py", line 208, in run
.......
.......
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/
site-packages/IPython/parallel/tests/__init__.py", line 69, in setup
    add_engines(1)
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/
site-packages/IPython/parallel/tests/__init__.py", line 88, in add_engines
    rc.spin()
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython/parallel/c$
    self._flush_iopub(self._iopub_socket)
  File "/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/
IPython/parallel/client/client.py", line 728, in _flush_iopub
    msg_id = parent['msg_id']
KeyError: 'msg_id'

----------------------------------------------------------------------
Ran 5 tests in 1.744s
FAILED (errors=1)

Test suite completed for system with the following information:
{'commit_hash': '<not found>',
 'commit_source': '(none found)',
 'ipython_path': '/mnt/gen2/TmpDir/portage/dev-python/ipython-0.12-r1/temp/tests-2.7/usr/lib64/python2.7/site-packages/IPython',
 'ipython_version': '0.12',
 'os_name': 'posix',
 'platform': 'Linux-3.2.1-gentoo-r2-amd64-x86_64-AMD_Athlon-tm-_II_X4_645_Processor-with-gentoo-2.1',
 'sys_executable': '/usr/bin/python2.7',
 'sys_platform': 'linux2',
 'sys_version': '2.7.3 (default, Jun 18 2012, 20:52:42) \n[GCC 4.5.3]'}

Tools and libraries available at test time:
   curses matplotlib pexpect pymongo qt sqlite3 tornado wx wx.aui zmq
Status:
ERROR - 2 out of 10 test groups failed.

I hope this makes it clear and workable. Thanks

@takluyver
Copy link
Member

I'm not familiar with gentoo, but it looks like you've got 0.12. Can you run the tests with a development version of IPython? We're coming up to a 0.13 release, so if there are bugs in the previous release, they may have already been fixed.

@idella
Copy link
Author

idella commented Jun 22, 2012

well being gentoo doesn't alter the running of the test really. It's run from an ebuild and once run should be the same as running it manually from the source dirs. Anyway, yes it is 0.12 and I just noticed I ran 0.12.0, so I ran 0.12.1 and got the same for the get_home_dir() error. The second error is entirely different,

IPython/parallel/tests/test_mongodb.py
..........
    raise AutoReconnect(', '.join(errors))
AutoReconnect: could not connect to localhost:27017: [Errno 111] Connection refused

So it supposedly got thru the doctest test in this run.

The 1st error is persistent and sure it may be a CPython fault.
HomeDirError: /var/tmp/portage is not a writable dir, is wrong. Although gentoo runs the build in a sandbox it routinely accesses and uses system sites such as /var/tmp and it is of course writable. Anyway I shall look at getting the dev version later, I suspect that means cloning the master repo or such

@takluyver
Copy link
Member

On 22 June 2012 12:22, idella
reply@reply.github.com
wrote:

I suspect that means cloning the master repo or such

Yep, or grab the beta release from a few days ago:
http://archive.ipython.org/testing/0.13.beta1/ipython-0.13.beta1.tar.gz

well being gentoo doesn't alter the running of the test really. It's run from an ebuild and once run should be the same as running it manually from the source dirs.

I suspect that it differs somehow. We test on Linux a lot, both
manually and automatically, and we're not seeing those errors. Here
are the automated test results:

https://jenkins.shiningpanda.com/ipython/job/ipython-multi/

@idella
Copy link
Author

idella commented Jun 22, 2012

ok with 0.12.1 I now have just the one fail still with the get_home_dir() error with python2.6 2.7 persistently,
but since it passes for you with persistent testing I suppose it may be something to do with the gentoo sandbox. I don't even follow the steps used in the ebuild to tun the test suite they are that complex but it does work.
bfroehle if you can outline how you think it's python itself that would suffice. We (at gentoo devland) caught something else lately that stemmed from Cpython and not the package.

The traceback cites test_path.py line 163, that's how the title relates to the pasted output, not making that up.

@with_environment
def test_get_home_dir_4():
    """get_home_dir() still works if $HOME is not set"""

    if 'HOME' in env: del env['HOME']
    # this should still succeed, but we don't know what the answer should be
    home = path.get_home_dir(True)
    nt.assert_true(path._writable_dir(home))        line 163.

It seems a pretty trivial test so we're not playing for sheep stations.
I have the beta -.13 and I can try it at leisure later.

@bfroehle
Copy link
Contributor

I was wrong earlier... was thinking path was os.path, not IPython.utils.path. :-/ sorry.

The assertion is failing because IPython.utils.path.get_home_dir is raising an exception. The code there seems pretty straightforward -- either /var/tmp/portage is not a directory or it is not writeable.

What is the output of ls -ld /var/tmp/portage?

@idella
Copy link
Author

idella commented Jun 23, 2012

archtester dev-python # ls -ld /var/tmp/portage
drwxr-xr-x 4 root root 4096 May 5 2011 /var/tmp/portage

@minrk
Copy link
Member

minrk commented Jun 23, 2012

Are you certain the test suite is run as root? It seems like a reasonable thing to run third-party tests as a restricted user, in which case this dir would obviously not be writable.

As root, what do you get for:

import os
path = '/var/tmp/portage'
print "writable:", os.access(path, os.W_OK)
print "dir:", os.path.isdir(path)

Because that's the sum-total of the _writable_dir function. If those are both True, then your tests are most likely not actually being run as root, and the failure is correct.

In any case, we might just skip this fairly unlikely test case, as it only runs afoul of contrived unrealistic environments like most testing sandboxes.

@idella
Copy link
Author

idella commented Jun 23, 2012

Are you certain the test suite is run as root?
FEATURES="userpriv usersandbox in /etc/make.conf. Now this is s likely source of the confusion. An emerge is always run as root, but these features make running in the sandbox with user privileges, and these settings are recommended hence I use them. However, I think skipping the test catches all best interests. If it fails, we at gentoo say that's not right, pass it to upstream, and it happens that the odd test like comes up that just doesn't 'play right' in a carefully crafted and enforced gentoo sandbox.

@bfroehle
Copy link
Contributor

Sounds like the best thing to do is to just get rid of that specific test. It seems to me that we can't really guarantee that '~' will be writable if $HOME is not set (in testing environments)... It is certainly an odd corner case...

@fperez
Copy link
Member

fperez commented Jun 23, 2012

Are we already running each test group with a tempfile-generated directory set as IPYTHONDIR? I don't recall if that's been merged or not, but that would solve this problem (and in general introduce a healthy decoupling between test groups and the enclosing environment).

@minrk
Copy link
Member

minrk commented Jun 23, 2012

For completeness:

The case this tests is that in any real, sensible (*ix) environment, the HOME variable is not necessary because the pwd package can be relied upon to get the home dir (as evidenced by the fact that this test has been passing for non-pathological environments for a long time). This comes up in various bundled app situations, where HOME need not be defined.

This is failing because pwd.getpwuid(os.getuid()).pw_dir (see the os.path.expanduser source) is not returning a valid choice. This is a bug in either the gentoo sandboxing in general or the usage thereof in this particular build (I don't know enough to be sure), because the passwd database has the home dir of the user as root-owned, which obviously makes no sense.

It is a bit silly to remove a perfectly functional and correct test, but at the same time, it's not really testing IPython - it's testing os.path.expanduser and the system passwd database, which do not give sensible values in this sandbox.

Are we already running each test group with a tempfile-generated directory set as IPYTHONDIR?

We are not, but I also don't believe that would change anything here. I think it's the sandbox itself that has pathological values, and once inside that sandbox the cwd for subprocesses should be irrelevant.

@fperez
Copy link
Member

fperez commented Jun 23, 2012

Thanks for the careful analysis, @minrk. I suggest we close this and let the gentoo folks either figure out their build machinery or report it upstream to Python if they deem it an issue with pwd/os.path (and they can argue their point with the python folks, not us). Does that sound reasonable?

@minrk
Copy link
Member

minrk commented Jun 23, 2012

I guess I should clarify my clarification a little bit :). While the test is fine and there is no IPython bug here, the fact that it mostly tests code totally outside IPython, I come down slightly in favor of removing the test, or at least removing its check for writability.

@fperez
Copy link
Member

fperez commented Jun 23, 2012

Fine with me, though let's only remove the writability check. As you said above, the test is otherwise fine, and it may help us catch other problems sometimes...

@minrk minrk closed this as completed in 405e48d Jun 24, 2012
jenshnielsen pushed a commit to jenshnielsen/ipython that referenced this issue Jun 24, 2012
Don't check writability in test for get_home_dir when HOME is undefined

Some pathological environments such as gentoo sandboxes can give invalid results.

The code path is at least executed, but no values are checked.

closes ipython#1998
mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014
Some pathological environments such as gentoo sandboxes can give invalid results.

closes ipython#1998
mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014
Don't check writability in test for get_home_dir when HOME is undefined

Some pathological environments such as gentoo sandboxes can give invalid results.

The code path is at least executed, but no values are checked.

closes ipython#1998
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants