Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

defer to stdlib for path.get_home_dir() #998

Merged
merged 3 commits into from

3 participants

@minrk
Owner

We have elaborate and fragile logic for determining
home dir, and it is ultimately less reliable than the stdlib behavior
used for os.path.expanduser('~'). This commit defers to that in
all cases other than a bundled Python in py2exe/py2app environments.

The one case where the default guess will not be correct, based on
inline comments, is on WinHPC, where all paths must be UNC (\\foo), and
thus HOMESHARE is the logical first choice. However, HOMESHARE is
the wrong answer in approximately all other cases where it is defined,
and the fix for WinHPC users is the trivial HOME=%HOMESHARE%.

This removes the various tests of our Windows path resolution logic,
which are no longer relevant. Further, $HOME is used by the stdlib
as first priority on all platforms, so tests for this behavior are
no longer posix-specific.

closes gh-970
closes gh-747

@minrk
Owner

I also made a small change, allowing IPython to start without requiring that $HOME is actually writable, because we almost never write anything there. Note that we already handle the actually important case of the ipython dir being writable, falling back on a temp dir if not.

If HOME does not exist, the only two references to HOME that I found (%cd with no args, and %logstart global) will raise proper, informative 'no such file' errors, but IPython is fully functional otherwise, so it seemed silly to have a fatal error on something that isn't actually a requirement for reasonably well behaved IPython.

@minrk
Owner

pinging @ellisonbg as the individual with the most experience on the one system that will be adversely affected by this change: Windows-based clusters. Do you still have a test system, where we can check what needs to be done when HOMESHARE is not the default choice? Is it difficult to set HOME or IPYTHON_DIR for jobs?

IPython/utils/path.py
((90 lines not shown))
else:
- raise HomeDirError('No valid home directory could be found for your OS')
+ raise HomeDirError('%s is not a writable dir, set $HOME env to override' % homedir)
@fperez Owner
fperez added a note

spell out 'environment variable' in full: windows users may be confused if they only see 'env', while the full word is easy to google for even if they don't know how/where to configure the environment in windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@fperez
Owner

I definitely like the simplification this gives, though I'm a little skittish at jettisoning logic that might actually protect a valid corner case perhaps not covered by expanduser('~'). What do you think of #154, for example?

I'd also like to hear @ellisonbg's opinion: after all the pain of getting all that working on the winhpc server he went through, the last thing we want is to break that!

So let's be cautious with this one, but if indeed we can remove those hacks for such a vastly simpler replacement, I'd be delighted! My only other feedback is a tiny fix on a user-facing message.

Finally, this should also (once we settle the policy decision) have a paragraph in the docs explaining how to configure things (and remove any possibly outdated info that could refer to the old logic).

But thanks for the big cleanup, I hope we can indeed merge the whole thing!

@ellisonbg
Owner
@minrk
Owner

I definitely don't trust expanduser to get all of the corner cases
correct unless we can verify that it handles everything that our
current code does.

No, expanduser will not get all of the corner cases, but it does a better job than our current code in almost all normal cases. The problem is that we simply cannot get the right answer on both WinHPC and the rest of Windows, because WinHPC wants HOMESHARE for its UNC path, which is the wrong answer approximately everywhere else that it is defined.

An advantage of expanduser is that HOME is first priority on all platforms, so it is the easiest and most natural to manually specify.

Using expanduser means that covering every possible cornercase is handled with one line of user code, setting $HOME prior to launching IPython, in all environments on all platforms.

Our choice is between:

A) requiring all WinHPC users to specify HOME=%HOMESHARE% (or just use IPYTHON_DIR, which is enough for the parallel apps)
B) requiring most non-WinHPC users to specify HOMESHARE=something (this is not guaranteed to be possible, and just using IPYTHON_DIR does not cut it for interactive IPython users).

I understand it would be nice to simplify our code, but before we do
that, let's figure out exactly what expanduser does.

os.path.expanduser priority on Windows:

  1. $HOME
  2. $USERPROFILE
  3. $HOMEDRIVE\$HOMEPATH

and on Unix:

  1. $HOME
  2. query passwd database via pwd module
@ellisonbg
Owner
@minrk
Owner

IIRC, HOMESHARE is not usually set on Windows and the current logic
will ignore it if it is not set and move onto the other options
(HOMEDRIVE+HOMEPATH, USERROFILE, My Documents, HOME). So the only
time HOMESHARE is used is when it is set and is actually the option
that likely should be used. We know this approach works as this is
how our code has done it for about 2 years now.

The fact that we have had multiple bug reports (including #747) of cases where HOMESHARE is the wrong choice would suggest otherwise.

I don't see how these are our only two options. If non-WinHPC users
don't set HOMESHARE, it is ignored and the other options are
attempted, which is what we do today.

The problem is the enterprise/computer lab environments, which regularly do set this env, where it is not the right answer. It would appear from the user reports that HOMESHARE is never right outside WinHPC even when defined, but obviously we won't be getting reports that we are doing the right thing. Sysadmins, not users, set HOMESHARE, but users can set HOME. So if HOMESHARE is wrong, there isn't much recourse if it is the first priority, but if HOME is wrong or undefined, it should be safe to override.

So the logical choices for first priority are to match the rest of the Python universe and use HOME, or use HOMESHARE, which has the benefit of guaranteeing UNC paths on WinHPC, but the disadvantages of being the wrong answer everywhere else that it is defined, along with being internally inconsistent with the rest of Python.

The fact is that both of these cases are guaranteed to get the wrong answer in some cases, so we just have to pick the one that is going to require some extra config. WinHPC seems the more logical choice to me, because it is already the more complicated and unconventional use case, and the fix is standard and less intrusive.

OK, thanks for tracking this down. Quite different logic than we have
though. This makes me wonder (even aside from the HOMESHARE issue)
why our ordering is so different.

Yes, I think we are exactly backwards. If we do restore HOMESHARE as first priority, we should use expanduser after that, and can still fall back on My Documents in the end, if we want.

@ellisonbg
Owner
@minrk
Owner

Given the situation, I think that using expanduser makes sense.

Okay, should I add the My Documents wreg bit back in as a fallback? One case we no longer handle on Windows is the lack of any environment, but I don't know when/where that might come up, if ever.

@fperez
Owner
@minrk
Owner

Okay, My Documents fallback is back (along with its test), and @fperez's comment on the error message is addressed.

minrk added some commits
@minrk minrk defer to stdlib for path.get_home_dir()
We have elaborate and fragile logic for determining
home dir, and it is ultimately less reliable than the stdlib behavior
used for `os.path.expanduser('~')`.  This commit defers to that in
all cases other than a bundled Python in py2exe/py2app environments.

The one case where the default guess will *not* be correct, based on
inline comments, is on WinHPC, where all paths must be UNC (`\\foo`), and
thus HOMESHARE is the logical first choice.  However, HOMESHARE is
the wrong answer in approximately all other cases where it is defined,
and the fix for WinHPC users is the trivial `HOME=%HOMESHARE%`.

This removes the various tests of our Windows path resolution logic,
which are no longer relevant. Further, $HOME is used by the stdlib
as first priority on *all* platforms, so tests for this behavior are
no longer posix-specific.

closes gh-970
closes gh-747
b5ca646
@minrk minrk allow IPython to run without writable home dir
get_ipython_dir() ensures that the *IPython* dir is writable, which is more relevant, but the home dir need not be writable. Some optional behaviors (e.g. `%logstart global`) will not work if the home dir is not writable, but IPython should not crash.  Approximately no other operations actually depend on writing directly to $HOME.
db42b5f
@minrk minrk restore My Documents fallback for get_home_dir on Windows 9df2cbb
@fperez
Owner

@minrk, this looks pretty much ready to merge, no? If so, feel free to go ahead with it, the last commits look fine.

Thanks for the cleanup and being patient to work through the issues on WinHPC with @ellisonbg!

@minrk
Owner

Sure, it seems well behaved to me. I'll give it a few iptests, then merge if nothing pops up.

@fperez
Owner
@minrk minrk merged commit 351c8fc into ipython:master
@ellisonbg ellisonbg referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Nov 23, 2011
  1. @minrk

    defer to stdlib for path.get_home_dir()

    minrk authored
    We have elaborate and fragile logic for determining
    home dir, and it is ultimately less reliable than the stdlib behavior
    used for `os.path.expanduser('~')`.  This commit defers to that in
    all cases other than a bundled Python in py2exe/py2app environments.
    
    The one case where the default guess will *not* be correct, based on
    inline comments, is on WinHPC, where all paths must be UNC (`\\foo`), and
    thus HOMESHARE is the logical first choice.  However, HOMESHARE is
    the wrong answer in approximately all other cases where it is defined,
    and the fix for WinHPC users is the trivial `HOME=%HOMESHARE%`.
    
    This removes the various tests of our Windows path resolution logic,
    which are no longer relevant. Further, $HOME is used by the stdlib
    as first priority on *all* platforms, so tests for this behavior are
    no longer posix-specific.
    
    closes gh-970
    closes gh-747
  2. @minrk

    allow IPython to run without writable home dir

    minrk authored
    get_ipython_dir() ensures that the *IPython* dir is writable, which is more relevant, but the home dir need not be writable. Some optional behaviors (e.g. `%logstart global`) will not work if the home dir is not writable, but IPython should not crash.  Approximately no other operations actually depend on writing directly to $HOME.
  3. @minrk
This page is out of date. Refresh to see the latest.
View
15 IPython/core/interactiveshell.py
@@ -552,10 +552,7 @@ def init_syntax_highlighting(self):
def init_pushd_popd_magic(self):
# for pushd/popd management
- try:
- self.home_dir = get_home_dir()
- except HomeDirError, msg:
- fatal(msg)
+ self.home_dir = get_home_dir()
self.dir_stack = []
@@ -1751,12 +1748,10 @@ def init_readline(self):
# Or if libedit is used, load editrc.
inputrc_name = os.environ.get('INPUTRC')
if inputrc_name is None:
- home_dir = get_home_dir()
- if home_dir is not None:
- inputrc_name = '.inputrc'
- if readline.uses_libedit:
- inputrc_name = '.editrc'
- inputrc_name = os.path.join(home_dir, inputrc_name)
+ inputrc_name = '.inputrc'
+ if readline.uses_libedit:
+ inputrc_name = '.editrc'
+ inputrc_name = os.path.join(self.home_dir, inputrc_name)
if os.path.isfile(inputrc_name):
try:
readline.read_init_file(inputrc_name)
View
120 IPython/utils/path.py
@@ -167,23 +167,25 @@ class HomeDirError(Exception):
pass
-def get_home_dir():
- """Return the closest possible equivalent to a 'home' directory.
-
- * On POSIX, we try $HOME.
- * On Windows we try:
- - %HOMESHARE%
- - %HOMEDRIVE\%HOMEPATH%
- - %USERPROFILE%
- - Registry hack for My Documents
- - %HOME%: rare, but some people with unix-like setups may have defined it
- * On Dos C:\
-
- Currently only Posix and NT are implemented, a HomeDirError exception is
- raised for all other OSes.
- """
+def get_home_dir(require_writable=False):
+ """Return the 'home' directory, as a unicode string.
- env = os.environ
+ * First, check for frozen env in case of py2exe
+ * Otherwise, defer to os.path.expanduser('~')
+
+ See stdlib docs for how this is determined.
+ $HOME is first priority on *ALL* platforms.
+
+ Parameters
+ ----------
+
+ require_writable : bool [default: False]
+ if True:
+ guarantees the return value is a writable directory, otherwise
+ raises HomeDirError
+ if False:
+ The path is resolved, but it is not guaranteed to exist or be writable.
+ """
# first, check py2exe distribution root directory for _ipython.
# This overrides all. Normally does not exist.
@@ -197,59 +199,11 @@ def get_home_dir():
if _writable_dir(os.path.join(root, '_ipython')):
os.environ["IPYKITROOT"] = root
return py3compat.cast_unicode(root, fs_encoding)
-
- if os.name == 'posix':
- # Linux, Unix, AIX, OS X
- try:
- homedir = env['HOME']
- except KeyError:
- # Last-ditch attempt at finding a suitable $HOME, on systems where
- # it may not be defined in the environment but the system shell
- # still knows it - reported once as:
- # https://github.com/ipython/ipython/issues/154
- from subprocess import Popen, PIPE
- homedir = Popen('echo $HOME', shell=True,
- stdout=PIPE).communicate()[0].strip()
- if homedir:
- return py3compat.cast_unicode(homedir, fs_encoding)
- else:
- raise HomeDirError('Undefined $HOME, IPython cannot proceed.')
- else:
- return py3compat.cast_unicode(homedir, fs_encoding)
- elif os.name == 'nt':
- # Now for win9x, XP, Vista, 7?
- # For some strange reason all of these return 'nt' for os.name.
- # First look for a network home directory. This will return the UNC
- # path (\\server\\Users\%username%) not the mapped path (Z:\). This
- # is needed when running IPython on cluster where all paths have to
- # be UNC.
- try:
- homedir = env['HOMESHARE']
- except KeyError:
- pass
- else:
- if _writable_dir(homedir):
- return py3compat.cast_unicode(homedir, fs_encoding)
-
- # Now look for a local home directory
- try:
- homedir = os.path.join(env['HOMEDRIVE'],env['HOMEPATH'])
- except KeyError:
- pass
- else:
- if _writable_dir(homedir):
- return py3compat.cast_unicode(homedir, fs_encoding)
-
- # Now the users profile directory
- try:
- homedir = os.path.join(env['USERPROFILE'])
- except KeyError:
- pass
- else:
- if _writable_dir(homedir):
- return py3compat.cast_unicode(homedir, fs_encoding)
-
- # Use the registry to get the 'My Documents' folder.
+
+ homedir = os.path.expanduser('~')
+
+ if not _writable_dir(homedir) and os.name == 'nt':
+ # expanduser failed, use the registry to get the 'My Documents' folder.
try:
import _winreg as wreg
key = wreg.OpenKey(
@@ -260,27 +214,12 @@ def get_home_dir():
key.Close()
except:
pass
- else:
- if _writable_dir(homedir):
- return py3compat.cast_unicode(homedir, fs_encoding)
-
- # A user with a lot of unix tools in win32 may have defined $HOME.
- # Try this as a last ditch option.
- try:
- homedir = env['HOME']
- except KeyError:
- pass
- else:
- if _writable_dir(homedir):
- return py3compat.cast_unicode(homedir, fs_encoding)
-
- # If all else fails, raise HomeDirError
- raise HomeDirError('No valid home directory could be found')
- elif os.name == 'dos':
- # Desperate, may do absurd things in classic MacOS. May work under DOS.
- return u'C:\\'
+
+ if (not require_writable) or _writable_dir(homedir):
+ return py3compat.cast_unicode(homedir, fs_encoding)
else:
- raise HomeDirError('No valid home directory could be found for your OS')
+ raise HomeDirError('%s is not a writable dir, '
+ 'set $HOME environment variable to override' % homedir)
def get_xdg_dir():
"""Return the XDG_CONFIG_HOME, if it is defined and exists, else None.
@@ -292,7 +231,7 @@ def get_xdg_dir():
if os.name == 'posix':
# Linux, Unix, AIX, OS X
- # use ~/.config if not set OR empty
+ # use ~/.config if empty OR not set
xdg = env.get("XDG_CONFIG_HOME", None) or os.path.join(get_home_dir(), '.config')
if xdg and _writable_dir(xdg):
return py3compat.cast_unicode(xdg, fs_encoding)
@@ -316,6 +255,7 @@ def get_ipython_dir():
home_dir = get_home_dir()
xdg_dir = get_xdg_dir()
+
# import pdb; pdb.set_trace() # dbg
ipdir = env.get('IPYTHON_DIR', env.get('IPYTHONDIR', None))
if ipdir is None:
View
62 IPython/utils/tests/test_path.py
@@ -118,7 +118,6 @@ def teardown_environment():
# Build decorator that uses the setup_environment/setup_environment
with_environment = with_setup(setup_environment, teardown_environment)
-
@skip_if_not_win32
@with_environment
def test_get_home_dir_1():
@@ -142,71 +141,34 @@ def test_get_home_dir_2():
#fake filename for IPython.__init__
IPython.__file__ = abspath(join(HOME_TEST_DIR, "Library.zip/IPython/__init__.py")).lower()
- home_dir = path.get_home_dir()
+ home_dir = path.get_home_dir(True)
nt.assert_equal(home_dir, abspath(HOME_TEST_DIR).lower())
@with_environment
-@skip_win32
def test_get_home_dir_3():
- """Testcase $HOME is set, then use its value as home directory."""
+ """get_home_dir() uses $HOME if set"""
env["HOME"] = HOME_TEST_DIR
- home_dir = path.get_home_dir()
+ home_dir = path.get_home_dir(True)
nt.assert_equal(home_dir, env["HOME"])
@with_environment
-@skip_win32
def test_get_home_dir_4():
- """Testcase $HOME is not set, os=='posix'.
- This should fail with HomeDirError"""
+ """get_home_dir() still works if $HOME is not set"""
- os.name = 'posix'
if 'HOME' in env: del env['HOME']
- nt.assert_raises(path.HomeDirError, path.get_home_dir)
+ # this should still succeed, but we don't know what the answer should be
+ home = path.get_home_dir(True)
+ nt.assert_true(path._writable_dir(home))
-
-@skip_if_not_win32
@with_environment
def test_get_home_dir_5():
- """Using HOMEDRIVE + HOMEPATH, os=='nt'.
-
- HOMESHARE is missing.
- """
-
- os.name = 'nt'
- env.pop('HOMESHARE', None)
- env['HOMEDRIVE'], env['HOMEPATH'] = os.path.splitdrive(HOME_TEST_DIR)
- home_dir = path.get_home_dir()
- nt.assert_equal(home_dir, abspath(HOME_TEST_DIR))
-
-
-@skip_if_not_win32
-@with_environment
-def test_get_home_dir_6():
- """Using USERPROFILE, os=='nt'.
-
- HOMESHARE, HOMEDRIVE, HOMEPATH are missing.
- """
-
- os.name = 'nt'
- env.pop('HOMESHARE', None)
- env.pop('HOMEDRIVE', None)
- env.pop('HOMEPATH', None)
- env["USERPROFILE"] = abspath(HOME_TEST_DIR)
- home_dir = path.get_home_dir()
- nt.assert_equal(home_dir, abspath(HOME_TEST_DIR))
-
-
-@skip_if_not_win32
-@with_environment
-def test_get_home_dir_7():
- """Using HOMESHARE, os=='nt'."""
-
- os.name = 'nt'
- env["HOMESHARE"] = abspath(HOME_TEST_DIR)
- home_dir = path.get_home_dir()
- nt.assert_equal(home_dir, abspath(HOME_TEST_DIR))
+ """raise HomeDirError if $HOME is specified, but not a writable dir"""
+ env['HOME'] = abspath(HOME_TEST_DIR+'garbage')
+ # set os.name = posix, to prevent My Documents fallback on Windows
+ os.name = 'posix'
+ nt.assert_raises(path.HomeDirError, path.get_home_dir, True)
# Should we stub wreg fully so we can run the test on all platforms?
Something went wrong with that request. Please try again.