Skip to content
This repository

defer to stdlib for path.get_home_dir() #998

Merged
merged 3 commits into from over 2 years ago

3 participants

Min RK Fernando Perez Brian E. Granger
Min RK
Owner

We have elaborate and fragile logic for determining
home dir, and it is ultimately less reliable than the stdlib behavior
used for os.path.expanduser('~'). This commit defers to that in
all cases other than a bundled Python in py2exe/py2app environments.

The one case where the default guess will not be correct, based on
inline comments, is on WinHPC, where all paths must be UNC (\\foo), and
thus HOMESHARE is the logical first choice. However, HOMESHARE is
the wrong answer in approximately all other cases where it is defined,
and the fix for WinHPC users is the trivial HOME=%HOMESHARE%.

This removes the various tests of our Windows path resolution logic,
which are no longer relevant. Further, $HOME is used by the stdlib
as first priority on all platforms, so tests for this behavior are
no longer posix-specific.

closes gh-970
closes gh-747

Min RK
Owner

I also made a small change, allowing IPython to start without requiring that $HOME is actually writable, because we almost never write anything there. Note that we already handle the actually important case of the ipython dir being writable, falling back on a temp dir if not.

If HOME does not exist, the only two references to HOME that I found (%cd with no args, and %logstart global) will raise proper, informative 'no such file' errors, but IPython is fully functional otherwise, so it seemed silly to have a fatal error on something that isn't actually a requirement for reasonably well behaved IPython.

Min RK
Owner

pinging @ellisonbg as the individual with the most experience on the one system that will be adversely affected by this change: Windows-based clusters. Do you still have a test system, where we can check what needs to be done when HOMESHARE is not the default choice? Is it difficult to set HOME or IPYTHON_DIR for jobs?

IPython/utils/path.py
((90 lines not shown))
282 206
     else:
283  
-        raise HomeDirError('No valid home directory could be found for your OS')
  207
+        raise HomeDirError('%s is not a writable dir, set $HOME env to override' % homedir)
1
Fernando Perez Owner
fperez added a note November 20, 2011

spell out 'environment variable' in full: windows users may be confused if they only see 'env', while the full word is easy to google for even if they don't know how/where to configure the environment in windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez
Owner

I definitely like the simplification this gives, though I'm a little skittish at jettisoning logic that might actually protect a valid corner case perhaps not covered by expanduser('~'). What do you think of #154, for example?

I'd also like to hear @ellisonbg's opinion: after all the pain of getting all that working on the winhpc server he went through, the last thing we want is to break that!

So let's be cautious with this one, but if indeed we can remove those hacks for such a vastly simpler replacement, I'd be delighted! My only other feedback is a tiny fix on a user-facing message.

Finally, this should also (once we settle the policy decision) have a paragraph in the docs explaining how to configure things (and remove any possibly outdated info that could refer to the old logic).

But thanks for the big cleanup, I hope we can indeed merge the whole thing!

Brian E. Granger
Owner
Min RK
Owner

I definitely don't trust expanduser to get all of the corner cases
correct unless we can verify that it handles everything that our
current code does.

No, expanduser will not get all of the corner cases, but it does a better job than our current code in almost all normal cases. The problem is that we simply cannot get the right answer on both WinHPC and the rest of Windows, because WinHPC wants HOMESHARE for its UNC path, which is the wrong answer approximately everywhere else that it is defined.

An advantage of expanduser is that HOME is first priority on all platforms, so it is the easiest and most natural to manually specify.

Using expanduser means that covering every possible cornercase is handled with one line of user code, setting $HOME prior to launching IPython, in all environments on all platforms.

Our choice is between:

A) requiring all WinHPC users to specify HOME=%HOMESHARE% (or just use IPYTHON_DIR, which is enough for the parallel apps)
B) requiring most non-WinHPC users to specify HOMESHARE=something (this is not guaranteed to be possible, and just using IPYTHON_DIR does not cut it for interactive IPython users).

I understand it would be nice to simplify our code, but before we do
that, let's figure out exactly what expanduser does.

os.path.expanduser priority on Windows:

  1. $HOME
  2. $USERPROFILE
  3. $HOMEDRIVE\$HOMEPATH

and on Unix:

  1. $HOME
  2. query passwd database via pwd module
Brian E. Granger
Owner
Min RK
Owner

IIRC, HOMESHARE is not usually set on Windows and the current logic
will ignore it if it is not set and move onto the other options
(HOMEDRIVE+HOMEPATH, USERROFILE, My Documents, HOME). So the only
time HOMESHARE is used is when it is set and is actually the option
that likely should be used. We know this approach works as this is
how our code has done it for about 2 years now.

The fact that we have had multiple bug reports (including #747) of cases where HOMESHARE is the wrong choice would suggest otherwise.

I don't see how these are our only two options. If non-WinHPC users
don't set HOMESHARE, it is ignored and the other options are
attempted, which is what we do today.

The problem is the enterprise/computer lab environments, which regularly do set this env, where it is not the right answer. It would appear from the user reports that HOMESHARE is never right outside WinHPC even when defined, but obviously we won't be getting reports that we are doing the right thing. Sysadmins, not users, set HOMESHARE, but users can set HOME. So if HOMESHARE is wrong, there isn't much recourse if it is the first priority, but if HOME is wrong or undefined, it should be safe to override.

So the logical choices for first priority are to match the rest of the Python universe and use HOME, or use HOMESHARE, which has the benefit of guaranteeing UNC paths on WinHPC, but the disadvantages of being the wrong answer everywhere else that it is defined, along with being internally inconsistent with the rest of Python.

The fact is that both of these cases are guaranteed to get the wrong answer in some cases, so we just have to pick the one that is going to require some extra config. WinHPC seems the more logical choice to me, because it is already the more complicated and unconventional use case, and the fix is standard and less intrusive.

OK, thanks for tracking this down. Quite different logic than we have
though. This makes me wonder (even aside from the HOMESHARE issue)
why our ordering is so different.

Yes, I think we are exactly backwards. If we do restore HOMESHARE as first priority, we should use expanduser after that, and can still fall back on My Documents in the end, if we want.

Brian E. Granger
Owner
Min RK
Owner

Given the situation, I think that using expanduser makes sense.

Okay, should I add the My Documents wreg bit back in as a fallback? One case we no longer handle on Windows is the lack of any environment, but I don't know when/where that might come up, if ever.

Fernando Perez
Owner
Min RK
Owner

Okay, My Documents fallback is back (along with its test), and @fperez's comment on the error message is addressed.

added some commits November 13, 2011
Min RK defer to stdlib for path.get_home_dir()
We have elaborate and fragile logic for determining
home dir, and it is ultimately less reliable than the stdlib behavior
used for `os.path.expanduser('~')`.  This commit defers to that in
all cases other than a bundled Python in py2exe/py2app environments.

The one case where the default guess will *not* be correct, based on
inline comments, is on WinHPC, where all paths must be UNC (`\\foo`), and
thus HOMESHARE is the logical first choice.  However, HOMESHARE is
the wrong answer in approximately all other cases where it is defined,
and the fix for WinHPC users is the trivial `HOME=%HOMESHARE%`.

This removes the various tests of our Windows path resolution logic,
which are no longer relevant. Further, $HOME is used by the stdlib
as first priority on *all* platforms, so tests for this behavior are
no longer posix-specific.

closes gh-970
closes gh-747
b5ca646
Min RK allow IPython to run without writable home dir
get_ipython_dir() ensures that the *IPython* dir is writable, which is more relevant, but the home dir need not be writable. Some optional behaviors (e.g. `%logstart global`) will not work if the home dir is not writable, but IPython should not crash.  Approximately no other operations actually depend on writing directly to $HOME.
db42b5f
Min RK restore My Documents fallback for get_home_dir on Windows 9df2cbb
Fernando Perez
Owner

@minrk, this looks pretty much ready to merge, no? If so, feel free to go ahead with it, the last commits look fine.

Thanks for the cleanup and being patient to work through the issues on WinHPC with @ellisonbg!

Min RK
Owner

Sure, it seems well behaved to me. I'll give it a few iptests, then merge if nothing pops up.

Fernando Perez
Owner
Min RK minrk merged commit 351c8fc into from November 23, 2011
Min RK minrk closed this November 23, 2011
Brian E. Granger ellisonbg referenced this pull request from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 3 unique commits by 1 author.

Nov 22, 2011
Min RK defer to stdlib for path.get_home_dir()
We have elaborate and fragile logic for determining
home dir, and it is ultimately less reliable than the stdlib behavior
used for `os.path.expanduser('~')`.  This commit defers to that in
all cases other than a bundled Python in py2exe/py2app environments.

The one case where the default guess will *not* be correct, based on
inline comments, is on WinHPC, where all paths must be UNC (`\\foo`), and
thus HOMESHARE is the logical first choice.  However, HOMESHARE is
the wrong answer in approximately all other cases where it is defined,
and the fix for WinHPC users is the trivial `HOME=%HOMESHARE%`.

This removes the various tests of our Windows path resolution logic,
which are no longer relevant. Further, $HOME is used by the stdlib
as first priority on *all* platforms, so tests for this behavior are
no longer posix-specific.

closes gh-970
closes gh-747
b5ca646
Min RK allow IPython to run without writable home dir
get_ipython_dir() ensures that the *IPython* dir is writable, which is more relevant, but the home dir need not be writable. Some optional behaviors (e.g. `%logstart global`) will not work if the home dir is not writable, but IPython should not crash.  Approximately no other operations actually depend on writing directly to $HOME.
db42b5f
Min RK restore My Documents fallback for get_home_dir on Windows 9df2cbb
This page is out of date. Refresh to see the latest.
15  IPython/core/interactiveshell.py
@@ -552,10 +552,7 @@ def init_syntax_highlighting(self):
552 552
 
553 553
     def init_pushd_popd_magic(self):
554 554
         # for pushd/popd management
555  
-        try:
556  
-            self.home_dir = get_home_dir()
557  
-        except HomeDirError, msg:
558  
-            fatal(msg)
  555
+        self.home_dir = get_home_dir()
559 556
 
560 557
         self.dir_stack = []
561 558
 
@@ -1751,12 +1748,10 @@ def init_readline(self):
1751 1748
             # Or if libedit is used, load editrc.
1752 1749
             inputrc_name = os.environ.get('INPUTRC')
1753 1750
             if inputrc_name is None:
1754  
-                home_dir = get_home_dir()
1755  
-                if home_dir is not None:
1756  
-                    inputrc_name = '.inputrc'
1757  
-                    if readline.uses_libedit:
1758  
-                        inputrc_name = '.editrc'
1759  
-                    inputrc_name = os.path.join(home_dir, inputrc_name)
  1751
+                inputrc_name = '.inputrc'
  1752
+                if readline.uses_libedit:
  1753
+                    inputrc_name = '.editrc'
  1754
+                inputrc_name = os.path.join(self.home_dir, inputrc_name)
1760 1755
             if os.path.isfile(inputrc_name):
1761 1756
                 try:
1762 1757
                     readline.read_init_file(inputrc_name)
120  IPython/utils/path.py
@@ -167,23 +167,25 @@ class HomeDirError(Exception):
167 167
     pass
168 168
 
169 169
 
170  
-def get_home_dir():
171  
-    """Return the closest possible equivalent to a 'home' directory.
172  
-
173  
-    * On POSIX, we try $HOME.
174  
-    * On Windows we try:
175  
-      - %HOMESHARE%
176  
-      - %HOMEDRIVE\%HOMEPATH%
177  
-      - %USERPROFILE%
178  
-      - Registry hack for My Documents
179  
-      - %HOME%: rare, but some people with unix-like setups may have defined it
180  
-    * On Dos C:\
181  
-
182  
-    Currently only Posix and NT are implemented, a HomeDirError exception is
183  
-    raised for all other OSes.
184  
-    """
  170
+def get_home_dir(require_writable=False):
  171
+    """Return the 'home' directory, as a unicode string.
185 172
 
186  
-    env = os.environ
  173
+    * First, check for frozen env in case of py2exe
  174
+    * Otherwise, defer to os.path.expanduser('~')
  175
+    
  176
+    See stdlib docs for how this is determined.
  177
+    $HOME is first priority on *ALL* platforms.
  178
+    
  179
+    Parameters
  180
+    ----------
  181
+    
  182
+    require_writable : bool [default: False]
  183
+        if True:
  184
+            guarantees the return value is a writable directory, otherwise
  185
+            raises HomeDirError
  186
+        if False:
  187
+            The path is resolved, but it is not guaranteed to exist or be writable.
  188
+    """
187 189
 
188 190
     # first, check py2exe distribution root directory for _ipython.
189 191
     # This overrides all. Normally does not exist.
@@ -197,59 +199,11 @@ def get_home_dir():
197 199
         if _writable_dir(os.path.join(root, '_ipython')):
198 200
             os.environ["IPYKITROOT"] = root
199 201
         return py3compat.cast_unicode(root, fs_encoding)
200  
-
201  
-    if os.name == 'posix':
202  
-        # Linux, Unix, AIX, OS X
203  
-        try:
204  
-            homedir = env['HOME']
205  
-        except KeyError:
206  
-            # Last-ditch attempt at finding a suitable $HOME, on systems where
207  
-            # it may not be defined in the environment but the system shell
208  
-            # still knows it - reported once as:
209  
-            # https://github.com/ipython/ipython/issues/154
210  
-            from subprocess import Popen, PIPE
211  
-            homedir = Popen('echo $HOME', shell=True,
212  
-                            stdout=PIPE).communicate()[0].strip()
213  
-            if homedir:
214  
-                return py3compat.cast_unicode(homedir, fs_encoding)
215  
-            else:
216  
-                raise HomeDirError('Undefined $HOME, IPython cannot proceed.')
217  
-        else:
218  
-            return py3compat.cast_unicode(homedir, fs_encoding)
219  
-    elif os.name == 'nt':
220  
-        # Now for win9x, XP, Vista, 7?
221  
-        # For some strange reason all of these return 'nt' for os.name.
222  
-        # First look for a network home directory. This will return the UNC
223  
-        # path (\\server\\Users\%username%) not the mapped path (Z:\). This
224  
-        # is needed when running IPython on cluster where all paths have to
225  
-        # be UNC.
226  
-        try:
227  
-            homedir = env['HOMESHARE']
228  
-        except KeyError:
229  
-            pass
230  
-        else:
231  
-            if _writable_dir(homedir):
232  
-                return py3compat.cast_unicode(homedir, fs_encoding)
233  
-
234  
-        # Now look for a local home directory
235  
-        try:
236  
-            homedir = os.path.join(env['HOMEDRIVE'],env['HOMEPATH'])
237  
-        except KeyError:
238  
-            pass
239  
-        else:
240  
-            if _writable_dir(homedir):
241  
-                return py3compat.cast_unicode(homedir, fs_encoding)
242  
-
243  
-        # Now the users profile directory
244  
-        try:
245  
-            homedir = os.path.join(env['USERPROFILE'])
246  
-        except KeyError:
247  
-            pass
248  
-        else:
249  
-            if _writable_dir(homedir):
250  
-                return py3compat.cast_unicode(homedir, fs_encoding)
251  
-
252  
-        # Use the registry to get the 'My Documents' folder.
  202
+    
  203
+    homedir = os.path.expanduser('~')
  204
+    
  205
+    if not _writable_dir(homedir) and os.name == 'nt':
  206
+        # expanduser failed, use the registry to get the 'My Documents' folder.
253 207
         try:
254 208
             import _winreg as wreg
255 209
             key = wreg.OpenKey(
@@ -260,27 +214,12 @@ def get_home_dir():
260 214
             key.Close()
261 215
         except:
262 216
             pass
263  
-        else:
264  
-            if _writable_dir(homedir):
265  
-                return py3compat.cast_unicode(homedir, fs_encoding)
266  
-
267  
-        # A user with a lot of unix tools in win32 may have defined $HOME.
268  
-        # Try this as a last ditch option.
269  
-        try:
270  
-            homedir = env['HOME']
271  
-        except KeyError:
272  
-            pass
273  
-        else:
274  
-            if _writable_dir(homedir):
275  
-                return py3compat.cast_unicode(homedir, fs_encoding)
276  
-
277  
-        # If all else fails, raise HomeDirError
278  
-        raise HomeDirError('No valid home directory could be found')
279  
-    elif os.name == 'dos':
280  
-        # Desperate, may do absurd things in classic MacOS. May work under DOS.
281  
-        return u'C:\\'
  217
+    
  218
+    if (not require_writable) or _writable_dir(homedir):
  219
+        return py3compat.cast_unicode(homedir, fs_encoding)
282 220
     else:
283  
-        raise HomeDirError('No valid home directory could be found for your OS')
  221
+        raise HomeDirError('%s is not a writable dir, '
  222
+                'set $HOME environment variable to override' % homedir)
284 223
 
285 224
 def get_xdg_dir():
286 225
     """Return the XDG_CONFIG_HOME, if it is defined and exists, else None.
@@ -292,7 +231,7 @@ def get_xdg_dir():
292 231
 
293 232
     if os.name == 'posix':
294 233
         # Linux, Unix, AIX, OS X
295  
-        # use ~/.config if not set OR empty
  234
+        # use ~/.config if empty OR not set
296 235
         xdg = env.get("XDG_CONFIG_HOME", None) or os.path.join(get_home_dir(), '.config')
297 236
         if xdg and _writable_dir(xdg):
298 237
             return py3compat.cast_unicode(xdg, fs_encoding)
@@ -316,6 +255,7 @@ def get_ipython_dir():
316 255
 
317 256
     home_dir = get_home_dir()
318 257
     xdg_dir = get_xdg_dir()
  258
+    
319 259
     # import pdb; pdb.set_trace()  # dbg
320 260
     ipdir = env.get('IPYTHON_DIR', env.get('IPYTHONDIR', None))
321 261
     if ipdir is None:
62  IPython/utils/tests/test_path.py
@@ -118,7 +118,6 @@ def teardown_environment():
118 118
 # Build decorator that uses the setup_environment/setup_environment
119 119
 with_environment = with_setup(setup_environment, teardown_environment)
120 120
 
121  
-
122 121
 @skip_if_not_win32
123 122
 @with_environment
124 123
 def test_get_home_dir_1():
@@ -142,71 +141,34 @@ def test_get_home_dir_2():
142 141
     #fake filename for IPython.__init__
143 142
     IPython.__file__ = abspath(join(HOME_TEST_DIR, "Library.zip/IPython/__init__.py")).lower()
144 143
 
145  
-    home_dir = path.get_home_dir()
  144
+    home_dir = path.get_home_dir(True)
146 145
     nt.assert_equal(home_dir, abspath(HOME_TEST_DIR).lower())
147 146
 
148 147
 
149 148
 @with_environment
150  
-@skip_win32
151 149
 def test_get_home_dir_3():
152  
-    """Testcase $HOME is set, then use its value as home directory."""
  150
+    """get_home_dir() uses $HOME if set"""
153 151
     env["HOME"] = HOME_TEST_DIR
154  
-    home_dir = path.get_home_dir()
  152
+    home_dir = path.get_home_dir(True)
155 153
     nt.assert_equal(home_dir, env["HOME"])
156 154
 
157 155
 
158 156
 @with_environment
159  
-@skip_win32
160 157
 def test_get_home_dir_4():
161  
-    """Testcase $HOME is not set, os=='posix'.
162  
-    This should fail with HomeDirError"""
  158
+    """get_home_dir() still works if $HOME is not set"""
163 159
 
164  
-    os.name = 'posix'
165 160
     if 'HOME' in env: del env['HOME']
166  
-    nt.assert_raises(path.HomeDirError, path.get_home_dir)
  161
+    # this should still succeed, but we don't know what the answer should be
  162
+    home = path.get_home_dir(True)
  163
+    nt.assert_true(path._writable_dir(home))
167 164
 
168  
-
169  
-@skip_if_not_win32
170 165
 @with_environment
171 166
 def test_get_home_dir_5():
172  
-    """Using HOMEDRIVE + HOMEPATH, os=='nt'.
173  
-
174  
-    HOMESHARE is missing.
175  
-    """
176  
-
177  
-    os.name = 'nt'
178  
-    env.pop('HOMESHARE', None)
179  
-    env['HOMEDRIVE'], env['HOMEPATH'] = os.path.splitdrive(HOME_TEST_DIR)
180  
-    home_dir = path.get_home_dir()
181  
-    nt.assert_equal(home_dir, abspath(HOME_TEST_DIR))
182  
-
183  
-
184  
-@skip_if_not_win32
185  
-@with_environment
186  
-def test_get_home_dir_6():
187  
-    """Using USERPROFILE, os=='nt'.
188  
-
189  
-    HOMESHARE, HOMEDRIVE, HOMEPATH are missing.
190  
-    """
191  
-
192  
-    os.name = 'nt'
193  
-    env.pop('HOMESHARE', None)
194  
-    env.pop('HOMEDRIVE', None)
195  
-    env.pop('HOMEPATH', None)
196  
-    env["USERPROFILE"] = abspath(HOME_TEST_DIR)
197  
-    home_dir = path.get_home_dir()
198  
-    nt.assert_equal(home_dir, abspath(HOME_TEST_DIR))
199  
-
200  
-
201  
-@skip_if_not_win32
202  
-@with_environment
203  
-def test_get_home_dir_7():
204  
-    """Using HOMESHARE, os=='nt'."""
205  
-
206  
-    os.name = 'nt'
207  
-    env["HOMESHARE"] = abspath(HOME_TEST_DIR)
208  
-    home_dir = path.get_home_dir()
209  
-    nt.assert_equal(home_dir, abspath(HOME_TEST_DIR))
  167
+    """raise HomeDirError if $HOME is specified, but not a writable dir"""
  168
+    env['HOME'] = abspath(HOME_TEST_DIR+'garbage')
  169
+    # set os.name = posix, to prevent My Documents fallback on Windows
  170
+    os.name = 'posix'
  171
+    nt.assert_raises(path.HomeDirError, path.get_home_dir, True)
210 172
 
211 173
 
212 174
 # Should we stub wreg fully so we can run the test on all platforms?
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.