Standardize our Paths

[dstufft]

We should follow system conventions for where we look for config files, cache files, log files, etc.

[Ivoz]

Proposals: (WIP, please edit / comment)

Configuration

Windows:

• C:\ProgramData\Pip\pip.conf
• %HOME%\pip\pip.ini (old)
• %HOME%\AppData\Roaming\Pip\pip.conf (new default)

Nix:

• /etc/pip.conf
• ~/.pip/pip.conf (old)
• $XDG_CONFIG_HOME/pip/pip.conf (new default)

Cache

Windows:

Nix:

• $XDG_CACHE_HOME/pip/

[qwcode]

for historical reference #582 (XDG compliance for linux) was temporarily merged 2 years ago (but reverted right before release) because the migration logic it included was considered too dangerous. so think about migration when considering this.

[pfmoore]

To confirm, this is replacing the current single file /pip/pip.ini (/.pip/pip.ini on Unix)?

1. I assume that on Windows you'd actually use the OS APIs to get the specific values for the 3 locations (if only because the actual location of ProgramData is OS-version specific). That will require SHGetSpecialFolderPath (and hence ctypes) to do so, as only ~ expansion is available in core Python.
2. You need to clarify how options override each other if specified in multiple places. For example, with --find-links suppose a site-specific value was added in /etc/pip.ini. How do I (as a user) add to that in ~/.pip/pip.ini. Conversely, how do I remove the one specified in /etc/pip.ini if I need to?
3. Is ~/pip/pip.ini purely for backward compatibility? That makes getting the merging rules right more complex - it's not clear to me whether the "legacy" or the "new" user file should take precedence...
4. You don't want C:\Users\<user>\AppData\Pip.You need to choose one of the local or roaming appdata locations. The roaming profile is more common, so that would be C:\Users\<user>\AppData\Roaming\Pip (or CSIDL_APPDATA, to be precise).

Actually, the location logic is a bit messy to get right in a cross-platform way. Rather than do it for pip and no-one else benefit, maybe we should create a separate "stdpaths" module on PyPI and then vendor it in pip. I'd be happy to provide the Windows knowledge for something like this. (I seem to remember that there's a module around somewhere that tries to do this, but it requires pywin32 and it seems fairly unmaintained - I never got a response to a patch I submitted to use ctypes - so starting from scratch may be better than using that).

[dstufft]

Creating a stdpaths module sounds like a great idea to me. I think this is something that a lot of projects get "wrong" (as we currently do) especially in the Windows sense.

[dstufft]

I started https://github.com/pypa/stdpaths and set it up in RTD/PyPI/Travis-CI. It doesn't have any actual code and I have some ideas for what it should look like. However I need to run out for a little bit so I had to stop.

[pfmoore]

@dstufft cool. I'll keep an eye on it and add PRs as appropriate. May do a PR straight up for some code to get the Windows standard locations via ctypes. That might be useful to expose directly, or just to use internally, we can thrash that out.

[dstufft]

Awesome. I'm researching XDG which is the "standard" on Linux.

Maybe @glyph could advise for how to handle on OSX.

On Apr 19, 2014, at 5:10 PM, Paul Moore notifications@github.com wrote:

@dstufft cool. I'll keep an eye on it and add PRs as appropriate. May do a PR straight up for some code to get the Windows standard locations via ctypes. That might be useful to expose directly, or just to use internally, we can thrash that out.

—
Reply to this email directly or view it on GitHub.

[Ivoz]

@pfmoore thanks for the tip on Roaming. I'm cursorily knowledgeable of the Windows conf paths, not as much as Linux. For the moment, I'm listing just the best place to put things to use as a reference.

My current idea would be to read each file in order, and simply replac e values present in previous files read with the ones in the newer one. So a priority order; The user's ~/.config/pip/pip.conf would replace any confliciting values in /etc/pip.conf.

Lastly, IMHO pip should never use anything requiring ctypes or ffi, so I'd be against using that to find out Windows stuff, even if the resulting method of finding stuff was less 'correct'.

@qwcode TBH I'd do something simpler than that PR - just look for the new config file path, and fall back to the old if it's not there, rather than trying to do any copying.

[glyph]

@dstufft I think it's OK to just use ~/Library/Caches/<your-bundle-id> where a bundle ID is a reversed domain name like a Java package identifier, something like org.pip-installer.pip. I can't find an authoritative source or Foundation framework API that will give you one of those paths, though, so I'm not sure how official that is.

[pfmoore]

@Ivoz yeah, we can thrash out detailed locations as we go.

I'm not sure "replace previous" is actually sufficiently well defined. Think about find-links, which can be specified multiple times in one file. Suppose location 1 says find-links A, B. Then location 2 wants to add C - should it have to specify A and B as well (and so need to be kept up to date as location 1 changes)? Then if location 3 wants to say "actually no, I want no find-links at all, how does it do that? Specifying nothing won't work as it would mean "use what the last location said". Obviously this is ridiculously unlikely to happen, and "Don't do that" is probably the best answer. But we need at a minimum to document the behaviour.

Also, why do you object to ctypes? To get the standard locations on Windows requires either ctypes or a C extension, and I'd have thought a C extension was much worse than ctypes. If your problem is that ctypes has the potential for segfaults and/or may not be present, that is 100% not the case for something that's Windows-only. It's quite possible to write code using ctypes to access the Windows API and it'll be perfectly robust (as far as I'm aware). If you know of something specific that concerns you, let me know.

[glyph]

@pfmoore ctypes definitely has the possibility to cause segfaults.

@Ivoz How about this as a compromise: isolate the ctypes code into a subprocess, provide emergency fallback functions in case that fails or crashes?

[glyph]

Aah! Here we go, official API confirmation, sort of:

>>> from Foundation import NSSearchPathForDirectoriesInDomains,
NSCachesDirectory, NSUserDomainMask
>>> NSSearchPathForDirectoriesInDomains(NSCachesDirectory, NSUserDomainMask,
True)
(
    "/Users/glyph/Library/Caches"
)

Pip could potentially have an optional pyobjc import and just re-implement this in Python.

[dstufft]

Yea anything that requires a C extension is right out, but optional C-exts are ok and if we can do it with ctypes reasonably (as in the Windows case it sounds like) just use that.

[dstufft]

Oh hey, look what I found https://github.com/ActiveState/appdirs

[dstufft]

So I've looked at appdirs and figured out if we use it, what the paths will be on the various systems. In the below lists *nix basically means any non Windows or OSX machines. All of the Windows paths are for Vista/Win7 and the Windows XP paths are whatever the Vista/Win7 paths would correlate too.

So I guess the question is, do these paths look accurate? The XDG ones look reasonable to me, as does the OSX ones (but I'm not real sure about. @glyph said bundle id should be a reverse domain name, whereas AppDir is using "app name". Though we could pass in org.pip-installer.pip as our app name if we should be using that instead of pip.

The Windows ones I have no idea on, they look reasonable to me but I have no idea about the distinctions between Local and Roaming and ProgramData and such, and there's also the sticking point that the location for the machine configuration file on Windows is apparently broken on Vista. Hopefully @pfmoore can give some guidance here!

It's also important to note that appdirs will expand the environment varaibles out for XDG, however it is just using os.expanduser("~/Library/Application Support/") and such hardcoded for OSX. For Windows it attempts to use pywin32 if it is available, then it falls back to using ctypes, if an ImportError occurs there it falls back to querying the registry. It specifically queries the values for CSIDL_APPDATA, CSIDL_COMMON_APPDATA, and CSIDL_LOCAL_APPDATA and then applies some opinions on top of those.

Machine Specific Configuration File

• *nix: $XDG_CONFIG_DIRS/pip/pip.conf
  • This will actually support multiple locations, all specified inside of $XDG_CONFIG_DIRS
• OSX: /Library/Application Support/pip/pip.conf
• Win: C:/ProgramData/PyPA/pip/pip.conf
  • Note, this directory is a hidden system folder on Vista, but a writeable, but hidden folder on Win7. I'm not sure what that means but appdirs says not to use this on Windows. Does a "Machine Specific" configuration file make sense on Windows?

User Specific Configuration File

• *nix: $XDG_CONFIG_HOME/pip/pip.conf
• OSX: $HOME/Library/Application Support/pip/pip.conf
• Win: %APPDATA%/Roaming/PyPA/pip/pip.conf

User Specific Cache Directory

• *nix: $XDG_CACHE_HOME/pip/{download,package}
• OSX: $HOME/Library/Caches/pip/{download,package}
• Win: %APPDATA%/Local/PyPA/pip/Cache/{download,package}

User Specific Log Directory

• *nix: $XDG_CACHE_HOME/pip/log
• OSX: $HOME/Library/Logs/pip
• Win: %AppData%/Local/PyPA/pip/Logs

XDG Default Values

On *nix the directories are configurable using environment variables, these environment variables have defaults which are:

• $XDG_CONFIG_DIRS = /etc/xdg
• $XDG_CONFIG_HOME = $HOME/.config
• $XDG_CACHE_HOME = $HOME/.cache

[dstufft]

Forgot to mention, the above post is aiming to figure out if we can just reuse appdirs instead of writing our own! So hopefully we can audit those values and determine if they look correct for us.

[glyph]

Can you just always specify the "appname" as "org.pip-installer.pip"? I know that this is less of a convention on other platforms, but it seems more "polite" to me, at least, to use a completely unambiguous name that isn't likely to collide with anything. (For example, pip pip pip pip etc)

[dstufft]

Well, we could. But I kind of feel like that makes editing the user configuration file kind of annoying? I'm not sure it makes sense at all on Windows since that is keyed by Vendor + app name (so it's PyPA/pip. But honestly the thing that annoys me the most about that is editing a ~/.config/org/pip-installer.pip/pip.conf seems real annoying to me.

Is that domain thing even really a convention on OSX? My ~/Library/Application Support only has official Apple apps and VLC that use it (and not even all Official Apple Apps!) (See: http://cl.ly/image/230I2h061M0h). It appears Apple themselves use Appname, Apple/Appname, and com.apple.appname.

[dstufft]

Perhaps stealing from Windows here, it makes sense to do PyPA/pip instead of just pip?

[glyph]

Is that domain thing even really a convention on OSX?

Remember that this is a platform with at least a 26-year history at this point. Some things don't use the latest conventions ;-).

But I kind of feel like that makes editing the user configuration file kind of annoying? ... But honestly the thing that annoys me the most about that is editing a ~/.config/org/pip-installer.pip/pip.conf

So, first of all, this is easy to fix.

Here are some ideas about how to make this transparent, and more convenient than the current scheme:

$ emacs `pip conf`
$ pip conf edit # runs $EDITOR
$ pip conf set foo=bar
$ # don't type anything just use environment variables why do you even have a config file

Also, maybe consider that if things actually work by default, users won't have to screw around with INI files quite so much? ;-) Consider that every time a user has to edit a config file, there's been some kind of failure.

See, for example, this apparently timeless essay by Havoc Pennington.

[dstufft]

Is that domain thing even really a convention on OSX?
Remember that this is a platform with at least a 26-year history at this point. Some things don't use the latest conventions ;-).

Sure, but I don't think the App Store, iCloud, and AirPort (three random Apple apps which don't use com.apple.appname) are apps with 26 year old histories and thus bogged down in legacy where they can't use the latest conventions!

But I kind of feel like that makes editing the user configuration file kind of annoying? ... But honestly the thing that annoys me the most about that is editing a ~/.config/org/pip-installer.pip/pip.conf

So, first of all, this is easy to fix.

Here are some ideas about how to make this transparent, and more convenient than the current scheme:

$ emacs `pip conf`
$ pip conf edit # runs $EDITOR
$ pip conf set foo=bar

I like this pip conf command, I added a ticket for it (#1736), I think regardless we should add it.

Also, maybe consider that if things actually work by default, users won't have to screw around with INI files quite so much? ;-) Consider that every time a user has to edit a config file, there's been some kind of failure.

See, for example, this apparently timeless essay by Havoc Pennington.

So I agree that we should get to a point where a user needing to edit their config file to get what we consider "best practices" and the most reasonable behaviors should be considered a failure of design rather than the status quo. However I don't think we'll be able to avoid ever opening the configuration file so it's still important to me to not make that annoying (However the pip conf idea I think resolved most of that).

I actually went ahead and registered the domain pypa.io and sent an email to the pypa-dev mailing list about it. Perhaps we can end up with pip.pypa.io, virtualenv.pypa.io, etc which we can then use io.pypa.pip as our app name in that case.

[apollo13]

Perhaps stealing from Windows here, it makes sense to do PyPA/pip instead of just pip?

Qt uses the following format: http://qt-project.org/doc/qt-5/QSettings.html#platform-specific-notes -- in our case this would be PyPA/pip.conf and is imo a good compromise in terms of length etc over PyPA/pip/pip.conf. It also maps nicely to Mac and Windows. Also note that Qt allows you to set OrganizationName and OrganizationDomain which allows it to use correct paths on Mac vs other *nix flavours.

[Ivoz]

I'd prefer /etc/pip.conf instead of /etc/xdg/pip/pip.conf. We really don't need an extra folder for configuration, except where the existing configuration convention has it making more sense than not (e.g in ~/.config). I really don't think we need an extra PyPA folder unless there are definite examples of more pypa apps which would really make sense grouped together.

I really like git's practice for configuration setting; i.e its git config [--global] <option> [<value>] syntax, to get or set a config value.

[pfmoore]

@dstufft %APPDATA% is actually C:\Users\<user>\AppData\Roaming on Win7 - it's what you're referring to as %APPDATA%\Roaming. What you're calling %APPDATA%\Local is actually %LOCALAPPDATA%.

I don't think it's right to use local for some things and roaming for others. I'm not sure I've ever seen an app that does that. I'd probably put them all in %APPDATA% (Roaming) as that's more common. FWIW, "Roaming" means "for this user on any machine" and "Local" means "for this user on this machine only" and there's not that many good use cases for local that I can see (given that roaming accounts are only common in corporate environments, so for many users the distinction is irrelevant anyway).

Program Data and Application Data are both hidden, so there are the same problems with both. In practice, those problems tend to be relatively minimal on Windows 7, but they can be distinctly irritating on earlier versions. Also, earlier versions had spaces in the directory names, which also made for an annoying user experience. (I'm not even sure there was a "program data" location in XP and earlier, so that may be an issue).

Generally, these folders are probably the right thing to use, but they are intended to hold files that are managed by an application, not by user editing (imagine a "preferences" dialog in a GUI app). So there will still be a "user experience" issue to consider, but it's consistent with other applications.

I'd like to see something similar to git config, as it gives users a means to set configuration without having to edit the file directly - so it matches the Windows expectation on how these directories will be used.

BTW, for preciseness, it would be better to talk in terms of CSIDLs on Windows - see http://msdn.microsoft.com/en-us/library/windows/desktop/bb762494(v=vs.85).aspx for details. But that's pretty horrible for non-Windows people, so I'm happy to keep translating :-) I'm assuming we're talking about

CSIDL_APPDATA (FOLDERID_RoamingAppData) - User specific stuff held under PyPA\pip in that location.
CSIDL_COMMON_APPDATA (FOLDERID_ProgramData) - Machine specific stuff (again under PyPA\pip).

I'm not enamoured of the extra PyPA directory - it's the "official" approach on Windows to have a vendor level, but often ignored. If Unix is having that level as well, we should include it for consistency, but if Unix doesn't use the extra level, I don't think it's necessary (or helpful) on Windows.

[pfmoore]

Hmm, after writing all that, I see that the original suggestions were direct from the definitions in ``appdirs`. Sigh. I should have read the original comment more carefully.

Summary - no, I don't like the choices made by appdirs. Enough to advocate writing our own version? I don't know. I'd be happier following appdirs conventions if it were better known and the latest version was newer than 2011 (given that there seems to have been activity on github since then)...

[dstufft]

@pfmoore Whoops, I haven't used Windows in awhile so I just guessed that %APPDATA% was the AppData folder itself. I think the reason that it's using Roaming for Some and Local for others is that the stuff it's using Local for is Cache/Logs. Stuff you don't really want synchronized via Active Directory and that can be "lost" if a Machine needs to. Is it really normal to store Cache files there? Or is there a better location for that?

[dstufft]

FWIW I'm not married to appdirs, the functionality is simple enough that it's not horrible to make our own.

[glyph]

I think that moving to the appdirs tracker would be a good start. Who wants to file an issue? :)

[jezdez]

FTR, I'm -0 on using the XDG spec, it's a super weird standard that only select packages seem to follow, and then only partially at best (e.g. /etc/xdg/ seems to be often ignored).

I can get behind using ~/.config but the "xdg" in /etc/xdg just screams "hostile to end users" to me. So if we can just follow the example of plenty of software out there and use /etc/:/etc/xdg for the global config path I think I can get over it. I hope we'll never have to discuss this again.

[glyph]

@jezdez I would prefer to just follow the standard as-is, flaws and all, because https://xkcd.com/927/ - but your observation is correct, few packages honor /etc/xdg.

[dstufft]

We decided to do both, we'll support /etc/xdg by default, but we'll also support /etc.

[Ivoz]

Given XDG is for X Desktop Group, pip not being a desktop application per se, and global configuration files being stored generally in /etc is already a *nix convention, I'd prefer to see a plain /etc by default. The different between the other suggestions in their standard and this one is that /etc is already a folder specifically for configuration files, and this is also a name stamp (../xdg) of a folder for no apparent reason. Having a .config/ folder in ~ makes perfect sense, /etc is already that folder for the system :/

[glyph]

@Ivoz I don't see what you're asking for. @dstufft already said that /etc/ will be supported, presumably for those reasons.

[r1chardj0n3s]

The only deviation from the above conversation in what I've implemented is that I've not taken @pfmoore's advice and I'm looking for the config in a c:\ProgramData\PyPA\pip\pip.conf since on my system the \ProgramData\ directory only contains vendor subdirectories, and I think it's appropriate for us to use PyPA as our vendor label. I don't have a Win XP system handy, but I'm following the appdirs formula there as well, including PyPA in the path.

[dstufft]

We need to pick one and use it. Right now the cache directory is not using a vendor directory, and your PR does. Whichever way we go we need to be consistent.

[pfmoore]

My PC has applications in C:\ProgramData. Ironically, Skype is one, VS (Visual Studio) is another, both MS owned... So it's certainly not a problem to omit the vendor. The reason I prefer to omit it is that it saves typing when manually editing the file ($env:APPDATA\Pip\pip.conf) And it's not as if PyPA is going to be publishing vast numbers of apps with config file needs, that might otherwise clutter the appdata directories...

But it's basically a bikeshed issue, and I agree with @dstufft, consistency is the key thing. I assume the cache directory is #1748 which is merged, so I'd say go with no vendor label rather than having to revisit that.

[r1chardj0n3s]

okie dokie @pfmoore I'll mod the PR :)

[pfmoore]

To be clear (as noted on IRC) I'm suggesting basically what @Ivoz said up above:

• %ProgramData%\Pip\pip.conf
• %USERPROFILE%\Pip\pip.ini
• %APPDATA%\Pip\pip.conf

So we have Vendor = (omitted) and Application = "Pip".

On my PC, ProgramData is C:\ProgramData, USERPROFILE is C:\Users and APPDATA is %USERPROFILE%\AppData\Roaming.

Note that the download cache (user_cache_dir) uses %LOCALAPPDATA%, not %APPDATA% (%USERPROFILE\AppData\Local). This is deliberate - config wants to be included in the roaming profile, but cache data doesn't.

[pfmoore]

Just noticed that #1978 is only looking at %ProgramData% not at the %APPDATA% bit, so ignore that as far as that PR is concerned...

[r1chardj0n3s]

Thanks @pfmoore I've resubmitted #1978 to fix it based on your advice above.

[dstufft]

I think the only thing left for this particular ticket is the default log file, and the user config.

[dstufft]

This is done now!

[piotr-dobrogost]

@dstufft

This is done now!

Nice, but it would be even nicer to have some link :)
Ok, I think this is it – #2021

[dstufft]

Eh, it's spread over a bunch of PRs, mostly recently #2021.

[grossag]

I am running into build failures on my company's locked-down build machines due to this code, when building Chromium. The traceback is:

2015-10-19 09:44:38 gobuilds.Compile : Traceback (most recent call last):
2015-10-19 09:44:38 gobuilds.Compile : File "bootstrap\bootstrap.py", line 234, in
2015-10-19 09:44:38 gobuilds.Compile : sys.exit(main(sys.argv[1:]))
2015-10-19 09:44:38 gobuilds.Compile : File "bootstrap\bootstrap.py", line 228, in main
2015-10-19 09:44:38 gobuilds.Compile : activate_env(opts.env_path, deps, opts.quiet)
2015-10-19 09:44:38 gobuilds.Compile : File "bootstrap\bootstrap.py", line 197, in activate_env
2015-10-19 09:44:38 gobuilds.Compile : install(deps)
2015-10-19 09:44:38 gobuilds.Compile : File "bootstrap\bootstrap.py", line 145, in install
2015-10-19 09:44:38 gobuilds.Compile : links = get_links(deps)
2015-10-19 09:44:38 gobuilds.Compile : File "bootstrap\bootstrap.py", line 76, in get_links
2015-10-19 09:44:38 gobuilds.Compile : import pip.wheel # pylint: disable=E0611
2015-10-19 09:44:38 gobuilds.Compile : File "d:\build\ob\sb-6201034\cayman_cef\cef\src\depot_tools\ENV\Lib\site-packages\pip__init__.py", line 13, in
2015-10-19 09:44:38 gobuilds.Compile : from pip.utils import get_installed_distributions, get_prog
2015-10-19 09:44:38 gobuilds.Compile : File "d:\build\ob\sb-6201034\cayman_cef\cef\src\depot_tools\ENV\Lib\site-packages\pip\utils__init__.py", line 18, in
2015-10-19 09:44:38 gobuilds.Compile : from pip.locations import (
2015-10-19 09:44:38 gobuilds.Compile : File "d:\build\ob\sb-6201034\cayman_cef\cef\src\depot_tools\ENV\Lib\site-packages\pip\locations.py", line 208, in
2015-10-19 09:44:38 gobuilds.Compile : for path in appdirs.site_config_dirs('pip')
2015-10-19 09:44:38 gobuilds.Compile : File "d:\build\ob\sb-6201034\cayman_cef\cef\src\depot_tools\ENV\Lib\site-packages\pip\utils\appdirs.py", line 183, in site_config_dirs
2015-10-19 09:44:38 gobuilds.Compile : path = os.path.normpath(_get_win_folder("CSIDL_COMMON_APPDATA"))
2015-10-19 09:44:38 gobuilds.Compile : File "d:\build\ob\sb-6201034\cayman_cef\cef\src\depot_tools\ENV\Lib\site-packages\pip\utils\appdirs.py", line 230, in _get_win_folder_with_pywin32
2015-10-19 09:44:38 gobuilds.Compile : directory = shell.SHGetFolderPath(0, getattr(shellcon, csidl_name), 0, 0)
2015-10-19 09:44:38 gobuilds.Compile : pywintypes.com_error: (-2147024893, 'The system cannot find the path specified.', None, None)

I have tried plumbing through the various appdata-related environment variables to the environment of the script, as well as confirming that the directories all exist and have rwx permissions. Is there some way that I can set an environment variable to have pip not write any values to local directories, so as to prevent the code from even trying to retrieve CSIDL_COMMON_APPDATA and CSIDL_LOCAL_APPDATA?

[xavfernandez]

@grossag Not sure about your issue (not used to windows), but you could try with latest pip:
the code from your traceback _get_win_folder_with_pywin32 is gone since pip 6.1