proposal: os: add UserConfigDir

[mvdan]

Go 1.11 added os.UserCacheDir in #22536, and 1.12 is adding os.UserHomeDir in #26463. This is great, and it covers plenty of common use cases.

UserCacheDir is particularly interesting, as its behavior changes greatly between platforms. For example, on Unix-y systems it tries $XDG_CACHE_HOME and falls back to $HOME/.cache, and on Windows it uses %LocalAppData%.

With those two functions, I believe there's only one piece missing; a configuration directory. Too many applications write "dot files" under the user's home directory, and that's bad practice. Some others hard-code $HOME/.config, which is only marginally better as it doesn't do the right thing on Windows/Mac.

Relying on XDG base directory vars like $XDG_CONFIG_HOME is no good either, as those don't exist on most systems.

We're only left with third-party libraries like https://github.com/shibukawa/configdir, which are moderately popular. In practice, popular software tends to write their own implementations. Below are some examples:

• https://github.com/uber-archive/gg/blob/77ecd52e4c4234a760ab90f3d977912ce6fae7d2/internal/gg/driver.go#L82
• https://github.com/peco/peco/blob/9542898d4c12b61cb2773a35193f7048bc3de154/config.go#L175
• https://github.com/helm/helm/pull/5166/files
• https://github.com/xetys/hetzner-kube/blob/0c180caa561414088872a9e33d2bad3c77aca760/cmd/root.go#L71
• Move .dockercfg to $XDG_CONFIG_HOME (a.k.a. ~/.config) moby/moby#20693
• Use xdg basedir spec on linux hashicorp/terraform#15389
• XDG Base Directory Specification rackspace/rack#338
• https://github.com/keybase/client/blob/b01c1c7aec87fe33fd1c332dcb24e00e1bde0a67/go/libkb/home.go#L81

I propose adding func UserConfigDir() (string, error), which would return:

• $XDG_CONFIG_HOME or $HOME/.config on Unix-y systems
• %APPDATA% on Windows, which is described as "Contains the full path to the Application Data directory of the logged-in user" (unlike %LOCALAPPDATA%, it's not for temporary files)
• $HOME/Library/Preferences on Mac

I don't know about plan9, but given that the cache dir is $home/lib/cache, perhaps $home/lib/config would work.

Please note that I'm not suggesting adding support for XDG base directories into the os package. Most notably, adding UserConfigDir might lead to some users requesting UserDataDir. However, I can think of multiple reasons why they're not equally important:

• Mac and Windows don't obey XDG, so they only have one persistent per-user directory
• Even on Linux, most apps store all their data in their config dir. For example: Chromium, Libreoffice, GIMP, and gcloud all seem to ignore XDG_DATA_HOME
• Most programs only need one persistent per-user directory, and it's often for config files; see the links above for examples

The only disadvantage I see with adding this API is that one could say we'd be encouraging users to store data files into XDG_CONFIG_HOME instead of XDG_DATA_HOME on XDG systems. However, those few programs that wish to differentiate their data directory from their config directory could simply do:

func userDataDir() (string, error)
    if dir := os.Getenv("XDG_DATA_HOME"); dir != "" {
        return dir, nil
    }
    // fall back to something sane on all platforms
    return os.UserConfigDir()
}

/cc @kolyshkin @bradfitz @rsc @ianlancetaylor @stapelberg @robpike

[mvdan]

Forgot to mention: happy to work on this myself early in the 1.13 cycle if the proposal gets accepted.

[robpike]

The directory on Plan 9 is $home/lib.

[mvdan]

I was referring to how UserCacheDir uses $home/lib/cache :)

[rsc]

@mvdan, yes but Rob's point is that the config dir is just $home/lib (not $home/lib/config). There's not an obvious cache dir on Plan 9 and I abused $home/lib by making a cache subdirectory.

Anyway...

We chatted about this at proposal review, and UserConfigDir seems fine. None of us can explain what UserDataDir would mean or be appropriate for vs $HOME, so let's be sure not to do that one. :-)

Marking approved for UserConfigDir.

[mvdan]

There's not an obvious cache dir on Plan 9 and I abused $home/lib by making a cache subdirectory.

Ah, I understand now. I guess we could always use $home/lib here; I doubt any programs would break if the cache directory is inside the data directory. But we can discuss that in the CL.

so let's be sure not to do that one

I personally tend to not find ~/.local/share useful on Linux, so I tend to agree that we shouldn't open that can of worms :)

I'll send a CL shortly, so that it can be reviewed early during the cycle.

[gopherbot]

Change https://golang.org/cl/160877 mentions this issue: os: add UserConfigDir

[rsc]

Ah, I understand now. I guess we could always use $home/lib here; I doubt any programs would break if the cache directory is inside the data directory. But we can discuss that in the CL.

FWIW, there's not much to discuss. The idea of a user config dir is well-defined on Plan 9 and is $home/lib. Yes, I stuck a go build cache in the user config dir. My bad. But that doesn't change what UserConfigDir should return on Plan 9.

[zombiezen]

One important detail of the XDG directory spec is that it permits a list of other search directories specified in XDG_CONFIG_DIRS. It would be a shame to introduce an API that does not permit multiple search paths, since this would be only a partial implementation of the base directory spec.

There is precedence for having multiple search locations on non-Linux platforms, too (e.g. /Library/Preferences on Mac).

[mvdan]

@zombiezen note that we're not implementing XDG, and that the existing UserCacheDir doesn't use directory lists either.

This is not to say they'll never be supported, but this proposal is simply about mirroring UserCacheDir with its config counterpart.

[zombiezen]

I understand, but I think the situation is slightly different. Cache directories are typically written to by the application, where as configuration directories are typically read from by the application. As such, it seems reasonable to me to surface the directories as a sequence of paths instead of just a single path.

[mvdan]

Sure, feel free to file a separate issue or proposal once the current proposal has been implemented. If we were to follow XDG more closely, I presume we'd want to do so consistently, so the change doesn't belong with this proposal.

I also presume there'll be some discussion for and against the feature, and I don't think this accepted proposal is the right place.

[zombiezen]

I'm not against this feature (apologies if my remarks came across that way). I wrote the go4.org/xdgdir package and would have loved to have had something like this proposal in the standard library.

My concern here is that if this function only returns one directory that most Go applications won't bother to call a second function to read the other directories, and thus continue to propagate a common issue in XDG support I've seen (e.g. Git's support) where they only consult a single directory. This would also affect non-Linux OSes like macOS which should also read from /Library/Preferences (and presumably other OSes, but I don't know the details for them).

@mvdan, how would you envision multiple configuration directories being surfaced?

[mvdan]

@zombiezen do Mac users generally obey XDG? If not, how can Mac have multiple config dirs? In my current CL, all platforms use exactly one directory, and as far as I'm aware, that's correct (ignoring XDG_CONFIG_DIRS).

I envision multiple configuration directories not to be surfaced. Surfacing multiple directories would make the API more complex, and it would break consistency with UserCacheDir. Moreover, like I said above, we're not implementing XDG simply because it's not portable. For example, we'll likely never have a UserDataDir because that doesn't exist on Windows nor Mac.

You can argue that this is encouraging developers to not implement XDG properly, and I disagree; see the last part of my original post. For example, one could imagine a more complex wrapper to better support XDG, like:

func UserConfigPath(name string) (string, error)
        // This may not implement XDG dirs perfectly, it's just an example.
        if list := os.Getenv("XDG_CONFIG_DIRS"); list != "" {
                for _, dir := range filepath.SplitList(list) {
                        if path := filepath.Join(dir, name); exists(path) {
                                return path, nil
                        }
                }
        }
        // fall back to something sane on all platforms
        dir, err os.UserConfigDir()
        if err != nil {
                return "", err
        }
        return filepath.Join(dir, name)
}

But at the end of the day, Go is cross-platform, and XDG isn't. And most users and developers want a directory, not a file. I personally think uncluttering $HOME is a great cause, which I'm trying to help with this proposal, but I also think that trying to push for full XDG compliance in all programs is a losing battle. Those that want to go the extra mile can add a bit of extra code, or use a third-party XDG package.

That being said, I'm happy to add a bit of context to the func's godoc. A short note could point out that, on systems supporting XDG like Linux, the data directory is usually separate from the config directory, and that search directories might also be configured by the user.

[zombiezen]

I'm specifically referring to the Library folder conventions on iOS/macOS. See the NSSearchPathForDirectoriesInDomain docs for more details (which is the canonical way on macOS Obj-C programs of obtaining the Library directories, which are usually at ~/Library, /Library, and /System/Library). My point is not that XDG is portable; my point is that searching multiple paths for configuration is a common need for well-behaved programs across systems.

[bradfitz]

This is an API to find the place to write your own files, though, not where to read other programs' files.

So we don't need to search.

As a bad analogy, GOPATH is also a list, and we use the list for searching, but when you use go get, we write to GOPATH[0]. Being an API for writing, we only need to return the first element on all operating systems or schemes (mac, XDG).

[zombiezen]

@bradfitz That seems reasonable. For context: I discovered this issue when reviewing #30411, which I think should be reading from other directories. I'll voice my concern over on the other issue.

Thanks @mvdan for the discussion. I think I understand your perspective better now.

[perillo]

I'm not sure if it is the right decision to add UserHomeDir, UserConfigDir, UserCacheDir and so on to the os package. They should be defined in a separate package, maybe in the golang.org/x namespace.

One of the reason is documentation, as an example as it is done by https://github.com/shibukawa/configdir.

One other reason is that the base directories should be configurable. On UNIX systems that support XDG Base directory configuration is done by setting the XDG_ environment variables, but with a portable API an interface should probably be used, e.g.

type BaseDir interface {
   DataDir() (string, error)
   ConfigDir() (string, error)
   StateDir() (string, error)
   CacheDir() (string, error)
   RuntimeDir() (string, error)
}

[rsc]

We could make this arbitrarily more complex. Or we could add os.UserConfigDir. The latter is what we decided to do. (Note in particular we explicitly rejected UserDataDir above. The point is not to implement XDG in full generality. The point is to provide a useful cross-platform functionality that happens to make reference to XDG on some Unix-like systems.)

[perillo]

I was not proposing to implement XDG, since XDG is UNIX specific and not designed for other operating systems. I mentioned XDG since it is the only pseudo standard that I know that tries to solve the problem of where to store application data.

Thanks.

[jimmyfrasche]

If a user has two or more valid config directories on their platform and manually adds or moves a config file to one that Go does not look at, they would not know why it did not work because they're following the convention of their platform.

If they figure out what's going on, they will consider it a bug in the program invoking UserConfigDir. They're not wrong.

UserConfigDir is 100% correct for finding the directory to write to—but incorrect for finding the directories to read from.

A separate func ConfigDirs() ([]string, error) that includes things like /etc and the like would be needed as well.

That's as much of an argument against UserConfigDir as it for ConfigDirs.

[4ad]

I don't know how I feel about $HOME/Library/Preferences on macOS. It might be standard location for macOS GUI apps, but it most certainly isn't the "normal" location for Unix command line programs that happen to run on macOS. This includes software shipped by Apple as well.

It would be very unexpected for me if the software that I develop on Unix, for Unix (no macOS-specific functionality) would behave this differently between other Unix-like systems and macOS.

$HOME/Library is hidden by default because with GUI apps it's expected that you use the GUI to manage it. In fact there are usually binary files in $HOME/Library/Preferences. With non GUI Unix programs it's more usual use edit configuration files in a text editor. It's pretty weird that this location would be invisible by default and rather hard to get to.

[mvdan]

@4ad my experience with Mac is very limited, so I was only following what I could find online. Do you have a better suggestion? I presume looking at XDG (i.e. a default of $HOME/.config) wouldn't be ideal either.

[4ad]

I don't know what to suggest. For better or worse, "real" mac apps and "unix apps" (for the lack of better terms) behave very differently on macOS, and people can and should be able to use Go to write both (albeit, I'd expect very few people write GUI mac apps in Go...).

[perillo]

If UserConfigDir is being added since it is need by the go tool (go env -w), then it should use the directory used by "unix apps".

But, IMHO, as I wrote in a previous comment, base directories are not trivial to define and there is a need of a separate package if you want the API to be useful for all kind of Go applications.

[deanishe]

Could that be because CLI apps tend to assume XDG paths on all Unix systems, including Mac?

That's exactly why. CLI programs tend to be UNIX programs, so they follow UNIX conventions (~/.config or ~/.myprogramrc).

Native Mac apps are supposed to ask the system where to put their stuff (so they get the correct path whether they're sandboxed or not), and the default is ~/Library/Application Support. Cross-platform apps typically emulate this, but there are also GUI apps with a Linux heritage that use ~/.config instead (and CLI programs that use ~/Library/Application Support).

UserCacheDir uses ~/Library/Caches as opposed to ~/.cache, so I think we should be consistent and use ~/Library/Application Support here

Sounds perfectly reasonable. It's certainly important to use ~/Library/Caches for UserCacheDir, imo. Native tools know how to handle that directory, so backup apps won't upload it to the cloud and system-maintenance tools recognise it as containing data that can be deleted to free up space. The same isn't necessarily true for ~/.cache.

[philoserf]

For the record:

In reference to macOS.

Since the ~/Library is by default hidden on Mac, the XDG should be used by command line applications leaving the ~/Library to native macOS applications .

[mvdan]

@philoserf that ship has sailed, in the sense that UserCacheDir already shipped one or two Go releases ago.

We also need to pick one directory. We ended up going for the directories that Apple documents well. If a program doesn't like that choice, they can easily override that with XDG or another standard on Mac.

[jimdoescode]

Why on Mac (Darwin) can't UserConfigDir behave similarly to Unix? Meaning, first check to see if $XDG_CONFIG_HOME is set, if so, then use it, otherwise use ~/Library/Application Support.

Admittedly most mac users probably don't have the XDG environment variables set up on their machines but those that do would probably be surprised that UserConfigDir ignores them entirely in favor of ~/Library/Application Support. I know I was :\

[mvdan]

@jimdoescode see #29960 (comment). If you want to follow XDG, you should use an XDG library. We can't change the behavior now even if we wanted to, anyway.

[mattn]

On non-English area, the name is localized. For example, on my environment(ja_JP.UTF-8):

$ ls -1
テンプレート
デスクトップ
ドキュメント
ビデオ
ピクチャ
ミュージック
公開

$ cat ~/.config/user-dirs.dirs
# This file is written by xdg-user-dirs-update
# If you want to change or add directories, just edit the line you're
# interested in. All local changes will be retained on the next run
# Format is XDG_xxx_DIR="$HOME/yyy", where yyy is a shell-escaped
# homedir-relative path, or XDG_xxx_DIR="/yyy", where /yyy is an
# absolute path. No other format is supported.
#
XDG_DESKTOP_DIR="$HOME/デスクトップ"
XDG_DOWNLOAD_DIR="$HOME/ダウンロード"
XDG_TEMPLATES_DIR="$HOME/テンプレート"
XDG_PUBLICSHARE_DIR="$HOME/公開"
XDG_DOCUMENTS_DIR="$HOME/ドキュメント"
XDG_MUSIC_DIR="$HOME/ミュージック"
XDG_PICTURES_DIR="$HOME/ピクチャ"
XDG_VIDEOS_DIR="$HOME/ビデオ"

$ cat /etc/xdg/user-dirs.defaults
# Default settings for user directories
#
# The values are relative pathnames from the home directory and
# will be translated on a per-path-element basis into the users locale
DESKTOP=Desktop
DOWNLOAD=Downloads
TEMPLATES=Templates
PUBLICSHARE=Public
DOCUMENTS=Documents
MUSIC=Music
PICTURES=Pictures
VIDEOS=Videos
# Another alternative is:
#MUSIC=Documents/Music
#PICTURES=Documents/Pictures
#VIDEOS=Documents/Videos

So if that API(UserConfigPath) returns a non-localized text and accesses it, it should fail.

[mvdan]

@mattn if you're saying that there is a bug in UserConfigDir, please open a new issue.

[mattn]

@mvdan Not a bug. Just for the note. If Go team will add new APIs to return specific path to the directory (ex XDG_DESKTOP_DIR), it have to look ~/.config/user-dirs.dirs properly.

[mvdan]

Ah, gotcha. Yeah, proper XDG support was never part of the proposal :)

[jimdoescode]

@mvdan thanks for the response. To be clear I was only proposing that the UserConfigDir behavior change if the $XDG_CONFIG_HOME variable is set. I'd wager that the vast majority of Mac users don't have it set and those that do have enough understanding of their system to handle such a change.

But I do understand the concern that existing systems might break as a result and I certainly can't speak for all Mac users on this.

[mvdan]

Yeah, existing users of this API could break, and we can't do that. There has been an extensive discussion about what to do on Mac on this issue, and we decided to go with what Apple documents well. I don't think doing a mix of Apple and XDG would have been a good idea, because it could have been pretty confusing for the end user, and it would have violated both Apple docs and the XDG spec.

[deanishe]

There is no one "right" directory for macOS, imo.

Expected behaviour (from a user's perspective) is generally that command-line programs will behave in a UNIX-like way and put their stuff in ~/.config/appname, while GUI apps will do the Cocoa thing and put their stuff in ~/Library/Application Support/AppName.

Given this dichotomy that the stdlib can't solve, I think the right solution is to return the platform's recommended location (i.e. ~/Library/Application Support), but for developers to follow UNIX customs if that is more appropriate for their software.

[jimdoescode]

@deanishe I agree there's not really a "right" place and the UserConfigDir documentation clearly defines what on a Mac it returns so expected behavior should be what's documented.

It was just disappointing to see that I had no control over it like a UNIX user might. Especially since I have taken the time to set up XDG and attempted, as best I could, to clean up both the Application Support directory and my home directory.

I guess making XDG PRs for the go apps that I use regularly is my current path forward.

I'm sorry for being late to the boat on this one and I appreciate the discussion and consideration you all put into this.

[deanishe]

I guess making XDG PRs for the go apps that I use regularly is my current path forward.

Seems the best solution. An XDG library might be useful, but full XDG support is undesirable on macOS, imo.

It's better to use ~/Library/Caches because the OS and system maintenance & backup tools always recognise the directory as being a cache, which is not necessarily true of ~/.cache.

FWIW, I always use ~/.config/appname on macOS for CLI programs.