Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: os: add UserConfigDir #29960

Closed
mvdan opened this issue Jan 28, 2019 · 33 comments

Comments

Projects
None yet
@mvdan
Copy link
Member

commented Jan 28, 2019

Go 1.11 added os.UserCacheDir in #22536, and 1.12 is adding os.UserHomeDir in #26463. This is great, and it covers plenty of common use cases.

UserCacheDir is particularly interesting, as its behavior changes greatly between platforms. For example, on Unix-y systems it tries $XDG_CACHE_HOME and falls back to $HOME/.cache, and on Windows it uses %LocalAppData%.

With those two functions, I believe there's only one piece missing; a configuration directory. Too many applications write "dot files" under the user's home directory, and that's bad practice. Some others hard-code $HOME/.config, which is only marginally better as it doesn't do the right thing on Windows/Mac.

Relying on XDG base directory vars like $XDG_CONFIG_HOME is no good either, as those don't exist on most systems.

We're only left with third-party libraries like https://github.com/shibukawa/configdir, which are moderately popular. In practice, popular software tends to write their own implementations. Below are some examples:

I propose adding func UserConfigDir() (string, error), which would return:

  • $XDG_CONFIG_HOME or $HOME/.config on Unix-y systems
  • %APPDATA% on Windows, which is described as "Contains the full path to the Application Data directory of the logged-in user" (unlike %LOCALAPPDATA%, it's not for temporary files)
  • $HOME/Library/Preferences on Mac

I don't know about plan9, but given that the cache dir is $home/lib/cache, perhaps $home/lib/config would work.

Please note that I'm not suggesting adding support for XDG base directories into the os package. Most notably, adding UserConfigDir might lead to some users requesting UserDataDir. However, I can think of multiple reasons why they're not equally important:

  • Mac and Windows don't obey XDG, so they only have one persistent per-user directory
  • Even on Linux, most apps store all their data in their config dir. For example: Chromium, Libreoffice, GIMP, and gcloud all seem to ignore XDG_DATA_HOME
  • Most programs only need one persistent per-user directory, and it's often for config files; see the links above for examples

The only disadvantage I see with adding this API is that one could say we'd be encouraging users to store data files into XDG_CONFIG_HOME instead of XDG_DATA_HOME on XDG systems. However, those few programs that wish to differentiate their data directory from their config directory could simply do:

func userDataDir() (string, error)
    if dir := os.Getenv("XDG_DATA_HOME"); dir != "" {
        return dir, nil
    }
    // fall back to something sane on all platforms
    return os.UserConfigDir()
}

/cc @kolyshkin @bradfitz @rsc @ianlancetaylor @stapelberg @robpike

@mvdan mvdan added the Proposal label Jan 28, 2019

@mvdan mvdan added this to the Proposal milestone Jan 28, 2019

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Jan 28, 2019

Forgot to mention: happy to work on this myself early in the 1.13 cycle if the proposal gets accepted.

@robpike

This comment has been minimized.

Copy link
Contributor

commented Jan 28, 2019

The directory on Plan 9 is $home/lib.

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Jan 28, 2019

I was referring to how UserCacheDir uses $home/lib/cache :)

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 30, 2019

@mvdan, yes but Rob's point is that the config dir is just $home/lib (not $home/lib/config). There's not an obvious cache dir on Plan 9 and I abused $home/lib by making a cache subdirectory.

Anyway...

We chatted about this at proposal review, and UserConfigDir seems fine. None of us can explain what UserDataDir would mean or be appropriate for vs $HOME, so let's be sure not to do that one. :-)

Marking approved for UserConfigDir.

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Jan 30, 2019

There's not an obvious cache dir on Plan 9 and I abused $home/lib by making a cache subdirectory.

Ah, I understand now. I guess we could always use $home/lib here; I doubt any programs would break if the cache directory is inside the data directory. But we can discuss that in the CL.

so let's be sure not to do that one

I personally tend to not find ~/.local/share useful on Linux, so I tend to agree that we shouldn't open that can of worms :)

I'll send a CL shortly, so that it can be reviewed early during the cycle.

@mvdan mvdan self-assigned this Jan 30, 2019

@gopherbot

This comment has been minimized.

Copy link

commented Feb 2, 2019

Change https://golang.org/cl/160877 mentions this issue: os: add UserConfigDir

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2019

Ah, I understand now. I guess we could always use $home/lib here; I doubt any programs would break if the cache directory is inside the data directory. But we can discuss that in the CL.

FWIW, there's not much to discuss. The idea of a user config dir is well-defined on Plan 9 and is $home/lib. Yes, I stuck a go build cache in the user config dir. My bad. But that doesn't change what UserConfigDir should return on Plan 9.

@zombiezen

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2019

One important detail of the XDG directory spec is that it permits a list of other search directories specified in XDG_CONFIG_DIRS. It would be a shame to introduce an API that does not permit multiple search paths, since this would be only a partial implementation of the base directory spec.

There is precedence for having multiple search locations on non-Linux platforms, too (e.g. /Library/Preferences on Mac).

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Feb 27, 2019

@zombiezen note that we're not implementing XDG, and that the existing UserCacheDir doesn't use directory lists either.

This is not to say they'll never be supported, but this proposal is simply about mirroring UserCacheDir with its config counterpart.

@zombiezen

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2019

I understand, but I think the situation is slightly different. Cache directories are typically written to by the application, where as configuration directories are typically read from by the application. As such, it seems reasonable to me to surface the directories as a sequence of paths instead of just a single path.

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Feb 27, 2019

Sure, feel free to file a separate issue or proposal once the current proposal has been implemented. If we were to follow XDG more closely, I presume we'd want to do so consistently, so the change doesn't belong with this proposal.

I also presume there'll be some discussion for and against the feature, and I don't think this accepted proposal is the right place.

@zombiezen

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2019

I'm not against this feature (apologies if my remarks came across that way). I wrote the go4.org/xdgdir package and would have loved to have had something like this proposal in the standard library.

My concern here is that if this function only returns one directory that most Go applications won't bother to call a second function to read the other directories, and thus continue to propagate a common issue in XDG support I've seen (e.g. Git's support) where they only consult a single directory. This would also affect non-Linux OSes like macOS which should also read from /Library/Preferences (and presumably other OSes, but I don't know the details for them).

@mvdan, how would you envision multiple configuration directories being surfaced?

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Mar 1, 2019

@zombiezen do Mac users generally obey XDG? If not, how can Mac have multiple config dirs? In my current CL, all platforms use exactly one directory, and as far as I'm aware, that's correct (ignoring XDG_CONFIG_DIRS).

I envision multiple configuration directories not to be surfaced. Surfacing multiple directories would make the API more complex, and it would break consistency with UserCacheDir. Moreover, like I said above, we're not implementing XDG simply because it's not portable. For example, we'll likely never have a UserDataDir because that doesn't exist on Windows nor Mac.

You can argue that this is encouraging developers to not implement XDG properly, and I disagree; see the last part of my original post. For example, one could imagine a more complex wrapper to better support XDG, like:

func UserConfigPath(name string) (string, error)
        // This may not implement XDG dirs perfectly, it's just an example.
        if list := os.Getenv("XDG_CONFIG_DIRS"); list != "" {
                for _, dir := range filepath.SplitList(list) {
                        if path := filepath.Join(dir, name); exists(path) {
                                return path, nil
                        }
                }
        }
        // fall back to something sane on all platforms
        dir, err os.UserConfigDir()
        if err != nil {
                return "", err
        }
        return filepath.Join(dir, name)
}

But at the end of the day, Go is cross-platform, and XDG isn't. And most users and developers want a directory, not a file. I personally think uncluttering $HOME is a great cause, which I'm trying to help with this proposal, but I also think that trying to push for full XDG compliance in all programs is a losing battle. Those that want to go the extra mile can add a bit of extra code, or use a third-party XDG package.

That being said, I'm happy to add a bit of context to the func's godoc. A short note could point out that, on systems supporting XDG like Linux, the data directory is usually separate from the config directory, and that search directories might also be configured by the user.

@zombiezen

This comment has been minimized.

Copy link
Contributor

commented Mar 1, 2019

I'm specifically referring to the Library folder conventions on iOS/macOS. See the NSSearchPathForDirectoriesInDomain docs for more details (which is the canonical way on macOS Obj-C programs of obtaining the Library directories, which are usually at ~/Library, /Library, and /System/Library). My point is not that XDG is portable; my point is that searching multiple paths for configuration is a common need for well-behaved programs across systems.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Mar 1, 2019

This is an API to find the place to write your own files, though, not where to read other programs' files.

So we don't need to search.

As a bad analogy, GOPATH is also a list, and we use the list for searching, but when you use go get, we write to GOPATH[0]. Being an API for writing, we only need to return the first element on all operating systems or schemes (mac, XDG).

@zombiezen

This comment has been minimized.

Copy link
Contributor

commented Mar 1, 2019

@bradfitz That seems reasonable. For context: I discovered this issue when reviewing #30411, which I think should be reading from other directories. I'll voice my concern over on the other issue.

Thanks @mvdan for the discussion. I think I understand your perspective better now.

@perillo

This comment has been minimized.

Copy link

commented Mar 1, 2019

I'm not sure if it is the right decision to add UserHomeDir, UserConfigDir, UserCacheDir and so on to the os package. They should be defined in a separate package, maybe in the golang.org/x namespace.

One of the reason is documentation, as an example as it is done by https://github.com/shibukawa/configdir.

One other reason is that the base directories should be configurable. On UNIX systems that support XDG Base directory configuration is done by setting the XDG_ environment variables, but with a portable API an interface should probably be used, e.g.

type BaseDir interface {
   DataDir() (string, error)
   ConfigDir() (string, error)
   StateDir() (string, error)
   CacheDir() (string, error)
   RuntimeDir() (string, error)
}
@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 1, 2019

We could make this arbitrarily more complex. Or we could add os.UserConfigDir. The latter is what we decided to do. (Note in particular we explicitly rejected UserDataDir above. The point is not to implement XDG in full generality. The point is to provide a useful cross-platform functionality that happens to make reference to XDG on some Unix-like systems.)

@perillo

This comment has been minimized.

Copy link

commented Mar 1, 2019

I was not proposing to implement XDG, since XDG is UNIX specific and not designed for other operating systems. I mentioned XDG since it is the only pseudo standard that I know that tries to solve the problem of where to store application data.

Thanks.

@jimmyfrasche

This comment has been minimized.

Copy link
Member

commented Mar 1, 2019

If a user has two or more valid config directories on their platform and manually adds or moves a config file to one that Go does not look at, they would not know why it did not work because they're following the convention of their platform.

If they figure out what's going on, they will consider it a bug in the program invoking UserConfigDir. They're not wrong.

UserConfigDir is 100% correct for finding the directory to write to—but incorrect for finding the directories to read from.

A separate func ConfigDirs() ([]string, error) that includes things like /etc and the like would be needed as well.

That's as much of an argument against UserConfigDir as it for ConfigDirs.

@4ad

This comment has been minimized.

Copy link
Member

commented Mar 3, 2019

I don't know how I feel about $HOME/Library/Preferences on macOS. It might be standard location for macOS GUI apps, but it most certainly isn't the "normal" location for Unix command line programs that happen to run on macOS. This includes software shipped by Apple as well.

It would be very unexpected for me if the software that I develop on Unix, for Unix (no macOS-specific functionality) would behave this differently between other Unix-like systems and macOS.

$HOME/Library is hidden by default because with GUI apps it's expected that you use the GUI to manage it. In fact there are usually binary files in $HOME/Library/Preferences. With non GUI Unix programs it's more usual use edit configuration files in a text editor. It's pretty weird that this location would be invisible by default and rather hard to get to.

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Mar 3, 2019

@4ad my experience with Mac is very limited, so I was only following what I could find online. Do you have a better suggestion? I presume looking at XDG (i.e. a default of $HOME/.config) wouldn't be ideal either.

@4ad

This comment has been minimized.

Copy link
Member

commented Mar 3, 2019

I don't know what to suggest. For better or worse, "real" mac apps and "unix apps" (for the lack of better terms) behave very differently on macOS, and people can and should be able to use Go to write both (albeit, I'd expect very few people write GUI mac apps in Go...).

@perillo

This comment has been minimized.

Copy link

commented Mar 3, 2019

If UserConfigDir is being added since it is need by the go tool (go env -w), then it should use the directory used by "unix apps".

But, IMHO, as I wrote in a previous comment, base directories are not trivial to define and there is a need of a separate package if you want the API to be useful for all kind of Go applications.

@gopherbot gopherbot closed this in ebdc24c Mar 5, 2019

@cuonglm

This comment has been minimized.

Copy link
Contributor

commented Mar 5, 2019

@mvdan The commit message seems to be wrong:

Add UserConfigDir, which is implemented in a similar manner to
UserConfigDir.

AFAICT, should be:

Add UserConfigDir, which is implemented in a similar manner to
UserCacheDir.
@mvdan

This comment has been minimized.

Copy link
Member Author

commented Mar 5, 2019

@Gnouc you're absolutely right - that was a brain fart of mine :) It's already merged though, so not much we can do about it.

@bobheadxi bobheadxi referenced this issue Jun 6, 2019

Open

all: go1.13 #611

0 of 3 tasks complete
@matt0xFF

This comment has been minimized.

Copy link

commented Jun 7, 2019

I saw this comment recently:

All sounds pretty good, with one exception: I think they've got UserConfigDir wrong on macOS.

~/Library/Preferences is for native Mac apps and is basically managed by NSUserDefaults. Anything that isn't a property list doesn't really belong in there. It's more akin to the Windows registry than to ~/.config.

~/.config is much more appropriate for command-line tools, and ~/Library/Application Support for GUI apps. (That's where users will expect such programs' config files/data to be.)

The feature is described:

the only remaining piece which is commonly needed and portable is a per-user directory to store persistent files.

For that broad-ish purpose, ~/Library/Preferences is very inappropriate. Putting a config file in ~/Library/Preferences is one thing, but other application data most certainly don't belong in there.

Two reasonable options, and one bad one. And they chose the bad one :(

This seems consistent with the documentation that @zombiezen found, which mentions Application Support being for among other things "configuration files", and regarding Preferences, "You should not create files in this directory yourself. Instead, use the NSUserDefaults class" (which will write a property list).

It seemed worth mentioning here, especially since there seemed to be uncertainty over what the best Mac directory is and Application Support wasn't previously considered.

@deanishe

This comment has been minimized.

Copy link

commented Jun 7, 2019

Regarding what @matt0xFF says, well-behaved cross-platform applications that keep all their persistent data in one place (Chrome and Firefox being good examples) use ~/Library/Application Support, not ~/Library/Preferences.

For my part, I think ~/.config would also be a sound choice, as that's where command-line programs typically store their data.

I don't think there's a particularly compelling argument to choose ~/.config over ~/Library/Application Support or vice versa: either is a justifiable choice. But ~/Library/Preferences is a poor one. Programs are just not supposed to put their own stuff in there.

Doing so currently doesn't cause any issues, but as the directory is supposed to be used only by NSUserDefaults, it's not hard to imagine Apple locking it down in a future version of macOS.

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Jun 7, 2019

Thank you both for your input. Luckily, we're in time to fix this before the final release.

I don't think there's a particularly compelling argument to choose ~/.config over ~/Library/Application Support or vice versa: either is a justifiable choice.

Is the only distinction that ~/.config is common with CLI apps, and the other for GUI apps?

Could that be because CLI apps tend to assume XDG paths on all Unix systems, including Mac? I'd assume that's wrong, unless Apple somehow endorses or documents the use of ~/.config or XDG.

UserCacheDir uses ~/Library/Caches as opposed to ~/.cache, so I think we should be consistent and use ~/Library/Application Support here. The docs page linked above seems to confirm that it's the right choice.

@gopherbot

This comment has been minimized.

Copy link

commented Jun 7, 2019

Change https://golang.org/cl/181177 mentions this issue: os: fix UserConfigDir on Darwin

@deanishe

This comment has been minimized.

Copy link

commented Jun 7, 2019

Could that be because CLI apps tend to assume XDG paths on all Unix systems, including Mac?

That's exactly why. CLI programs tend to be UNIX programs, so they follow UNIX conventions (~/.config or ~/.myprogramrc).

Native Mac apps are supposed to ask the system where to put their stuff (so they get the correct path whether they're sandboxed or not), and the default is ~/Library/Application Support. Cross-platform apps typically emulate this, but there are also GUI apps with a Linux heritage that use ~/.config instead (and CLI programs that use ~/Library/Application Support).

UserCacheDir uses ~/Library/Caches as opposed to ~/.cache, so I think we should be consistent and use ~/Library/Application Support here

Sounds perfectly reasonable. It's certainly important to use ~/Library/Caches for UserCacheDir, imo. Native tools know how to handle that directory, so backup apps won't upload it to the cloud and system-maintenance tools recognise it as containing data that can be deleted to free up space. The same isn't necessarily true for ~/.cache.

@philoserf

This comment has been minimized.

Copy link

commented Jun 25, 2019

For the record:

In reference to macOS.

Since the ~/Library is by default hidden on Mac, the XDG should be used by command line applications leaving the ~/Library to native macOS applications .

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Jun 25, 2019

@philoserf that ship has sailed, in the sense that UserCacheDir already shipped one or two Go releases ago.

We also need to pick one directory. We ended up going for the directories that Apple documents well. If a program doesn't like that choice, they can easily override that with XDG or another standard on Mac.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.