proposal: os: add application data directory functions (i.e. os.UserDataDir and os.DataDir)

[cowtoolz]

In #22536, #26463, and #29960, the os package was greatly improved by adding a standard way to get a user's standard cache, home, and config directories—missing still is an easy way to get the correct directories for storing both shared and user-specific application data. Something like os.DataDir and os.UserDataDir respectively, mapped to the following locations:

	*nix/linux	windows	macos†	plan9†
user specific	$XDG_DATA_HOME or ~/.local/share	%AppData%	~/Library/Application Support	$home/lib ?
shared	$XDG_DATA_DIRS or /usr/share/ (or maybe /usr/local/share/)	%ProgramData%	/Library/Application Support	/lib ?

^†I have never actually used macos or plan9, so this information may be of dubious accuracy

I'm sort of shocked go doesn't already have this, considering every platform seems to have a standardized way to handle these sorts of files and for good reason—this would be endlessly useful.

This proposal is just to start the discussion and to crowdsource the knowledge required to implement this on plan9 and macos (and other platforms like ios)—I don't have any experience outside of Linux and Windows.

Possible issues

On Windows, it appears that %appdata% is also being returned from os.UserConfigDir. This should be made clear in the documentation to prevent possible file collisions.

The same issue will appear on macos if the correct data dir to use is ~/Library/Application Support; also on plan9 if ~/lib is the correct data dir.

[cowtoolz]

@gopherbot add ExpertNeeded

[cowtoolz]

@gopherbot add WaitingForInfo

[cowtoolz]

I should note that the user-specific function described in this proposal has briefly been mentioned before in #29960 by @mvdan. His comments are as follows:

Please note that I'm not suggesting adding support for XDG base directories into the os package. Most notably, adding UserConfigDir might lead to some users requesting UserDataDir. However, I can think of multiple reasons why they're not equally important: Mac and Windows don't obey XDG, so they only have one persistent per-user directory. Even on Linux, most apps store all their data in their config dir. For example: Chromium, Libreoffice, GIMP, and gcloud all seem to ignore XDG_DATA_HOME. Most programs only need one persistent per-user directory, and it's often for config files; see the links above for examples
The only disadvantage I see with adding this API is that one could say we'd be encouraging users to store data files into XDG_CONFIG_HOME instead of XDG_DATA_HOME on XDG systems.

They suggest this workaround for systems that respect the XDG base dir spec:

func userDataDir() (string, error)
    if dir := os.Getenv("XDG_DATA_HOME"); dir != "" {
        return dir, nil
    }
    // fall back to something sane on all platforms
    return os.UserConfigDir()
}

I do understand the sentiment here—on one hand, because of this the possibility for collisions exists and there is some degree of redundancy on a few platforms.

I do have a serious problem with this though:

Even on Linux, most apps store all their data in their config dir. For example: Chromium, Libreoffice, GIMP, and gcloud all seem to ignore XDG_DATA_HOME. Most programs only need one persistent per-user directory, and it's often for config files...

This seems to be quite a dubious collection of claims to me—I can think of equally many programs that do use multiple folders for application data (regardless of their respect for the XDG base dir spec). Plus, the idea that most programs only store data in the form of configs and cache files is probably untrue (and seems very opinionated to be enforced via a language's standard library).

In this proposal, I'm not advocating for the enforcement of the XDG base dir spec through the os package, but the spec doesn't exist for no reason and is based on precedent/convention; we can fall back on convention for any platform anyway. The inclusion of the user-specific function described in this proposal don't require a change in how all programs in go must do things, but allows those who wish to architect software in this way to do so without having to manually bodge over the existing functionality of the os package.

[apparentlymart]

I wrote an external library with a goal similar to this proposal's: github.com/apparentlymart/go-userdirs/userdirs.

Creating a cross-platform over what is, quite frankly, a mess of conflicting conventions was quite challenging and I'm still not really sure I quite nailed it, although I am using this library in production in a few places with reasonable success.

Since I wrote that library a long time ago I don't have all of the context loaded in my brain to thoroughly compare what I ran into there with what this proposal proposes, but I do have one immediate observation:

Each of these platform conventions has both a different set of base directories and a different convention for how different applications should cohabit within those directories. I don't think just returning the base directory is sufficient for a true cross-platform abstraction.

For example, given a hypothetical program called "That Escalated Quickly" by the vendor "AbstractionCorp", the conventional directory to use for that program on these platforms might be:

• Windows: %APPDATA%\AbstractionCorp\That Escalated Quickly
• Linux with XDG: $XDG_DATA_HOME/that-escalated-quickly
• macOS: ~/Library/Application Support/com.abstractioncorp.that-escalated-quickly

(I'm not sure what's conventional on plan9. My library doesn't support it.)

Assuming that a calling application really does want to follow the platform conventions, they'd need more than just the base directory that they are supposed to create their directory inside. Under the current form of this proposal, it seems like they'd still need to write OS-specific code to achieve that, which makes me think that they then might as well just handle the entire problem themselves (or use a third-party library like mine, if it meets their needs).

[mvdan]

I don't oppose adding these APIs to os per se, but I don't support it either unless we've gained more experience first, given the quirks to be figured out per the comments above. That should first happen in a third party Go module. If it's clear that Go users really need this third party Go module, and its API has been well thought out, then the case for a standard library proposal is much stronger.

[cowtoolz]

Great points @apparentlymart—I think that if this was the first proposal for one of these os.User___Dir functions that your point about diverging conventions would be at least a point of contention for me. However, os.UserConfigDir already points to Application Support and %AppData%, so this obviously isn't too big of a point of contention for everyone else (at least not big enough to prevent the addition of such a function).

[cowtoolz]

If anything, it seems os.UserConfigDir is at least a misnomer in 3/4 cases—the primary function of %AppData%[1], ~/Library/Application Support[2], and $home/lib[3] is (as their names suggest) the storage of user-specific application data and/or other application related files. This of course includes configs, but to say applications don't store much more than config files is plain silly. These three directories are the hardcoded locations of os.UserConfigDir on windows, macos, and plan9.

The remaining hardcoded location, for Unix/Linux, is the only explicitly "config" directory, ~/.config. It's the only platform that provides a distinct user-specific "config" directory as opposed to a more general "data" directory, so I see os.UserConfigDir as much more of an enforcement/endorsement of the XDG base directory specification than a prospective os.UserDataDir considering only 1 of 4 (Unix/Linux/XDG) provides a "config" directory while 4 of 4 provide a "data" directory.

Maybe os.UserDataDir isn't the best name (due to the phrase "data dir" being associated with XDG), but that's aside the point of this comment and is easily solved.

1. https://learn.microsoft.com/en-us/windows/win32/win_cert/certification-requirements-for-windows-desktop-apps#10-apps-must-install-to-the-correct-folders-by-default
2. https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html#//apple_ref/doc/uid/TP40010672-CH2-SW1
3. I can't find any legit docs, but I did install 9front to check out my $home/lib and it appears to be the same story

[apparentlymart]

Oh yes, the distinction between "configuration directory" and "data directory" -- or lack thereof -- was another interesting wart in the design of my "userdirs" library, which I ended up resolving just with some documentation:

On some systems, ConfigDirs and DataDirs may overlap, so applications which scan the contents of the configuration directories should impose some additional filtering to distinguish configuration files from data files.

In other words, callers that intend to treat my library as a cross-platform abstraction must carefully name what they place in each of those directories so that they can be distinguished by something other than just what directory they are in. In the applications where I've used this, I've used some different strategies depending on the situation, including:

• Creating a fixed-named subdirectory under whatever directory I get from userdirs, with a different name for config vs. data, thereby essentially reinventing the config vs. data distinction one level deeper just in case the top-level abstraction can't provide it.
• Letting all of the files cohabit in the same directory when appropriate and then using filename prefixes/suffixes or fixed filenames to distinguish the config files from the data files.

I was intentionally vague in the docstring I included above because I was trying to leave room for adding new platform conventions in future, but FWIW here's exactly what my library currently implements in the version that's current at the time of writing this:

• Linux and other Unix-like platforms that have historically typically used the X Window System and its associated conventions/descendents: maps directly to the XDG specification, v0.8, since I designed the API around its concepts.
• Windows (windows), based on knownfolderid and its associated conventions:
  • Config dirs: only the "roaming application data" directory
  • Data dirs: only the "roaming application data" directory
  • Cache dir: the "local application data" directory
  • ("local" vs. "roaming" here differs if the current user has a network-mounted roaming profile; it's a matter of whether the data could follow them between different computers)
• macOS (darwin), based on macOS Standard Directories: Where Files Reside:
  • Config dirs: the user-specific "application support" directory only
  • Data dirs: the user-specific "application support" directory and the global "application support" directory, where the former is the "data home" in XDG terms
  • Cache dir: the user-specific "caches" directory

In practice then, on both Windows and macOS the "config home" and the "data home" end up being the same directory. Only the XDG implementation actually distinguishes those.

Others might disagree with my interpretations of the Windows and macOS conventions. I'm sharing them only in case it's helpful in evaluating this proposal, or in formulating successor proposals.

[cowtoolz]

I've done some cursory research to find "real-world uses cases." Below are my findings:

As for precedent for os.UserDataDir existing, here are the public github repos that:

• use the xdg.DataHome function from @adrg's module github.com/adrg/xdg (including rancher-sandbox/rancher-desktop, earthly/earthly)
• directly reference $XDG_DATA_HOME via os.Getenv (including caddyserver/caddy, aquasecurity/trivy, tailscale/tailscale moby/buildkit)
• directly reference ~/.local/share (including tailscale/tailscale, helm/helm, JetBrains/intellij-community)
• directly reference %AppData% (including googleapis/google-cloud-go, keybase/client, perkeep/perkeep)
• reference ~/Library/Application Support^† (also JetBrains/intellij-community and keybase/client)

^† It's possible some of these could be referring to /Library/Application Support instead of ~/Library/Application Support

As for precedent for os.DataDir existing, here are the public github repos that:

• directly reference /usr/share (including kubernetes/kubernetes, grafana/grafana, aquasecurity/trivy, rancher/rancher, containers/podman, canonical/lxd)
• directly reference /usr/local/share (including google/certificate-transparency-go, kubernetes/kubernetes, hashicorp/vault)
• directly reference %ProgramData% via os.Getenv (including tailscale/tailscale, usememos/memos)
• directly reference C:/ProgramData (including googleapis/google-api-go-client)

Of course, this doesn't prove that these need to exist, but it's a good starting point for digging into the details of how people are working around this.