Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider following XDG base directory spec #383

Closed
cesarandreu opened this issue Jul 5, 2017 · 25 comments
Closed

Consider following XDG base directory spec #383

cesarandreu opened this issue Jul 5, 2017 · 25 comments
Milestone

Comments

@cesarandreu
Copy link

@cesarandreu cesarandreu commented Jul 5, 2017

The docs show the default config location is ~/.elvish/rc.elv. Please consider following the XDG base directory spec. The default location would then be ~/.config/elvish/rc.elv.

Moving configs into a folder helps reduce clutter. In the last few years many tools have slowly been adding support: XDG Base Directory support. For comparison, fish shell uses ~/.config/fish/config.fish as its startup file, and lazy-loaded functions go in ~/.config/fish/functions/.

On macOS most tools generally continue using ~/.config as the default location. However, should you choose to follow Apple's guidelines (macOS Library Directory Details), you would use ~/Library/Application Support/elvish/ to place any configs, data, and files. Two other commonly used folders are Caches (~/Library/Caches/elvish/) and Logs (~/Library/Logs/elvish/).

@zchee
Copy link

@zchee zchee commented Jul 6, 2017

FYI: I wrote xdg basedir package for Go.
https://github.com/zchee/go-xdgbasedir

@xiaq xiaq added the comp:misc label Mar 20, 2018
@fehnomenal
Copy link
Contributor

@fehnomenal fehnomenal commented Apr 7, 2019

I want to take a stab on this.

@xiaq are you OK with the following design?

  • ~/.elvish/ exists: Do not change behavior because this is most likely a legacy (pre-xdg) install. Output a message on launch to advertise moving to $E:XDG_CONFIG_HOME/elvish.
  • Add a flag -config that defaults to $E:XDG_CONFIG_HOME/elvish or ~/.config/elvish.
    dataDir from
    var dataDir string
    gets assigned that flag value.

I would use @zchee's implementation with the Native mode to honor Apple's guidelines, see https://github.com/zchee/go-xdgbasedir/blob/7a0b8922e178cedb342090b90ef662084d88eba8/xdgbasedir.go#L16-L27

@xiaq
Copy link
Member

@xiaq xiaq commented Apr 7, 2019

I'm OK with following XDG. However if we are going to move directory, better make it the last directory move we ever have to do. The current .elvish directory has 3 things:

  • .elvish/rc.elv: This is configuration.
  • .elvish/db: This is user's data.
  • .elvish/lib: This is libraries.

XDG has a clear distinction between configuration and data, so clearly rc.elv and db should be in different places. I don't know which bucket .elvish/lib falls into though, the specification does not say. It's worthwhile researching what other languages that respect XDG do.

I'd also prefer that we have a separate implementation in Elvish instead of depending on go-xdgbasedir. I don't mean to discredit their effort, but the main reason is that this behavior needs to be documented in Elvish's documentation, and it's easier to make sure that the behavior and documentation are in sync when they are part of the same codebase.

Elvish also creates temporary files for the socket used for communication with the storage daemon; this is now always /tmp/elvish-$uid. We can use XDG's runtime directory for this, but this can be done separately.

To summarize there are several more things to do:

  1. Researching where to put .elvish/lib.
  2. Writing Elvish's own implementation for finding the directories.
  3. Documenting the behavior in Elvish's documentation.
  4. [Optional] Follow XDG for the runtime directory too.

@SolitudeSF
Copy link

@SolitudeSF SolitudeSF commented Apr 7, 2019

maybe the lib should be sourced from multiple places, to also account for potential elvish code managed by system package manager and maybe to separate epm fetched libs and user's. something like /usr/share/elvish/ -> $XDG_DATA_HOME/elvish/lib -> $XDG_CONFIG_HOME/elvish/lib. maybe thats overkill.

@xiaq
Copy link
Member

@xiaq xiaq commented Apr 7, 2019

@SolitudeSF Yes, I think this is a good opportunity to introduce a system-wide lib too.

On a related note, after reading the XDG spec, it seems that the particular definition it uses for "data files" is not so much as "user's activity data", but more like "assets used in programs", as it cites /usr/share as one of the default paths for data.

In this light, it seems that @fehnomenal's original idea of treating everything as config would make a lot of sense. And since we are back to dealing with just one directory, it makes more sense to implement it ourselves now.

@fehnomenal
Copy link
Contributor

@fehnomenal fehnomenal commented Apr 9, 2019

A system-wide lib would be really cool. In preparation to it I will implement the complete XDG Basedir spec. The lib could then be placed somewhere in $XDG_DATA_DIRS.

For the current task I will place

  • rc and libs into $XDG_CONFIG_HOME/elvish
  • db into $XDG_CACHE_HOME/elvish (it only contains "temporary" data right? I.e. is not configuration)
  • the socket into $XDG_RUNTIME_DIR/elvish.sock
  • daemon logs into $XDG_DATA_HOME/elvish (debatable but is there a better place?)

@xiaq
Copy link
Member

@xiaq xiaq commented Apr 9, 2019

The specification defines cache as "non-essential data", which is pretty vague. But a useful litmus test would be "if cache files are removed, the behavior of the program does not change, except maybe the next startup is slower". This is not true for db.

The XDG base directory specification is written in too vague a way, and it is impossible to make good choices. Treating the entire .elvish directory as config might as well be the least bad one.

@xiaq
Copy link
Member

@xiaq xiaq commented Apr 9, 2019

Hmm, after looking at how Fish stores its history in ~/.local/share/fish, I am leaning a bit towards treating db as a data file now.

@krader1961
Copy link
Contributor

@krader1961 krader1961 commented Apr 9, 2019

The way I think about it is whether the information is unique to a system, in which case it belongs in ~/.local/share/appname, or is part of the apps configuration, in which case it belongs in ~/.config/appname. In other words, is it something I probably want to sync to every system on which I run the app (such as when setting up a new system)? If yes, ~/.config, else ~/.local/share.

@zzamboni
Copy link
Contributor

@zzamboni zzamboni commented Apr 10, 2019

Call me old-fashioned, but I find XDG confusing and cumbersome, I much prefer plain dot-dirs under $HOME. But I won't object to the change, as I see a lot of other tools following XDG 😐. Also, it's great if a system-wide config dir comes from this 😄

@xiaq
Copy link
Member

@xiaq xiaq commented Apr 10, 2019

Right. I am less sure whether the complexity is worthwhile too, especially as the spec doesn't really have good definitions of how to use the different kinds of directories.

The spec is also more complex than I had, and likely most people had, expected. It's a good idea to adopt some standard, but that doesn't equate to adopting any standard. In this particular case, I am no longer sure whether the XDG base directory spec is the correct standard to adopt.

@SolitudeSF
Copy link

@SolitudeSF SolitudeSF commented Apr 11, 2019

People expect that XDG is supported. Storing everything in $HOME is incorrect approach.

@ptman
Copy link

@ptman ptman commented Apr 11, 2019

If one wants to see how the XDG spec can be adapted to different situations, the arch wiki page is quite good: https://wiki.archlinux.org/index.php/XDG_Base_Directory

@Shados
Copy link

@Shados Shados commented Sep 4, 2019

@xiaq ultimately, adopting a standard that no one else is using is largely equivalent to not adopting a standard. A standard can only functionally be considered a "standard" if it is actually in widespread use, and in this problem space there are really only two candidates that pass that test: XDG, or chuck everything in $HOME/.<something>.

I also agree with @krader1961 that the primary distinction between the use of XDG_DATA_HOME and XDG_CONFIG_HOME is by whether or not data is machine-specific, in practice. To my knowledge, most applications that follow the XDG spec do store "history" files in XDG_DATA_HOME.

@krader1961
Copy link
Contributor

@krader1961 krader1961 commented Mar 17, 2020

FWIW, the current situation causes problems when ~/.elvish is on a shared filesystem such as NFS. In my case I symlink it to my Dropbox filesystem folder. This causes Dropbox to create copies with names like db (Krader MacBook Pro's conflicted copy 2020-03-11). The important thing here is that the command history database (the db file) is fundamentally local to a machine. Whereas everything else, including the .elvish/lib directory, is almost certainly stuff you want shared with all the machines on which you use elvish.

The XDG standard is not the only way to deal with the above problem. But it's probably the solution with the broadest support.

@krader1961
Copy link
Contributor

@krader1961 krader1961 commented Feb 8, 2021

If nothing else the situation regarding the location of the Elvish database file needs to be changed. I've been using the rsync command to copy my ~/.elvish directory to the VMs that I use for testing changes. When syncing to Windows 10 that results in this error:

rsync: [receiver] rename "/c/Users/krade/.elvish/.db.T34m17" -> ".elvish/db": Device or resource bus
y (16)

Even on UNIX like systems, where that operation does not result in a fatal error, it is arguably wrong to copy the interactive command database from one system to another when syncing the Elvish config files. Syncing interactive command and location history between systems should be done independent of the Elvish config.

@xiaq xiaq added this to the 0.16.0 milestone May 4, 2021
@xiaq xiaq removed the maybe label May 4, 2021
@eternaleye
Copy link

@eternaleye eternaleye commented May 9, 2021

@xiaq Regarding the idea of treating everything as config, the rule of thumb I've found best to apply is that it's best if "config" is restricted to some finite, enumerable collection of configurable values, while "data" captures any data that could not be reconstructed simply by enumerating the configurable values, and setting them according to your preferences. This makes it a distinction between "inconvenient to reproduce" and "potentially infeasible to reproduce". Things like history are by nature "data", while by contrast things like "beep/don't beep on tab completion with no candidates" are config.

Shells tend to have a more complicated relationship with this than most GUI applications, as their config is often more nuanced, and itself may be "non-enumerable" due to being dynamically scripted. However, this should (IMO) be read as making it more like data, rather than making everything config.

@zzamboni
Copy link
Contributor

@zzamboni zzamboni commented May 9, 2021

@eternaleye I disagree - I think "config" should be anything that is needed to bring the program/application to the settings that you need to use it. In this sense, rc.elv (Elvish), init.el (Emacs), and other code-based configuration files must definitely be considered config, despite their non-enumerable nature.

Another way to put it: config is everything I could not bear to lose (e.g. I would check it into a git repo, possibly share it among machines, etc.), and data is anything that could be eventually reconstructed (e.g. history, command DB, etc.).

@solson
Copy link

@solson solson commented May 9, 2021

I agree with @eternaleye's interpretation. I could (with difficulty) reconstruct my shell configuration, but I could never possibly reconstruct the tens of thousands of unique command invocations stored in my shell history. Another example of XDG "data" that might be more illustrative is video game save files, which are infeasible to reproduce. An example of XDG "config" would be the graphics/audio settings inside the game, which are easily reproduced.

Of course, shell config is much more complicated than a bunch of checkboxes and dropdowns, so it hews closer to XDG "data" in terms of how strictly it needs to be preserved (and why I keep mine backed up and in Git), as @eternaleye mentioned. But I just take this as an illustration of what the XDG config/data distinction is intended to mean in general. For a shell, I would expect it to take after Fish and put config/functions in $XDG_CONFIG_HOME/elvish and history or other user data in $XDG_DATA_HOME/elvish.

EDIT: @krader1961 marked this as confusing — is there a specific point I should elaborate on? My only real disagreement is with the idea that history "could be eventually reconstructed". I couldn't reconstruct mine, and I would be irritated to lose it. I think we otherwise already agree that Elvish config should be in XDG_CONFIG_HOME and history should be in XDG_DATA_HOME (or maybe XDG_STATE_HOME).

@xiaq
Copy link
Member

@xiaq xiaq commented Jun 27, 2021

The XDG base directory spec was actually updated shortly after the last comment was posted :)

It now includes the concept of "state data" that is a good home for ~/.elvish/db:

There is a single base directory relative to which user-specific state data should be written. This directory is defined by the environment variable $XDG_STATE_HOME.

...

$XDG_STATE_HOME defines the base directory relative to which user-specific state files should be stored. If $XDG_STATE_HOME is either not set or empty, a default equal to $HOME/.local/state should be used.

The $XDG_STATE_HOME contains state data that should persist between (application) restarts, but that is not important or portable enough to the user that it should be stored in $XDG_DATA_HOME. It may contain:

  • actions history (logs, history, recently used files, …)

  • current state of the application that can be reused on a restart (view, layout, open files, undo history, …)

@xiaq
Copy link
Member

@xiaq xiaq commented Jun 27, 2021

The remaining problem is where lib should go. I actually agree with @SolitudeSF that libraries should be searched in both config directories and data directories, because there are two different types of libraries:

  • Libraries you write yourself and modify frequently. Such libraries can be thought as an extension of rc.elv, you want to bring it across your machines. So these libraries should be considered configs.
  • Libraries somebody else wrote, and you downloaded. Such libraries can be easily recreated from a epm:install command, so should be considered data.

To summarize:

file base directory default
db $XDG_STATE_HOME ~/.local/state
rc.elv $XDG_CONFIG_HOME ~/.config
lib (search) $XDG_CONFIG_HOME, $XDG_DATA_HOME, $XDG_DATA_DIRS ~/.config, ~/.local/share, /usr/local/share, /usr/share
lib (epm:install location) $XDG_DATA_HOME ~/.local/share

The actual directory used by Elvish will be base directory + /elvish; for example the default path for db will be ~/.local/state/elvish/db.

@xiaq
Copy link
Member

@xiaq xiaq commented Jun 27, 2021

For macOS I think I'll follow XDG too, since Apple's guidelines seem to be geared towards GUI applications. Having to type ~/Library/Application Support/Elvish/rc.elv to change your startup config also feels ridiculous.

@zzamboni
Copy link
Contributor

@zzamboni zzamboni commented Jun 28, 2021

I generally like the proposal. Would rc.elv then live in ~/.config/elvish/ as well? That would make it easier to keep it as a checked-out git repository, rather than mixed with other files in ~/.config/.

@xiaq
Copy link
Member

@xiaq xiaq commented Jun 28, 2021

Would rc.elv then live in ~/.config/elvish/ as well?

Yes.

xiaq added a commit that referenced this issue Jul 10, 2021
Tests will be added in a followup commit before this branch is merged.

This addresses #383.
@xiaq
Copy link
Member

@xiaq xiaq commented Aug 6, 2021

This is now merged into the master branch. I'll close this issue after having the behavior properly documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet