config: history files should be stored in $XDG_STATE_HOME

[akyoto]

Describe the bug

According to the freedesktop basedir specification, configuration and data should be separated.

In case the link stops working, here is the relevant quote:

 $XDG_DATA_HOME defines the base directory relative to which user-specific data files should be stored.
If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used.

$XDG_CONFIG_HOME defines the base directory relative to which user-specific configuration files should be stored.
If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used. 

History is not the configuration of the shell, it's data, so it should be in the data path.
Users still have the option to add ~/.local/share/nushell to their dotfiles repository if people really want to track it.

---

In order to publish a public dotfiles repository used by others, nushell is currently the only program on my Linux installation that requires specific gitignore file type filters to do so:

!/.config/nushell/*.nu
!/.config/nvim
!/.config/tmux
!/.config/alacritty
!/.config/htop

If nushell adopts the freedesktop specification by default, we can reduce it to a simple !/.config/nushell.

How to reproduce

Install nushell and check the contents of the configuration folder.

Expected behavior

config.nu and env.nu in $XDG_CONFIG_HOME/nushell, defaults to ~/.config/nushell.
History files in $XDG_DATA_HOME/nushell, defaults to ~/.local/share/nushell.

Screenshots

No response

Configuration

| key                | value                                                            |
| ------------------ | ---------------------------------------------------------------- |
| version            | 0.84.0                                                           |
| branch             | makepkg                                                          |
| commit_hash        | d2abb8603abab43ec4a6d4ed56f9018790b99dce                         |
| build_os           | linux-x86_64                                                     |
| build_target       | x86_64-unknown-linux-gnu                                         |
| rust_version       | rustc 1.71.1 (eb26296b5 2023-08-03) (Arch Linux rust 1:1.71.1-1) |
| cargo_version      | cargo 1.71.1                                                     |
| build_time         | 2023-08-23 02:21:39 +00:00                                       |
| build_rust_channel | release                                                          |
| allocator          | standard                                                         |
| features           | dataframe, default, extra, sqlite, trash, which, zip             |
| installed_plugins  |                                                                  |

Additional context

Edit: While writing this issue, I learned something new: There is also a $XDG_STATE_HOME which I wasn't aware of until now and it defaults to ~/.local/state. It specifically mentions "actions history" as a use case. This seems to be the best location for Linux installations.

[fdncred]

Nushell isn't XDG compliant, as you've noticed. There is a PR to fake it for the config file but not for the history. We talked about that PR recently and it may not ever land.

The real crux of the issue is that nushell is cross platform and not just a Linux shell. So, nushell can't rely on Linux things like this. The second issue is that in order to read the XDG env vars at startup or login, you'd have to know where the config is before trying to read it, cross platform. So, you can't read a config file to know where the config file is and then read where the config file is that you found.

[akyoto]

The second issue is that in order to read the XDG env vars at startup or login, you'd have to know where the config is before trying to read it

Hmm, I see the problem.

In that case, would it be acceptable for the developers of nushell and the users to simply treat the XDG env vars as undefined at nushell startup because of the problem you mentioned?

The spec mentions that if the env vars are empty, the default values must be assumed by the app, not the OS.
The default values that the app should assume are ~/.config/nushell and ~/.local/state/nushell.

It would be a compromise and not fully compliant, because the XDG env vars are not read by nushell.
It instead assumes the XDG env vars are always empty, but this compromise would at least separate configuration and state.

So this is what I was thinking of (in pseudo-code):

if linux {
	configDir = homeDir + ".config/nushell/"
	stateDir = homeDir + ".local/state/nushell/"
}

This skips reading the env vars at startup because of the problem you mentioned.
Only half-compliant, but I think it's possible to find some common ground here.

[fdncred]

This is essentially hard coding the paths, which is what we do now, but the paths are set by the dirs-next crate https://docs.rs/dirs-next/2.0.0/dirs_next/fn.config_dir.html and we only use the config_dir() function.

nushell/crates/nu-path/src/helpers.rs

Lines 9 to 12 in d2abb86

	pub fn config_dir() -> Option<PathBuf> {
	dirs_next::config_dir()
	}

[akyoto]

I see, so the problem is that crate.

It doesn't define the ~/.local/state path, probably because other systems don't really separate state and data directories.

The ideal case would be for the crate to define something like state_dir() which only changes the Linux path and returns normal data paths for other systems. However, that's currently not implemented.

At least the crate has ~/.local/share via data_dir() and data_local_dir() which seems promising as a location to store the history.

It's a compromise, yes, because theoretically state would fit better, but I also think a point can be made that this is an over-engineered separation from the XDG side and a data location would be enough for most applications.

---

Some Linux/XDG elitists are definitely going to hate me for this comment, but I think data_dir() would be fine for the location of the history in a portable program. At least it's better than the current situation.

For example, the Arch Wiki defines data_dir() as:

Where user-specific data files should be written

I am pretty sure history counts as "user-specific data files".
Let me know if I'm talking nonsense.

[myrkvi]

Isn't XDG_DATA_HOME analogous to /usr/share, but user-specific?

/usr/share is generally used to store data that is not dependent on platform. E.g. documentation (man files, html, etc). As far as I understand it, it therefore shouldn't contain variable data, such as a history file.

[IanManske]

Ideally this would be the case, but in practice I don't think many programs end up using XDG_STATE_HOME?
For example, I currently have 3 folders/files in XDG_STATE_HOME, whereas my XDG_DATA_HOME contains 30 folders/files. This includes history files for lf, zathura, and delta, as well as my whole Steam library (yikes). Of course, this doesn't necessarily mean we should follow suit, but it isn't unheard of to store variable data in XDG_DATA_HOME even if this isn't the best/correct location.

The fish shell had a similar issue were it was discussed whether XDG_STATE_HOME or XDG_DATA_HOME would be a better location for shell history.

[IanManske]

Actually, it looks like the dirs-next package is not maintained anymore. dirs seems to be maintained once again. It also has a state_dir function, but the function only applies to Linux.

[fdncred]

We used to use dirs, then it went unmaintained and switched to dirs-next. Ugh with these dirs crates!

[rgwood]

The second issue is that in order to read the XDG env vars at startup or login, you'd have to know where the config is before trying to read it, cross platform. So, you can't read a config file to know where the config file is and then read where the config file is that you found.

I'm not sure I understand this. Nushell just needs to check the relevant XDG environment variable, right? That doesn't involve reading a config file.

[utkarshgupta137]

Moreover, most modern terminals have the ability to setup the environment variables for you. Even the infamously feature-less Alacritty has this. So I don't understand why we're depriving those users the ability to choose where there files go. The argument that you can't provide a feature to anyone just because you can't provide the feature to everyone seems pretty bad to me.

[fdncred]

I'm not sure I understand this. Nushell just needs to check the relevant XDG environment variable, right? That doesn't involve reading a config file.

@rgwood Here's my reasoning. If nushell is your login shell, what environment is it reading from to get XDG info? If there is no environment, then you have to read an env.nu file from somewhere to get XDG info that tells you where to read the other config files from.

The real crux here is being able to document how to set system level environment variables, for the operating systems we support, without regard to login shell. Otherwise, we'll get inundated with how to set these and how to support these features and in my view could cripple nushell.

There is the terminal work-around that @utkarshgupta137 mentions, but for me, that seems like more of a hopeful-hack and difficult to support where we'd be reliant on upstream terminal to support this to use this nushell feature.

So, we're back to being this, if we are reliant only on nushell, you have to load a env.nu file from somewhere that has the XDG env var set, and then go to that location and load those config files.

My solution to all of this was how the closed PR was finally implemented, which was to always read from ~/.config, on every os and default to that, unless the os level directories already exist, then read from those. So, essentially there is no environment reading, it's just hard coded to one of two possible places.

The argument that you can't provide a feature to anyone just because you can't provide the feature to everyone seems pretty bad to me.

@utkarshgupta137 This is called cross platform development. We try our very best to offer everything everywhere unless it is expressly prohibited by the environment. For instance, there is a registry query command on Windows that only works on Windows because there is no Windows registry on other platforms. Likewise, there is an exec command that exists for MacOS and Linux but not Windows because Windows doesn't allow another process to take over the current process without extraordinary efforts.

[utkarshgupta137]

There is the terminal work-around that @utkarshgupta137 mentions, but for me, that seems like more of a hopeful-hack and difficult to support where we'd be reliant on upstream terminal to support this to use this nushell feature.

So, we're back to being this, if we are reliant only on nushell, you have to load a env.nu file from somewhere that has the XDG env var set, and then go to that location and load those config files.

I personally don't find this argument convincing. Should Nushell also stop supporting ANSI colors because some terminals might not support ANSI colors?

@utkarshgupta137 This is called cross platform development. We try our very best to offer everything everywhere unless it is expressly prohibited by the environment. For instance, there is a registry query command on Windows that only works on Windows because there is no Windows registry on other platforms. Likewise, there is an exec command that exists for MacOS and Linux but not Windows because Windows doesn't allow another process to take over the current process without extraordinary efforts.

So isn't this an argument in favor of supporting features even if they're not available everywhere? Although, I disagree with the premise that XDG isn't supported on Windows, since quite a few CLI & terminals tools use it.
I'm sure that there are terminals that support Windows & allow defining environment variables at the terminal level.

[rgwood]

@rgwood Here's my reasoning. If nushell is your login shell, what environment is it reading from to get XDG info?

Nushell's launched by the OS or your terminal, and they should be able to set environment variables for a login shell. On Linux you can edit /etc/environment or whatever, on Windows you can use the UI, there's probably an equivalent on macOS but even if there isn't I assume you can launch your terminal with environment variables to be inherited by the shell.

I think we can assume that users who care about this feature will be able to figure out a way to launch Nushell with a specific environment variable set.

[fdncred]

On Linux you can edit /etc/environment

I've tested this in the path and concluded that it didn't work. Maybe I did something wrong. This is the contents of my /etc/environment on WSL/Ubuntu.

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
XVAR1="1"
XDG_BLAH="/some/path"

I have nushell set to my login shell. I launch WSL and there is neither XVAR1 nor XDG_BLAH in my environment.

[IanManske]

@rgwood Correct me if I'm wrong, but the problem is that many XDG variables should depend on the user (i.e. $HOME). So, these XDG variables cannot be set in /etc/environment but must be set by a shell or some other program. (Also, I'm not sure if macOS has an /etc/environment equivalent anyways.)

For example, my system's login manager loads environment variables using sh and ~/.profile when it detects a default shell it doesn't know. So, this wouldn't be a problem in my case, since I could set the XDG variables in ~/.profile and use sh to load my initial environment which future terminal and nushell instances will inherit. But, if I were to start nushell directly by logging into a tty (e.g., ctrl+alt+f3), then I don't think there is a way to set the XDG variables beforehand.

Quoting the fish shell issue I linked before:

these directory variables never really worked well for fish since they need to be set before fish uses them - and there isn't typically a great place to do that before your shell runs, because historically unix systems configure this stuff in the shell.

Using a terminal emulator like @utkarshgupta137 mentioned, delegating it to the login manager, or running another shell beforehand are some ways around this. But the downside is that it will be on the user to ensure a consistent environment for nushell in every place that it can be run interactively. For example, the XDG variables for VSCode's integrated terminal would need to match the variables for an external terminal. And again, I think a raw tty simply won't work in this case.

Then again, I agree that users should have to option to set XDG variables and try to make it work if they want. That is, we should respect the variables if they exist, since technically every OS we support has the concept of environment variables -- it's just a matter of setting them before nushell starts. The fish and elvish shells are both cross platform and both seem to support XDG even despite the trickiness around it.

Otherwise, another possible solution is to make an equivalent to /etc/zshenv (a system level env.nu) that is loaded before all user .nu files. Here, XDG variables could be set using:

$env.XDG_STATE_HOME = ($env.HOME | path join 'state_dir')

This would enforce a similar XDG_STATE_HOME across all users though.

[rgwood]

^good points, thanks; you're right that system env vars may not be appropriate for something user-specific.

Then again, I agree that users should have to option to set XDG variables and try to make it work if they want. That is, we should respect the variables if they exist, since technically every OS we support has the concept of environment variables -- it's just a matter of setting them before nushell starts.

This seems correct to me.

[akyoto]

Isn't XDG_DATA_HOME analogous to /usr/share, but user-specific?

[...] As far as I understand it, it therefore shouldn't contain variable data, such as a history file.

I think you can't have "immutable" and "user-specific" in the same sentence. By the very definition, if the contents of the data depend on the user, then the data is mutable. If you mean "not runtime mutable after written once", then why isn't that data in /usr/share to begin with?

I don't believe ~/.local/share is meant to be immutable.

---

Ideally this would be the case, but in practice I don't think many programs end up using XDG_STATE_HOME?
For example, I currently have 3 folders/files in XDG_STATE_HOME, whereas my XDG_DATA_HOME contains 30 folders/files.

Can confirm. 36 entries in XDG_DATA_HOME. 3 in XDG_STATE_HOME.

---

The folks over at Neovim were also surprised that XDG_STATE_HOME exists, but ultimately adopted the new spec.

Personally, I think that yet another directory is a little over-engineered. It's hard enough to just keep config and data working between different systems and now we even have state to worry about.

---

In the end, what matters most is that we have config and data separated. Whether nushell maintainers decide to classify their history as data or state doesn't matter to me. fish decided their history is data, nushell maintainers can pick any of the two.

The separation itself, of config and history, is what's important.

[amtoine]

The separation itself, of config and history, is what's important.

i think this is very sensible and i quite dislike myself having the history and plugin.nu files in the same directory as config.nu, env.nu and login.nu 😕

[NotTheDr01ds]

On Linux you can edit /etc/environment

I've tested this in the path and concluded that it didn't work. Maybe I did something wrong. This is the contents of my /etc/environment on WSL/Ubuntu.

The only thing you did wrong is assume that WSL would behave the same as a "normal" Linux system ;-). In most cases, you'd be correct, but this is one of those areas where WSL behaves differently.

On a normal Linux system, /etc/environment is processed by PAM through pam_env.so (first reference I could find).

However, WSL typically doesn't run the login process, and thus PAM never runs, and thus /etc/environment is never processed. There are other side-effects to this that only surface on WSL.

But that's why you didn't see any change here. You could manually "force" it by running sudo login, which would more closely mimic "normal" Linux behavior. However, that's certainly not "automatic" configuration, and wouldn't be a "solution", just a proof-of-concept to demonstrate what's happening.

If, for WSL, you wanted to set it for one environment, as mentioned, you could handle that through the parent process (e.g. Windows Terminal or VSCode), but as you mentioned, those wouldn't necessarily be in sync.

However, a better solution with WSL would be to (untested, at least not recently):

• Set XDG_DATA_HOME (et. al.) as a Windows environment variable.
• Set the WSLENV environment variable in Windows to include XDG_DATA_HOME

And that will keep the environment the same for all WSL sessions regardless of where you launch it from.

---

Other random thoughts on this issue. I don't think this adds anything new, but summarizes several viewpoints above:

• I agree that the priority is getting history.txt the heck out of .config. IMHO, it doesn't belong there. Any of the below would accomplish the goal:

  • I'd probably prefer (by a slim margin) XDG "convention" over the XDG "spec" here and support $XDG_DATA_HOME (with the default/fallback of $HOME/.local/share - I'm used to that with Fish and many others. I don't think we should waste time quibbling over whether History is "user-specific data" or "user-specific state data".

  • I'd also be fine with $XDG_STATE_HOME/$HOME/.local/state - Nix and home-manager use that, but it seems to me to be a more clear case of being state data in there.

  • Quickest and dirtiest, just hardcode to $HOME/.local/share, but I think the flexibility of reading the environment is probably worthwhile.

[devyn]

I think most critically I would just like to be able to run nu with fully separate config when testing in-development versions, which won't support my installed plugins and may not have the "extra" commands compiled in. I can set config file, env file, plugins file with command line arguments, but there isn't an argument to change the history file location, just to disable it.

I do still want to have history, as I very often just want to cargo run and then do some of the same things - though just running with -c can sometimes be good enough for that purpose.

It would be more convenient to just be able to change the directories nu uses for all of these things all at once, and on Linux and for some open source apps ported to macOS and Windows, the standard for that definitely is the XDG_ environment variables if they exist, and reasonable defaults failing that.

[NotTheDr01ds]

Any chance this could/should be coupled with the work in #12103 if/when that happens? It seems to be tightly related and probably needs the exact same skill-level to implement.

[NotTheDr01ds]

@akyoto After considering this for a while, do you have any preference for $XDG_STATE_HOME over $XDG_DATA_HOME?

My thought is that history is better classified as "data" than "state". If you agree, would you update the title of the issue to avoid confusion?

I'm okay either way, it just seems to me that DATA is the better of the two.

[fdncred]

Any chance this could/should be coupled with the work in #12103 if/when that happens? It seems to be tightly related and probably needs the exact same skill-level to implement.

We discussed it and decided against doing this together with 12103.

[akyoto]

@akyoto After considering this for a while, do you have any preference for $XDG_STATE_HOME over $XDG_DATA_HOME?

I've settled on using XDG_STATE_HOME for history, personally.

See xdg-ninja recommendations for some more opinions.