Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let Cargo put data into platform-specific directories #1615

Open
wants to merge 8 commits into
base: master
from

Conversation

@tbu-
Copy link
Contributor

@tbu- tbu- commented May 14, 2016

(rendered)

@tbu-
Copy link
Contributor Author

@tbu- tbu- commented May 14, 2016

There's a stale pull request to do something like this: rust-lang/cargo#2127, I'll gladly update it if it is desired.

I figured that it might be big enough of a change to justify an RFC.


* This increases the complexity of where to find the files Cargo uses. This can
be alleviated by providing library functions for Cargo related tools and a
command line to tell which paths are in use.

This comment has been minimized.

@steveklabnik

steveklabnik May 14, 2016
Member

It's not just this though; it's also things like documentation, explaining how cargo works to the user, etc

This comment has been minimized.

@tbu-

tbu- May 14, 2016
Author Contributor

Good point, added to the section.


```
cache: .cache/cargo
config .config/cargo

This comment has been minimized.

@plietar

plietar May 16, 2016

These should use the XDG_CONFIG_HOME and XDG_CACHE_HOME environment variables

This comment has been minimized.

@tbu-

tbu- May 16, 2016
Author Contributor

They are, perhaps I should clarify that a little. We're following the XDG spec, and by default it's these directories.

This comment has been minimized.

@Ericson2314

Ericson2314 May 17, 2016
Contributor

+1 on clarification.

This comment has been minimized.

@tbu-

tbu- May 18, 2016
Author Contributor

Done.


```
cache: Library/Caches/Cargo
config: Library/Cargo

This comment has been minimized.

@withoutboats

withoutboats May 16, 2016
Contributor

I'm currently using OS X, and the config for all of the CLI tools I have any interest in configuring are stored somewhere under my home directory. I've never touched a config file under Library.

This comment has been minimized.

@tbu-

tbu- May 16, 2016
Author Contributor

I don't know a lot about OS X. Isn't there a Library folder in the home directory as well?

This comment has been minimized.

@Sean1708

Sean1708 May 16, 2016

Others may disagree but in my experience the unwritten rule of OSX is that GUI apps use ~/Library and CLI apps use unixy locations.

This comment has been minimized.

@withoutboats

withoutboats May 16, 2016
Contributor

My CLI tools configs and caches are all at the same location as they'd be on Linux, including some that are following XDG.

This comment has been minimized.

@plietar

plietar May 16, 2016

Homebrew uses /Library/Caches/Homebrew/ and pip ~/Library/Caches/pip/, so I guess both could be valid.
On the other hand, configuration should definitely use XDG.

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented May 16, 2016

I personally think all CLI tools should adopt XDG on all *nix platforms as the only reasonable attempt at a standard that anyone's produced. There's no actual standard on any *nix platform as far as I know, and I don't see a way that storing the files in the way specified by XDG is worse than storing them in ~/.cargo and ~/.rustup.

I don't think that Rust tools should do something different on OS X from what they do on other *nixen, for the reasons cited in the issue.

@tbu-
Copy link
Contributor Author

@tbu- tbu- commented May 16, 2016

@withoutboats Indeed, and that's also what pip does. I've written this alternative down in the Alternatives section.

* Putting caches in designated cache directories allows backup tools to ignore
them.
* Using a `.cargo` directory on Windows is not idiomatic at all, no programs
for Windows would such a directory.

This comment has been minimized.

@randomPoison

randomPoison May 16, 2016

Typo: "no programs for Windows would such a directory".

This comment has been minimized.

@tbu-

tbu- May 16, 2016
Author Contributor

Fixed.

tbu- added 2 commits May 16, 2016
This swaps an alternative with the appropiate section in detailed
design.
@tbu-
Copy link
Contributor Author

@tbu- tbu- commented May 16, 2016

@withoutboats I exchanged the plan for OS X with one of the alternatives. Now everything is the same across non-Windows platforms and only Windows differs.

@fenhl
Copy link

@fenhl fenhl commented May 18, 2016

Another huge upside to this is that it removes clutter from ~.

@liigo
Copy link
Contributor

@liigo liigo commented May 20, 2016

so it interacts well with other tools on that platform

Could you elabrate? What tools will interacts well (better) with Cargo, and why?

Edit: cache or disk cleanup tools will benifit at least, as the motivation section said.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented May 24, 2016

One drawback that may want to be added to this is that there are a good number of other apps and such which don't follow the XDG spec on Linux, and likely for their own "good reasons". This in turn means that if one day the spec becomes widely adopted it will likely need to be changed in one form or another. Put another way, many other major projects aren't following this spec, and to follow the spec it's quite likely that it'll need some sort of breaking change in one way or another. It's seems like it may be risky to hitch ourselves to this bandwagon?

Also, are there any other apps which follow XDG on OSX? Is the spec even defined for OSX? I'd be quite hesitant to follow a Linux spec on OSX as it seems like we'd just be winding up in the same situation of OSX users thinking that we're not doing the "native thing"

@sfackler
Copy link
Member

@sfackler sfackler commented May 24, 2016

@alexcrichton On my MacBook, I have "configstore", "fish", "git', "gtk-2.0", "htop", and "nvim" directories in my ~/.config.

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented May 24, 2016

On Linux, my impression is that all per-app dotfiles/dotdirs either predate or are done in ignorance of the XDG spec. Separating configuration from caching is quite useful and I cannot envision any downsides.

@reddraggone9
Copy link

@reddraggone9 reddraggone9 commented May 24, 2016

On Linux, my impression is that all per-app dotfiles/dotdirs either predate or are done in ignorance of the XDG spec.

That's been my experience as well. @alexcrichton Are you aware of any projects that would follow the spec if some sort breaking change were implemented? Most cases I've seen where it was brought up and rejected were due to inertia.

@jmesmon
Copy link

@jmesmon jmesmon commented May 24, 2016

@alexcrichton on linux:

% ls .config 
Trolltech.conf  d-feet   gconf     gtk-2.0  i3status      octave    radare2     xbuild
chromium        dconf    git       htop     menus         pianobar  systemd     xfce4
configstore     freerdp  gpicview  i3       monitors.xml  pulse
% ls .cache 
bower                                                                            fontconfig     mozilla
chromium                                                                         g-ir-scanner   msmtp.log
dconf                                                                            gitg           pacaur
dmenu_run                                                                        gstreamer-1.0  pip
event-sound-cache.tdb.4e8b8a8a9ece45529a7ea0bb66e3f2b7.x86_64-unknown-linux-gnu  guile          xsession-errors-:0.0
event-sound-cache.tdb.two.x86_64-unknown-linux-gnu                               jedi

Is there a specific program with issues with xdg that you're thinking of?

@liigo
Copy link
Contributor

@liigo liigo commented Jun 21, 2016

From the RFC text:

Benefits include:

  • [1] Using a standard directory for binary outputs can allow the user to execute Cargo-installed binaries without modifying their PATH variable. E.g. on Fedora this is apparantly already the case, in Debian there's a ticket to include it.
  • [2] Putting caches in designated cache directories allows backup tools to ignore them.
  • [3] Using a .cargo directory on Windows is not idiomatic at all, no programs for Windows would use such a directory.
  • [4] Platform specific clean-up tools such as the Disk Cleanup Wizard work with Cargo (it wouldn't be very useful to try to modify the Wizard instead of Cargo to make this work).

[1]: the installer can/already do this (one off job). not a benefit.
[2]: to make 3rd-party tools happy.
[3]: git uses .git, difficult to say it's wrong. not a benefit.
[4]: to make 3rd-party tools happy.

So, there is only one benefit: to make 3rd-party tools happy, to help them find/clean cargo's cache data.

And this is not a benefit for us, the users and developers of cargo. We get little benefit on this.

But it will make difficulty to our life: we must remember more dirs, if we want to find them; we must do more os-specific work to implement it and maintain it; we can't easily create a portable cargo that all in one directory.

One more thing: it's not clear what will be put into cache directory. (e.g. Will .cargo\registry in cache? Will some of target/debug/* in cache?)

(XDG spec is nice. But what's the benefit if we follow it?)

@fenhl
Copy link

@fenhl fenhl commented Jun 21, 2016

@liigo It is not the only benefit. See my comment from earlier. This should also be added to the RFC text.

@tbu-
Copy link
Contributor Author

@tbu- tbu- commented Jun 21, 2016

@liigo

[1]: the installer can/already do this (one off job). not a benefit.
[2]: to make 3rd-party tools happy.
[3]: git uses .git, difficult to say it's wrong. not a benefit.
[4]: to make 3rd-party tools happy.

[1] is a benefit because that way, you don't end up with a path in $PATH for each package manager you have.
[3] Just because git does it wrong, doesn't mean that we can't do it right.
[2] and [4] is not to make 3rd party tools happy, but to make Cargo work with them.

I think it's pretty clear that all of these are benefits. Whether they're worth the trade-off is what the RFC discussion should be about, but claiming that they aren't benefits isn't really helping.

EDIT: Regarding making 3rd party tools happy vs interacting with them: I as a user would be happy if the backup tool would know to ignore the Cargo cache. This is not for the benefit of the 3rd party tool, but for the benefit of me, the user of Cargo.

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented Jun 21, 2016

@liigo as a cargo user who sometimes uses other tools also, making cargo interoperate with them better definitely makes my life better. :-)

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jul 5, 2016

@reddraggone9

@alexcrichton Are you aware of any projects that would follow the spec if some sort breaking change were implemented?

To me it seems like everyone would want to implement the change but may not be able to. If no one actually implements it then no one's following the spec, which seems to defeat the purpose of having it in the first place?

It seems to me that if updates can't happen, then the spec is frozen as-is today. Then if it's frozen as-is there are major tools which have good reasons for not following it, and there's little hope for those reasons to get resolved?


@jmesmon

Is there a specific program with issues with xdg that you're thinking of?

Looking in my home folder, programs which don't follow xdg include:

  • ssh
  • rubygems
  • bundler
  • npm
  • git (maybe)
  • zsh/bash
  • pip
  • docker
  • cmake
  • ccache

Notably including many other languages' package managers.


I forget if this has been brought up before, but I've at least discussed it with others. The point about operating with third-party programs is also an interesting one. There are programs which don't follow XDG, so any robust third-party program would already have to handle non-conforming programs with some heuristics. In that sense you may not actually get any benefit at all if Cargo follows XDG because the heuristics are already correctly classifying Cargo. If Cargo isn't falling in the right bucket but could easily do so for some popular programs, that would also be a reasonable course of action I think.

@tbu-
Copy link
Contributor Author

@tbu- tbu- commented Jul 5, 2016

Both pip and git support XDG.

The point about operating with third-party programs is also an interesting one. There are programs which don't follow XDG, so any robust third-party program would already have to handle non-conforming programs with some heuristics.

I use rsnapshot for backups. The heuristic is the user adding exceptions on their own. It would be much nicer if Cargo didn't need an exception.


I mentioned advantages and disadvantages in the RFC. It seems that at least the advantages are real for some users. Maybe the discussion should focus on how to weigh these arguments against each other. Especially on Windows, there seems little resistance against using the Windows-specific directories.

@reddraggone9
Copy link

@reddraggone9 reddraggone9 commented Jul 5, 2016

@alexcrichton

To me it seems like everyone would want to implement the change but may not be able to.

Not everyone. For some programs which have been around longer than I've been alive, following the XDG spec as a fallback would be a change in functionality that has worked for decades. Inertia gets in the way there. "If it ain't broke, don't fix it." I'm not aware of any project that has been told about the spec and decided not to follow it but would have followed it if the spec had X changed where X wasn't just keep doing what already works.

If no one actually implements it then no one's following the spec, which seems to defeat the purpose of having it in the first place?

People do implement it:

$ ls .config
abrt                          goa-1.0        Pinta
anjuta                        google-chrome  ppsspp
autostart                     gtk-2.0        pulse
beets                         gtk-3.0        QtProject.conf
cef_user_data                 gvbam          ranger
Clementine                    hexchat        retroarch
CuteMarkEd Project            htop           Rygel
dconf                         ibus           rygel.conf
dleyna-renderer-service.conf  imsettings     sealert.conf
dleyna-server-service.conf    inkscape       sonata
easytag                       insta-site     syncthing
enchant                       jstest-gtk     systemd
eog                           keepnote       tilda
epiphany                      libreoffice    tomboy
evince                        libvirt        totem
evolution                     markdownlint   tracker
filezilla                     menus          Trolltech.conf
gambatte                      mimeapps.list  unity3d
gconf                         monitors.xml   user-dirs.dirs
gedit                         mono.addins    user-dirs.locale
gespeaker                     MonoGame       VirtualBox
git                           mopidy         vlc
gnome-boxes                   mpv            watch-later
gnome-builder                 mupen64plus    yelp
gnome-control-center          myapp          youtube-dl
gnome-initial-setup-done      nautilus       Zeal
gnome-session                 nvim
gnote                         PCSX2

Just not everyone:

$ ls -d .?*
..              .git-credential-cache  .lastpass        .rbtools-cookies
.adobe          .gitk                  .lesshst         .sdkman
.bash_aliases   .gnome                 .local           .serverauth.22085
.bash_history   .gnupg                 .m2              .serverauth.4384
.bash_logout    .gradle                .macromedia      .speech-dispatcher
.bashrc         .grails                .mozilla         .ssh
.cache          .grails_history        .multirust       .subversion
.cargo          .groovy                .netrc           .Xauthority
.config         .gstreamer-0.10        .npm             .xinitrc
.emacs.d        .ICEauthority          .nvimlog         .xsel.log
.esd_auth       .idlerc                .pki             .zshrc
.exercism.json  .inputrc               .profile
.gimp-2.8       .java                  .python_history

It seems to me that if updates can't happen, then the spec is frozen as-is today. Then if it's frozen as-is there are major tools which have good reasons for not following it, and there's little hope for those reasons to get resolved?

If you're going to make that argument, having at least one example would be nice. You have a point about most other languages' package managers though. You can add sdkman to the list.

@jmesmon
Copy link

@jmesmon jmesmon commented Jul 5, 2016

@alexcrichton

I asked about programs that have issues with XDG, not ones simply that don't impliment it. I'm looking for a downside, a complication, that would cause a program to actively avoid XDG.

Right now one hasn't been presented.

there are major tools which have good reasons for not following it

I don't believe that is the case. At the least, no one has noted a reason (other than a program being old or inertia) that things don't impliment it. Please let us know if there is some reason you're aware of.

@brson
Copy link
Contributor

@brson brson commented Jul 12, 2016

My opinions on this subject (not commenting directly on the content of the RFC, which I've only skimmed - sorry).

I think we should change where .cargo is located on windows. With back-compat code to support old installs. We chose the wrong place to put this directory. It should keep the .cargo name for consistency across platforms.

It's important for consistency that users can find Rust tools in expected places. There is no universal agreement on Unixes about the future of the XDG spec, and we should not use it by default. For those that want XDG we should consider making it optional.

@jplatte
Copy link
Contributor

@jplatte jplatte commented Oct 11, 2020

This is what Firefox did too, they split out only the cache. (and IIRC there's an open ticket for splitting things further but it's progressing at about the same rate as this one..)

@kornelski
Copy link
Contributor

@kornelski kornelski commented Oct 11, 2020

I've started working on moving cache folders: rust-lang/cargo#8686 It's quite tricky to make all the tests pass :/

@spacekookie
Copy link
Member

@spacekookie spacekookie commented Oct 13, 2020

@kornelski you're aware of rust-lang/cargo#8063 ?

If you have the time and energy some help debugging the cache lock panic on that branch would be appreciated. I have to rebase it onto the current changes, but otherwise this version has been working for me (with some minor modification to remove an assert - which might be able to be removed anyway?)

@eode
Copy link

@eode eode commented Dec 2, 2020

This PR is linked to issue rust-lang/cargo#1734, on which there has been some progress.

A specification for which directory user-installed executables should go in is now present in the XDG Spec (merged, but not yet released). The official directory is ~/.local/bin. They decided not to use an env var -- I've grumped about that on their related issue, but so it goes. Relevant text from the spec follows:

There is a single base directory relative to which
user-specific executable files may be written.

User-specific executable files may be stored in
$HOME/.local/bin. Distributions should ensure
this directory shows up in the UNIX $PATH
environment variable, at an appropriate place.

Since $HOME</literal> might be shared between systems of different achitectures, installing compiled binaries to $HOME/.local/bincould cause problems when used on systems of differing architectures. This is often not a problem, but the fact that $HOME` becomes partially achitecture-specific
if compiled binaries are placed in it should be kept in mind.

edit: Fixed link

@Kixunil
Copy link

@Kixunil Kixunil commented Dec 2, 2020

@eode #1734 doesn't seem to be related to this. Am I missing something or you made a mistake?

@eode
Copy link

@eode eode commented Dec 2, 2020

@Kixunil I made a mistake in my original post. i edited after the post, but you probably got the original. It should be rust-lang/cargo#1734.

@soc
Copy link

@soc soc commented Dec 2, 2020

The only activity was a text update to the spec acknowledging that .local/bin is a thing, I don't think this is has any impact on the ticket itself.

@eode
Copy link

@eode eode commented Dec 2, 2020

@robertvazan
Copy link

@robertvazan robertvazan commented Mar 20, 2021

One issue that wasn't mentioned before is that $HOME might not be writable at all when an app (e.g. an IDE) is sandboxed in a container, for example flatpak. In these cases, cargo/rustup fails by default. Tinkering with environment variables is needed to make it work. Using XDG variables would make cargo/rustup work without additional configuration.

I ran into this issue when trying to setup Eclipse Corrosion in sandboxed Eclipse (eclipse/corrosion#366).

@Kixunil
Copy link

@Kixunil Kixunil commented Mar 20, 2021

What's actually blocking this? Seems like most people agree with this change, am I right?

BTW, for those who still think that if other languages don't follow XDG, Rust shouldn't either, here's an incomplete list of what other languages don't have:

  • memory safety
  • compile-time type checking
  • explicit casts

I hope you can see my point: Rust is awesome because it does things right not because some other language does the same thing.

@kornelski
Copy link
Contributor

@kornelski kornelski commented Mar 21, 2021

I would love if Cargo put its data in more appropriate directories. I've tried implementing it, but unfortunately I failed. The code appears to have the paths configurable, but changing the configuration in any way breaks a ton of tests. In many cases it's merely fragile tests that have old paths hardcoded, but some things (cargo clean IIRC) seem to actually break if you change paths at all.

@kornelski
Copy link
Contributor

@kornelski kornelski commented Mar 21, 2021

But I'm also unhappy that the RFC hasn't been updated to use correct paths on macOS. Lumping macOS with "Unixy" systems here is a mistake, because Apple has deviated from Unix conventions. Apple has its own macOS directory guidelines and APIs for getting correct directories. Freedesktop's XDG is not a macOS spec.

At very minimum, I'd like the RFC to change .cache to Library/Caches. This is not merely a matter of taste of whether one likes Linux-flavored or Apple-flavored paths. The Caches directory has a special status on macOS which affects backups, full-disk search, and disk cleanup. Putting large cargo directories anywhere else has visible detrimental effect on macOS performance.

@soc
Copy link

@soc soc commented Mar 22, 2021

What's actually blocking this? Seems like most people agree with this change, am I right?

In short: Those who care are not in charge, and those in charge do not care enough about it.
(See rust-lang/cargo#8686 (comment).)

@aloucks
Copy link

@aloucks aloucks commented Mar 22, 2021

What's actually blocking this? Seems like most people agree with this change, am I right?

Most of people in this thread agree with the change, but that doesn't mean everyone does. I primarily use Windows but I still prefer my command line tools to use "unixy" config paths (and I use bash, even on Windows). I find value in knowing whatever platform I'm using, I'll find all things related to cargo my ~/.cargo directory.

@lnicola
Copy link

@lnicola lnicola commented Mar 22, 2021

@aloucks for you it's a matter of preference, but for others it's a matter of unnecessarily synchronizing ~10 GB over possibly slow or expensive Internet connections.

@eli-schwartz
Copy link

@eli-schwartz eli-schwartz commented Mar 22, 2021

I still prefer my command line tools to use "unixy" config paths

https://archive.is/vfXl2

What is "unixy" about ~/.cargo? It doesn't conform to the platform standards for Windows or macOS, nor the generic unix freedesktop.org standards which predominate on a number of Unix platforms including but not limited to Linux. It serves no one, not even "unix". It's only slightly less pathetic than "look for configuration files in $PWD", which many other glorious terrible "unix" programs do, especially if they're written in Java.

Your "unixy" paths are a pox on my unix system, cursing me to the cruel and depressing fate of having thousands of programs that all think they're important enough to reserve top billing in my ~ rather than somewhere predictable and sensible.

And, as mentioned in the past, it's actively causing tremendous damage to store user-specific files that are analogous to $TMPDIR but should not be stored in RAM, might be useful across reboots, and should still be segregated out to not be synced, backed up, or kept when the disk is running low on space.

knowing whatever platform I'm using, I'll find all things related to cargo my ~/.cargo directory

This is inconsistent with most other tools these days, I strongly advise you to learn how to predict where tools will store their files per platform standards.

@ChrisDenton
Copy link

@ChrisDenton ChrisDenton commented Mar 22, 2021

Hm, I think it should always be possible to configure cargo to put data into user-defined directories, be that ~/.cargo or wherever else.

But to my mind that's a separate issue from supporting platform-native directories.

pfrenssen added a commit to pfrenssen/rust-rfcs that referenced this pull request Mar 22, 2021
Ref suggestion rust-lang#1615 (comment)
@pfrenssen
Copy link

@pfrenssen pfrenssen commented Mar 22, 2021

Created PR to update the RFC with macOS-specific default paths as suggested by @kornelski in #1615 (comment).

Can someone who uses macOS verify if this is correct?

@pfrenssen
Copy link

@pfrenssen pfrenssen commented Mar 22, 2021

@ChrisDenton Your suggestion is already included in the RFC. The RFC proposes to make the directories user-configurable so you will be able to keep ~/.cargo if you prefer this.

@aspenluxxxy
Copy link

@aspenluxxxy aspenluxxxy commented Mar 22, 2021

@pfrenssen macOS dirs look right to me.

@pfrenssen
Copy link

@pfrenssen pfrenssen commented Mar 22, 2021

@aspenluxxxy thanks!
@tbu- If this looks OK to you, care to merge https://github.com/tbu-/rust-rfcs/pull/1?

@soc
Copy link

@soc soc commented Mar 22, 2021

Why are we still kicking this severely outdated RFC, given that we had at least 3 different implementation attempts, and none of them bothered to follow the RFC even remotely?
The problem is not the RFC, regardless of how often people try to poke it with a stick and pretend that it is still alive.

If you want to help, then give @spacekookie a hand with rust-lang/cargo#9178.

@spacekookie
Copy link
Member

@spacekookie spacekookie commented Mar 24, 2021

@soc I was told by the cargo maintainers that there needed to be an RFC for this feature before they were comfortable merging anything.

Considering that this RFC is essentially dead (and should be closed), I made an attempt to restart the conversation here: spacekookie#1

It's a draft and doesn't cover most of the things brought up in this thread yet, but I felt it was probably going to be easier to work on a new version than to keep this one alive.

@eode
Copy link

@eode eode commented Mar 24, 2021

@robertvazan

One issue that wasn't mentioned before is that $HOME might not be writable at all when an app (e.g. an IDE) is sandboxed in a container, for example flatpak. In these cases, cargo/rustup fails by default. Tinkering with environment variables is needed to make it work. Using XDG variables would make cargo/rustup work without additional configuration.

I ran into this issue when trying to setup Eclipse Corrosion in sandboxed Eclipse (eclipse/corrosion#366).

I'd be laughing if I weren't crying right now.

This is the use-case I was looking for and didn't have at the time when the $XDG_BIN_HOME matter was being decided. I warned them that someone in the future would need it, even if that use wasn't clear now. ..and here it is, and I'll have to go argue for this all over again..

Thanks for posting that use-case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Tracker
Merge proposed
Linked issues

Successfully merging this pull request may close these issues.

None yet