Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let Cargo put data into platform-specific directories #1615

Open
wants to merge 7 commits into
base: master
from

Conversation

Projects
@tbu-
Copy link
Contributor

tbu- commented May 14, 2016

(rendered)

@tbu-

This comment has been minimized.

Copy link
Contributor Author

tbu- commented May 14, 2016

There's a stale pull request to do something like this: rust-lang/cargo#2127, I'll gladly update it if it is desired.

I figured that it might be big enough of a change to justify an RFC.


* This increases the complexity of where to find the files Cargo uses. This can
be alleviated by providing library functions for Cargo related tools and a
command line to tell which paths are in use.

This comment has been minimized.

@steveklabnik

steveklabnik May 14, 2016

Member

It's not just this though; it's also things like documentation, explaining how cargo works to the user, etc

This comment has been minimized.

@tbu-

tbu- May 14, 2016

Author Contributor

Good point, added to the section.


```
cache: .cache/cargo
config .config/cargo

This comment has been minimized.

@plietar

plietar May 16, 2016

These should use the XDG_CONFIG_HOME and XDG_CACHE_HOME environment variables

This comment has been minimized.

@tbu-

tbu- May 16, 2016

Author Contributor

They are, perhaps I should clarify that a little. We're following the XDG spec, and by default it's these directories.

This comment has been minimized.

@Ericson2314

Ericson2314 May 17, 2016

Contributor

+1 on clarification.

This comment has been minimized.

@tbu-

tbu- May 18, 2016

Author Contributor

Done.


```
cache: Library/Caches/Cargo
config: Library/Cargo

This comment has been minimized.

@withoutboats

withoutboats May 16, 2016

Contributor

I'm currently using OS X, and the config for all of the CLI tools I have any interest in configuring are stored somewhere under my home directory. I've never touched a config file under Library.

This comment has been minimized.

@tbu-

tbu- May 16, 2016

Author Contributor

I don't know a lot about OS X. Isn't there a Library folder in the home directory as well?

This comment has been minimized.

@Sean1708

Sean1708 May 16, 2016

Others may disagree but in my experience the unwritten rule of OSX is that GUI apps use ~/Library and CLI apps use unixy locations.

This comment has been minimized.

@withoutboats

withoutboats May 16, 2016

Contributor

My CLI tools configs and caches are all at the same location as they'd be on Linux, including some that are following XDG.

This comment has been minimized.

@plietar

plietar May 16, 2016

Homebrew uses /Library/Caches/Homebrew/ and pip ~/Library/Caches/pip/, so I guess both could be valid.
On the other hand, configuration should definitely use XDG.

@withoutboats

This comment has been minimized.

Copy link
Contributor

withoutboats commented May 16, 2016

I personally think all CLI tools should adopt XDG on all *nix platforms as the only reasonable attempt at a standard that anyone's produced. There's no actual standard on any *nix platform as far as I know, and I don't see a way that storing the files in the way specified by XDG is worse than storing them in ~/.cargo and ~/.rustup.

I don't think that Rust tools should do something different on OS X from what they do on other *nixen, for the reasons cited in the issue.

@tbu-

This comment has been minimized.

Copy link
Contributor Author

tbu- commented May 16, 2016

@withoutboats Indeed, and that's also what pip does. I've written this alternative down in the Alternatives section.

* Putting caches in designated cache directories allows backup tools to ignore
them.
* Using a `.cargo` directory on Windows is not idiomatic at all, no programs
for Windows would such a directory.

This comment has been minimized.

@randomPoison

randomPoison May 16, 2016

Typo: "no programs for Windows would such a directory".

This comment has been minimized.

@tbu-

tbu- May 16, 2016

Author Contributor

Fixed.

tbu- added some commits May 16, 2016

Use XDG on OS X as well
This swaps an alternative with the appropiate section in detailed
design.
@tbu-

This comment has been minimized.

Copy link
Contributor Author

tbu- commented May 16, 2016

@withoutboats I exchanged the plan for OS X with one of the alternatives. Now everything is the same across non-Windows platforms and only Windows differs.

@fenhl

This comment has been minimized.

Copy link

fenhl commented May 18, 2016

Another huge upside to this is that it removes clutter from ~.

@liigo

This comment has been minimized.

Copy link
Contributor

liigo commented May 20, 2016

so it interacts well with other tools on that platform

Could you elabrate? What tools will interacts well (better) with Cargo, and why?

Edit: cache or disk cleanup tools will benifit at least, as the motivation section said.

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented May 24, 2016

One drawback that may want to be added to this is that there are a good number of other apps and such which don't follow the XDG spec on Linux, and likely for their own "good reasons". This in turn means that if one day the spec becomes widely adopted it will likely need to be changed in one form or another. Put another way, many other major projects aren't following this spec, and to follow the spec it's quite likely that it'll need some sort of breaking change in one way or another. It's seems like it may be risky to hitch ourselves to this bandwagon?

Also, are there any other apps which follow XDG on OSX? Is the spec even defined for OSX? I'd be quite hesitant to follow a Linux spec on OSX as it seems like we'd just be winding up in the same situation of OSX users thinking that we're not doing the "native thing"

@sfackler

This comment has been minimized.

Copy link
Member

sfackler commented May 24, 2016

@alexcrichton On my MacBook, I have "configstore", "fish", "git', "gtk-2.0", "htop", and "nvim" directories in my ~/.config.

@Ericson2314

This comment has been minimized.

Copy link
Contributor

Ericson2314 commented May 24, 2016

On Linux, my impression is that all per-app dotfiles/dotdirs either predate or are done in ignorance of the XDG spec. Separating configuration from caching is quite useful and I cannot envision any downsides.

@reddraggone9

This comment has been minimized.

Copy link

reddraggone9 commented May 24, 2016

On Linux, my impression is that all per-app dotfiles/dotdirs either predate or are done in ignorance of the XDG spec.

That's been my experience as well. @alexcrichton Are you aware of any projects that would follow the spec if some sort breaking change were implemented? Most cases I've seen where it was brought up and rejected were due to inertia.

@jmesmon

This comment has been minimized.

Copy link

jmesmon commented May 24, 2016

@alexcrichton on linux:

% ls .config 
Trolltech.conf  d-feet   gconf     gtk-2.0  i3status      octave    radare2     xbuild
chromium        dconf    git       htop     menus         pianobar  systemd     xfce4
configstore     freerdp  gpicview  i3       monitors.xml  pulse
% ls .cache 
bower                                                                            fontconfig     mozilla
chromium                                                                         g-ir-scanner   msmtp.log
dconf                                                                            gitg           pacaur
dmenu_run                                                                        gstreamer-1.0  pip
event-sound-cache.tdb.4e8b8a8a9ece45529a7ea0bb66e3f2b7.x86_64-unknown-linux-gnu  guile          xsession-errors-:0.0
event-sound-cache.tdb.two.x86_64-unknown-linux-gnu                               jedi

Is there a specific program with issues with xdg that you're thinking of?

@liigo

This comment has been minimized.

Copy link
Contributor

liigo commented Jun 21, 2016

From the RFC text:

Benefits include:

  • [1] Using a standard directory for binary outputs can allow the user to execute Cargo-installed binaries without modifying their PATH variable. E.g. on Fedora this is apparantly already the case, in Debian there's a ticket to include it.
  • [2] Putting caches in designated cache directories allows backup tools to ignore them.
  • [3] Using a .cargo directory on Windows is not idiomatic at all, no programs for Windows would use such a directory.
  • [4] Platform specific clean-up tools such as the Disk Cleanup Wizard work with Cargo (it wouldn't be very useful to try to modify the Wizard instead of Cargo to make this work).

[1]: the installer can/already do this (one off job). not a benefit.
[2]: to make 3rd-party tools happy.
[3]: git uses .git, difficult to say it's wrong. not a benefit.
[4]: to make 3rd-party tools happy.

So, there is only one benefit: to make 3rd-party tools happy, to help them find/clean cargo's cache data.

And this is not a benefit for us, the users and developers of cargo. We get little benefit on this.

But it will make difficulty to our life: we must remember more dirs, if we want to find them; we must do more os-specific work to implement it and maintain it; we can't easily create a portable cargo that all in one directory.

One more thing: it's not clear what will be put into cache directory. (e.g. Will .cargo\registry in cache? Will some of target/debug/* in cache?)

(XDG spec is nice. But what's the benefit if we follow it?)

@fenhl

This comment has been minimized.

Copy link

fenhl commented Jun 21, 2016

@liigo It is not the only benefit. See my comment from earlier. This should also be added to the RFC text.

@tbu-

This comment has been minimized.

Copy link
Contributor Author

tbu- commented Jun 21, 2016

@liigo

[1]: the installer can/already do this (one off job). not a benefit.
[2]: to make 3rd-party tools happy.
[3]: git uses .git, difficult to say it's wrong. not a benefit.
[4]: to make 3rd-party tools happy.

[1] is a benefit because that way, you don't end up with a path in $PATH for each package manager you have.
[3] Just because git does it wrong, doesn't mean that we can't do it right.
[2] and [4] is not to make 3rd party tools happy, but to make Cargo work with them.

I think it's pretty clear that all of these are benefits. Whether they're worth the trade-off is what the RFC discussion should be about, but claiming that they aren't benefits isn't really helping.

EDIT: Regarding making 3rd party tools happy vs interacting with them: I as a user would be happy if the backup tool would know to ignore the Cargo cache. This is not for the benefit of the 3rd party tool, but for the benefit of me, the user of Cargo.

@withoutboats

This comment has been minimized.

Copy link
Contributor

withoutboats commented Jun 21, 2016

@liigo as a cargo user who sometimes uses other tools also, making cargo interoperate with them better definitely makes my life better. :-)

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jul 5, 2016

@reddraggone9

@alexcrichton Are you aware of any projects that would follow the spec if some sort breaking change were implemented?

To me it seems like everyone would want to implement the change but may not be able to. If no one actually implements it then no one's following the spec, which seems to defeat the purpose of having it in the first place?

It seems to me that if updates can't happen, then the spec is frozen as-is today. Then if it's frozen as-is there are major tools which have good reasons for not following it, and there's little hope for those reasons to get resolved?


@jmesmon

Is there a specific program with issues with xdg that you're thinking of?

Looking in my home folder, programs which don't follow xdg include:

  • ssh
  • rubygems
  • bundler
  • npm
  • git (maybe)
  • zsh/bash
  • pip
  • docker
  • cmake
  • ccache

Notably including many other languages' package managers.


I forget if this has been brought up before, but I've at least discussed it with others. The point about operating with third-party programs is also an interesting one. There are programs which don't follow XDG, so any robust third-party program would already have to handle non-conforming programs with some heuristics. In that sense you may not actually get any benefit at all if Cargo follows XDG because the heuristics are already correctly classifying Cargo. If Cargo isn't falling in the right bucket but could easily do so for some popular programs, that would also be a reasonable course of action I think.

@tbu-

This comment has been minimized.

Copy link
Contributor Author

tbu- commented Jul 5, 2016

Both pip and git support XDG.

The point about operating with third-party programs is also an interesting one. There are programs which don't follow XDG, so any robust third-party program would already have to handle non-conforming programs with some heuristics.

I use rsnapshot for backups. The heuristic is the user adding exceptions on their own. It would be much nicer if Cargo didn't need an exception.


I mentioned advantages and disadvantages in the RFC. It seems that at least the advantages are real for some users. Maybe the discussion should focus on how to weigh these arguments against each other. Especially on Windows, there seems little resistance against using the Windows-specific directories.

@reddraggone9

This comment has been minimized.

Copy link

reddraggone9 commented Jul 5, 2016

@alexcrichton

To me it seems like everyone would want to implement the change but may not be able to.

Not everyone. For some programs which have been around longer than I've been alive, following the XDG spec as a fallback would be a change in functionality that has worked for decades. Inertia gets in the way there. "If it ain't broke, don't fix it." I'm not aware of any project that has been told about the spec and decided not to follow it but would have followed it if the spec had X changed where X wasn't just keep doing what already works.

If no one actually implements it then no one's following the spec, which seems to defeat the purpose of having it in the first place?

People do implement it:

$ ls .config
abrt                          goa-1.0        Pinta
anjuta                        google-chrome  ppsspp
autostart                     gtk-2.0        pulse
beets                         gtk-3.0        QtProject.conf
cef_user_data                 gvbam          ranger
Clementine                    hexchat        retroarch
CuteMarkEd Project            htop           Rygel
dconf                         ibus           rygel.conf
dleyna-renderer-service.conf  imsettings     sealert.conf
dleyna-server-service.conf    inkscape       sonata
easytag                       insta-site     syncthing
enchant                       jstest-gtk     systemd
eog                           keepnote       tilda
epiphany                      libreoffice    tomboy
evince                        libvirt        totem
evolution                     markdownlint   tracker
filezilla                     menus          Trolltech.conf
gambatte                      mimeapps.list  unity3d
gconf                         monitors.xml   user-dirs.dirs
gedit                         mono.addins    user-dirs.locale
gespeaker                     MonoGame       VirtualBox
git                           mopidy         vlc
gnome-boxes                   mpv            watch-later
gnome-builder                 mupen64plus    yelp
gnome-control-center          myapp          youtube-dl
gnome-initial-setup-done      nautilus       Zeal
gnome-session                 nvim
gnote                         PCSX2

Just not everyone:

$ ls -d .?*
..              .git-credential-cache  .lastpass        .rbtools-cookies
.adobe          .gitk                  .lesshst         .sdkman
.bash_aliases   .gnome                 .local           .serverauth.22085
.bash_history   .gnupg                 .m2              .serverauth.4384
.bash_logout    .gradle                .macromedia      .speech-dispatcher
.bashrc         .grails                .mozilla         .ssh
.cache          .grails_history        .multirust       .subversion
.cargo          .groovy                .netrc           .Xauthority
.config         .gstreamer-0.10        .npm             .xinitrc
.emacs.d        .ICEauthority          .nvimlog         .xsel.log
.esd_auth       .idlerc                .pki             .zshrc
.exercism.json  .inputrc               .profile
.gimp-2.8       .java                  .python_history

It seems to me that if updates can't happen, then the spec is frozen as-is today. Then if it's frozen as-is there are major tools which have good reasons for not following it, and there's little hope for those reasons to get resolved?

If you're going to make that argument, having at least one example would be nice. You have a point about most other languages' package managers though. You can add sdkman to the list.

@jmesmon

This comment has been minimized.

Copy link

jmesmon commented Jul 5, 2016

@alexcrichton

I asked about programs that have issues with XDG, not ones simply that don't impliment it. I'm looking for a downside, a complication, that would cause a program to actively avoid XDG.

Right now one hasn't been presented.

there are major tools which have good reasons for not following it

I don't believe that is the case. At the least, no one has noted a reason (other than a program being old or inertia) that things don't impliment it. Please let us know if there is some reason you're aware of.

@brson

This comment has been minimized.

Copy link
Contributor

brson commented Jul 12, 2016

My opinions on this subject (not commenting directly on the content of the RFC, which I've only skimmed - sorry).

I think we should change where .cargo is located on windows. With back-compat code to support old installs. We chose the wrong place to put this directory. It should keep the .cargo name for consistency across platforms.

It's important for consistency that users can find Rust tools in expected places. There is no universal agreement on Unixes about the future of the XDG spec, and we should not use it by default. For those that want XDG we should consider making it optional.

@mitsuhiko

This comment has been minimized.

Copy link
Contributor

mitsuhiko commented Apr 15, 2018

Whatever this ends up being, please add a way to invoke cargo and friends and have it dump out where the files are going to end up. Otherwise interfacing with these files will be a nightmare for people writing shell scripts etc.

@flying-sheep

This comment has been minimized.

Copy link

flying-sheep commented Apr 15, 2018

@mitsuhiko Don’t worry: the PR for implementing this in cargo (rust-lang/cargo#5183) does this already by adding a cargo dirs command. I’m sure the same will be done for rustup.

@mitsuhiko

This comment has been minimized.

Copy link
Contributor

mitsuhiko commented Apr 15, 2018

@flying-sheep on phone atm so I can't check the commit but last time I saw that there was no way to get that output in parsable format that can be used in a shell script. Just wanted to clarify that I meant it needs something like CACHE_DIR="$(cargo dirs --cache)".

@soc

This comment has been minimized.

Copy link

soc commented Apr 15, 2018

@brson Sorry, I had trouble understanding parts of your comments, I hope I got the gist of them:

I suppose this is because of recent additions the require env variable to be be set to opt into the new behavior?

No, the code checks whether .cargo exists and uses the legacy dir structure.
Requiring the XDG env vars to be set doesn't make sense, as most people are happy with the defaults and don't need to explicitly set the env vars. The whole point of standards is that applications follow them without requiring users to restate the same fact over and over.

it still seems to me like this RFC is underspecified

I agree that the RFC is underspecified. I believe that the best way forward is to write some code to fix the issues, so that people can discuss specific details and allow changes to be proposed as code. This makes sure that everyone is on the same page, and avoids that people waste time talking past each other.

the complexities of mainlining continuity with all past and future toolchains, and past and future rustup are not well understood

I think it is important to consider all possible use cases and setups and make sure all of them work. That was my motivation when I wrote some code, and asked for feedback and ways how things could fail.

But I think it's important to also not over-complicate things: I think it's counter-productive to come up with complex migration or symlinking schemes that are not absolutely required, while there is no full understanding of the changes and their effects.

@mitsuhiko Yes, that's intended to work. I think the current PR doesn't implement this at the moment, but what you propose is exactly the idea how it should work.

@Diggsey

This comment has been minimized.

Copy link
Contributor

Diggsey commented Apr 15, 2018

I would like this functionality (finding the relevant directories) to be implemented in a separate crate and published to crates.io. This avoids the issue of different implementations between cargo/rustup and allows talking about a specific version (I imagine there will be changes as issues are found).

Then rustup/cargo could add experimental support (enabled via an env variable) allowing some time to iterate on the design and make sure it works on all the systems rust supports before enabling it by default.

@soc

This comment has been minimized.

Copy link

soc commented Apr 15, 2018

@Diggsey It is worthwhile? Isn't it simply checking .cargo etc. exists?

@tbu-

This comment has been minimized.

Copy link
Contributor Author

tbu- commented Apr 15, 2018

This avoids the issue of different implementations between cargo/rustup and allows talking about a specific version (I imagine there will be changes as issues are found).

I believe cargo and rustup need to have different versions. cargo only has to check whether it needs to use old-style or new-style directories, but e.g. rustup needs to know whether it'll call an old-style cargo on new-style directores and create symlinks in that case.

@Diggsey

This comment has been minimized.

Copy link
Contributor

Diggsey commented Apr 15, 2018

@soc Most of this RFC is dedicated to explaining exactly which paths are used in which circumstances - seems easily complicated enough to justify a crate.

@tbu- rustup needs additional logic to create the symlinks, but the rest of the logic should be shared.

@Diggsey

This comment has been minimized.

Copy link
Contributor

Diggsey commented Apr 15, 2018

One thing that is under-specified is how rustup will determine whether the symlinks need to exist for cargo?

What springs to mind is to add a new subcommand to cargo at the same time as these changes are made - old cargo versions will error when passed that subcommand. However, this kind of check is expensive and so cannot be done on every invocation.

Presumably the symlinks will be created somewhere unobtrusive, and then rustup will set CARGO_HOME appropriately before invoking cargo?

@soc

This comment has been minimized.

Copy link

soc commented Apr 15, 2018

@Diggsey

which paths are used in which circumstances - seems easily complicated enough to justify a crate

But that functionality is already living in a distinct crate.

One thing that is under-specified is how rustup will determine whether the symlinks need to exist for cargo?

I'm not planning to use symlinks.

add a new subcommand to cargo at the same time as these changes are made - old cargo versions will error when passed that subcommand

I'm fine with adding the command earlier.

@brson

This comment has been minimized.

Copy link
Contributor

brson commented Apr 16, 2018

@brson Sorry, I had trouble understanding parts of your comments, I hope I got the gist of them:

Sorry, I was typing drunk on a phone! Happens.

@Diggsey @soc @tbu-, fwiw I recognize the tide is firmly against me here, (and also my last post here is hilariously incomprehensible).

I have though, in the meantime, dropped a massive comment-bomb in @soc's cargo PR, detailing in extreme detail the impact I think this has on rustup.

I suggest you give it a read if you have an entire afternoon to spare.

@soc

This comment has been minimized.

Copy link

soc commented Apr 16, 2018

@brson Nevermind! Your help is really appreciated!

@brson

This comment has been minimized.

Copy link
Contributor

brson commented Apr 16, 2018

The RFC text that exists seems fine, modulo a few small questions (see review above)

@nrc

The RFC differs from the cargo PR in some ways:

The PR adds a CARGO_DATA_DIR that is not in this RFC.

The PR's prioritization algorithm is different from that described in this PR (and is currently buggy, in ways I described in my review of it).

In this RFC, "If any of the new variables CARGO_BIN_DIR, CARGO_CACHE_DIR, CARGO_CONFIG_DIR are set and nonempty, use the new directory structure." is unclear. I believe it is saying if any of those are set, use those values, and also use the default platform-specific values for those that aren't specified. (Note that the posted Cargo PR differs dramatically from this interpretation - for new-env-vars that are not set it falls back to the CARGO_HOME-derived values, and in some cases appears to set some directories to the empty string. Nor does it do the step of checking for new-style directories before checking legacy-style directories. Needs work).

FWIW I agree with the algorithm in this PR, and it supports the rustup design I laid out here, specifically because it allows rustup to set CARGO_HOME, CARGO_CACHE_DIR, etc all at the same time, and new toolchains will use the new system while old toolchains use the old system; just works, no capability sniffing or such.

The PR uses mac-specific directories not XDG.

Also, this RFC says two sentences about rustup: "Rustup will replicate Cargo's priorisation algorithm. If the results differ from what the executed version of Cargo will do, Rustup will add environment variables CARGO_BIN_DIR, CARGO_CACHE_DIR, CARGO_CONFIG_DIR for the new versions of Cargo, and add symlinks for the old versions of Cargo.".

I honestly don't even know what this is saying. Anyway I wrote in vast detail how to make rustup work with this scheme. Edit: After thinking about this for a while, I think part of what this is saying is something similar to my linked proposal, where CARGO_DATA_DIR/oldcargo/ is filled with symlinks to the "new" directories, and rustup calls that CARGO_HOME. The text here about "If the results differ from what the executed version of Cargo will do, Rustup will add environment variables CARGO_BIN_DIR, CARGO_CACHE_DIR, CARGO_CONFIG_DIR for the new versions of Cargo" I think is unnecessary - rustup will just always set all the env-vars and new toolchains will use the new ones and old toolchains the old ones, and because the new and old are symlinked together it'll all just work in lovely harmony.

In this RFC: "Cargo (and Rustup) are going to gain a new subcommand, cargo dirs" is unclear. Is there going to be a rustup dirs command? If so, it just prints the cargo dirs? rustup also has dirs.

This RFC makes no explicit backwards-compatibility affordances for tools that expect to be able to find cargo by preserving a ~/.cargo of symlinks for a transition period, like rustup did with ~/.multirust e.g.. On the other hand, as long as tools are run via cargo foo, under a rustup installation, if CARGO_HOME is set to something like CARGO_DATA_DIR/oldcargo, and it's filled with symlinks to the new world, they'll work whether or not there's a transitional ~/.cargo directory of symlinks. Non-rustup installs are just on their own wrt tooling backcompat.

I think you should go ahead and change rustup's directory structure at the same time - I kinda don't understand why there's such an uproar about cargo not obeying XDG, but not rustup. Some of the description in my previous link about modifying rustup for this alludes to how to change rustup in a similar way, but doesn't go into detail.

I'm not privy at this point to your plans for rustup/cargo integration, but it would be pretty trivial to just throw rustup's data into cargo's directories; or alternate create another set of RUSTUP_CACHE_DIR, etc variables that all behave the same way for rustup.

I mentioned this in the cargo PR, but I recommend feature-gating this (probably with an environment variable) in cargo until rustup is also similarly modified and they are extensively tested to work correctly together, with past and future toolchains, and through upgrades.

@jtojnar

This comment has been minimized.

Copy link

jtojnar commented Apr 16, 2018

I kinda don't understand why there's such an uproar about cargo not obeying XDG, but not rustup.

I would say people who care about XDG specs probably also care about rest of their system being organized, so they would be more likely to use their distribution’s package manager to install Rust, rather than rustup.

@soc

This comment has been minimized.

Copy link

soc commented Apr 17, 2018

I kinda don't understand why there's such an uproar about cargo not obeying XDG, but not rustup.

I think that's only because people might use cargo everyday, but rustup only occasionally, so cargo is the more "prominent" offender in this regard.

soc added a commit to soc/cargo that referenced this pull request Apr 28, 2018

Add support for platform-defined standard directories
This change stops cargo from violating the operating system rules
regarding the placement of config, cache, ... directories on Linux,
macOS and Windows.

Existing directories and overrides are retained.

The precedence is as follows:

1) use the `CARGO_HOME` environment variable if it exists (legacy)
2) use `CARGO_CACHE_DIR`, `CARGO_CONFIG_DIR` etc. env vars if they exist
3) use the ~/.cargo directory if it exists (legacy)
4) follow operating system standards

A new cargo command, `dirs`,  is added, which can provide path
information to other command line tools.

Fixes:
  rust-lang#1734
  rust-lang#1976
  rust-lang/rust#12725

Addresses:
  rust-lang/rfcs#1615
  rust-lang#148,
  rust-lang#3981
@kornelski

This comment has been minimized.

Copy link
Contributor

kornelski commented May 6, 2018

macOS is not Linux

Please don't group macOS and Linux as "Unix" platforms. File system layout on macOS is very different, and it isn't any more similar to Linux than it is to Windows. While some Unix-like directories exist on macOS for compatibility, they are not a good choice for well-behaved programs. macOS doesn't care about XDG or anything like that.

Using Linux layout on macOS is as jarring and bizarre as if this RFC proposed Linux cache in /var/Windows/TEMP/ and binaries in ~/.cargo/Program Files (x86)/ because both are "Desktop" platforms.

macOS has its own custom filesystem layout and Apple-designed conventions.

On macOS $HOME is not for application use. Dotfiles there are a clumsy transplant from foreign platforms.

While an occasional .git/config is tolerated on macOS, a large cache folder is not. Caches must be in ~/Library/Caches. There's no doubt about it.

Having said that, I'd advise against putting Cargo's config files in ~/Library/Preferences, because it wouldn't be doing it correctly. That folder is for plists from macOS's "user defaults" subsystem, which Cargo doesn't use. So ~/.cargo/config is a lesser evil.

@soc

This comment has been minimized.

Copy link

soc commented May 6, 2018

@kornelski Are you commenting on the text of the RFC or on the code of the implementation?

@kornelski

This comment has been minimized.

Copy link
Contributor

kornelski commented May 6, 2018

Text of the RFC

@soc

This comment has been minimized.

Copy link

soc commented May 6, 2018

@kornelski I think the RFC doesn't reflect the state of the art. Apart from that, deciding on the directories is pretty much the least important thing, it can pretty much be adapted in the last minute. I'd be more interested in help and advice on making the necessary changes and migration mechanisms to cargo and rustup.

@FranklinYu

This comment has been minimized.

Copy link

FranklinYu commented Dec 8, 2018

So ~/.cargo/config is a lesser evil.

@kornelski Any comment/objection about placing configuration in $XDG_CONFIG_HOME/cargo (falling back to ~/.config/cargo) on macOS? Git is doing that as well.

And what about $XDG_DATA_HOME?

System knows how to clear it (~/Library/Caches) when necessary

Does macOS really clean up this directory? It does this on iOS, but similar behavior on macOS isn’t mentioned by the official documentation (or I didn’t find it).

@ghost

This comment has been minimized.

Copy link

ghost commented Feb 4, 2019

FYI, I should point out that at least according to the opinion of a Debian developer, that people are misusing the XDG Base spec:

People often misunderstand what XDG standards were created for.

Imagine that you're writing some graphical application in those old days
before XDG standards appeared. And you want to put a link to it to the main
menu of your DE/WM. Where would you put it? ~/.gnome2/vfolders/applications?
~/.kde/share/applnk? Maybe .icewm/menu? Or all of them? What if you want
to autostart it on login? ~/.kde/Autostart? ~/.kde/share/autostart?
~/.gnome2/autostart?

The problem arises when MULTIPLE INDEPENDENT apps need SAME files.
So they came together and created XDG standard. It looks like:
[autostart-spec]
system-wide autostart files are placed in $XDG_CONFIG_DIRS/autostart/
user-specific overrides go to $XDG_CONFIG_HOME/autostart/
"based on the desktop base directory specification".
[menu-spec]
.menu files are placed in $XDG_CONFIG_DIRS/menus/
.desktop files are placed in $XDG_DATA_DIRS/applications/
user overrides go to $XDG_DATA_HOME/applications/ and $XDG_CONFIG_HOME/menus
"according to the desktop base directory specification".
and so on.

The "Base Directory Specification" itself is just html page to reference,
a base for other XDG specifications, that's why it's called "base".
As its original author said [1]:

XDG Base Directory spec is intended for use by other specification.
For example the XDG Menu specification and Autostart specification
refer to the XDG Base Directory specification instead of reinventing
their own filesystem locations / hierarchy.
It just gives the meaning to directories, used by other XDG standards,
which brought peace and clarity to the mess of desktop environments.

Those XDG standards were created by "X Desktop Group" only to define
unified directories for COMMON files of multiple X desktop environments,
not for some rogue applications to hide their own private files.
Each of files placed in those directories is extensively documented
by other XDG standards.

Later some people started to abuse those directories and put there files,
that never supposed to be there. Those people don't really think about
standards or unification. Usually they just enable displaying hidden files
in their file manager, see a lot of dotfiles in a home directory and think
that "this is wrong". They start searching how to "fix" this, find xdg
basedir-spec, and use it as an excuse for moving ~/.appname files, to
~/.config/appname, or worse, split them among .config, .local, .cache...
They don't think about /etc/xdg, they don't read FHS or other XDG standards,
they don't care about people who have to do 2-4 times more work to find and
migrate settings of selected application to another machine, they just
don't want to see dotfiles.

But don't blame XDG standard for that, blame people abusing it
to reduce the number of dotfiles in their home directory.

In other words, using the spec to enforce application-specific configuration on Linux systems is already a bit of a stretch, let alone extending it to Mac OS and Windows (which already have their own "FHS" so to speak). If we are following the Linux FHS, section 3.8.2 states:

3.8.2. Requirements

User specific configuration files for applications are stored in the user’s home directory in a file that starts with the ’.’ character (a “dot file”). If an application needs to create more than one dot file then they should be placed in a subdirectory with a name starting with a ’.’ character, (a “dot directory”). In this case the configuration files should not start with the ’.’ character.

So most applications are already following the actual standard and not the one that is causing a lot of controversy in many issue trackers. Just my 2 cents.

@joshtriplett

This comment has been minimized.

Copy link
Member

joshtriplett commented Feb 4, 2019

@mitsuhiko

This comment has been minimized.

Copy link
Contributor

mitsuhiko commented Feb 4, 2019

Since this RFC is effectively stalled for two years already and the current solution works, is there a big reason for not just closing this RFC down?

@flying-sheep

This comment has been minimized.

Copy link

flying-sheep commented Feb 4, 2019

@remyabel Serge cherry-picked one of the motivations for XDG and ignored the others. the spec notably does not limit itself to a specific subset of applications. Serge is wrong.

You also left out the FHS section 3.8.3, which comes directly after the one you quoted:

3.8.3. Home Directory Specifications and Conventions

A number of efforts have been made in the past to standardize the layout of home directories, including the XDG Base Directories specification and the GLib conventions on user directory contents. Additional efforts in this direction are possible in the future. To accomodate software which makes use of these specifications and conventions, distributions may create directory hierarchies which follow the specifications and conventions. Those directory hierarchies may be located underneath home directories.

I can’t help but see that omission as intellectually dishonest, no way you missed that.

Since this RFC is effectively stalled for two years already and the current solution works, is there a big reason for not just closing this RFC down?

It doesn’t for me. I don’t want caches on my home partition, I have ~/.cache mounted somewhere else and I want applications to respect that.

Also the general idea has been approved. IDK if closing and opening a new one is the right way or updating this one, but closing this without replacement is definitely not the right way.

@kornelski

This comment has been minimized.

Copy link
Contributor

kornelski commented Feb 4, 2019

Any comment/objection about placing configuration in $XDG_CONFIG_HOME/cargo (falling back to ~/.config/cargo) on macOS? Git is doing that as well.

@FranklinYu If $XDG_CONFIG_HOME is ~/Library/Preferences then $XDG_CONFIG_HOME/cargo/config is OK. Better would be $XDG_CONFIG_HOME/org.rust-lang.cargo/config.

However, beware of just mapping CARGO_HOME there, because $XDG_CONFIG_HOME/cargo/registry is very inappropriate. It'd have to be in $XDG_CACHE_HOME/cargo/registry or better in $XDG_CACHE_HOME/org.rust-lang.cargo/registry.

So there's no single place in macOS that is valid for CARGO_HOME. Cargo's directory contains very different things, like config, executables and caches, each of which has a separate dedicated directory on macOS, and these directories have different user-visible behavior (cross-device sync, automatic backups, automatic deletion) so choosing wrong is not just a matter of good style, but can cause performance degradation or data loss.

If you can't separate them properly, then it's better to use ~/.cargo. It's still bad and alien for macOS standards, but less bad than dropping hundreds of thousands of cache files and gigabytes of data in a preferences directory that is meant only for a bunch of 1KB text files, or keeping config in a cache directory that is not backed up and gets purged when running low on disk space.

@joshtriplett

This comment has been minimized.

Copy link
Member

joshtriplett commented Feb 5, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.