Support platform-defined standard directories #5183

soc · 2018-03-15T00:50:24Z

This change stops cargo from violating the operating system rules
regarding the placement of config, cache, ... directories on Linux,
macOS and Windows.

Existing directories and overrides are retained.

The precedence is as follows:

use the CARGO_HOME environment variable if it exists (legacy)
use CARGO_CACHE_DIR, CARGO_CONFIG_DIR etc. env vars if they exist
use the ~/.cargo directory if it exists (legacy)
follow operating system standards

A new cargo command, dirs, is added, which can provide path
information to other command line tools.

Fixes:
#1734
#1976
rust-lang/rust#12725

Addresses:
rust-lang/rfcs#1615
#148,
#3981

rust-highfive · 2018-03-15T00:50:35Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

soc · 2018-03-15T00:51:38Z

Hi everyone, I would love to get some feedback on this (and I couldn't reach anyone at #cargo).
There are probably still things that are wrong and need to be adapted, but I'd like to get some feedback early.

soc · 2018-03-15T03:06:05Z

New cargo dirs command

Examples:

with existing .cargo directory

$ target/debug/cargo dirs
CARGO_CACHE_DIR:  "/home/soc/.cargo"
CARGO_CONFIG_DIR: "/home/soc/.cargo"
CARGO_DATA_DIR:   "/home/soc/.cargo"
CARGO_BIN_DIR:    "/home/soc/.cargo/bin"

without existing .cargo directory

$ target/debug/cargo dirs
CARGO_CACHE_DIR:  "/home/soc/.cache/cargo"
CARGO_CONFIG_DIR: "/home/soc/.config/cargo"
CARGO_DATA_DIR:   "/home/soc/.local/share/cargo"
CARGO_BIN_DIR:    "/home/soc/.local/bin/"

soc · 2018-03-15T03:31:53Z

src/cargo/util/config.rs

+    // This is written in the most straight-forward way possible, because it is
+    // hard as-is to understand all the different options, without trying to
+    // save lines of code.
+    pub fn cargo_dirs() -> CargoDirs {


Should Config::cargo_dirs be renamed/moved to CargoDirs::new?

Yeah, new would looks slightly better I think, though current option is OK as well!

bors · 2018-03-15T05:22:25Z

☔ The latest upstream changes (presumably #5176) made this pull request unmergeable. Please resolve the merge conflicts.

aturon · 2018-03-16T04:34:46Z

cc @nrc

matklad

Overall, this looks great to me @soc!

One thing I am really worried about is how are we going to test this =/

we want to test it across at least mac/windows/linux (and probably on windows, there's also msvc/mingw/sigwin/linux subsystem axis?)
we want to test different fallback scenarios
we want to test this in conjunction with rustup

All this together implies to me that plain #[test] tests ain't gonna work here at all :(

I am thinking about a really heavy weight solution, like preparing docker images for different initial state of the machines, and then writing tests as bash scripts, which install rustup, create cargo project, etc. But, one does not simply create a docker image for windows I guess 🤷‍♂️ ?

It's also interesting that this PR actually does two things:

it refactors Cargo to support CargoDirs instead of monolithic CARGO_HOME.
it changes default locations for stuff, using directories and environment variables.

I wonder if it makes sense to split this over two pull request, and implement a refactoring first, while preserving current behavior fully. That way, we can separately check that the refactoring does not introduce regressions by itself, and then maximally concentrate on the fallback bits.

@rust-lang/cargo

matklad · 2018-03-17T10:15:01Z

src/cargo/ops/cargo_install.rs

@@ -122,7 +143,7 @@ pub fn install(
    if installed_anything {
        // Print a warning that if this directory isn't in PATH that they won't be
        // able to run these commands.
-        let dst = metadata(opts.config, &root)?.parent().join("bin");
+        let dst = metadata(opts.config, &Filesystem::new(root.config_dir))?.parent().join("bin");


This should be root.bin_dir I guess?

matklad · 2018-03-17T10:15:01Z

src/cargo/util/config.rs

+    // This is written in the most straight-forward way possible, because it is
+    // hard as-is to understand all the different options, without trying to
+    // save lines of code.
+    pub fn cargo_dirs() -> CargoDirs {


Yeah, new would looks slightly better I think, though current option is OK as well!

matklad · 2018-03-17T10:15:01Z

src/cargo/util/config.rs

+        let home_dir = ::home::home_dir().ok_or_else(|| {
+            format_err!("Cargo couldn't find your home directory. \
+                         This probably means that $HOME was not set.")
+        }).unwrap();


I think unwraps may panic in some obscure, yet real world scenario, so we really need proper error handlng. Changing the return type to CargoResult<CargoDirs> and replacing .unwraps with ? should do the trick I think?

As for an example of weird scenario, there were bug in Cargo about ::std::env::current_exe call failing, because Cargo was executed in chroot without procfs :)

matklad · 2018-03-17T10:15:01Z

src/cargo/util/config.rs

+        let mut cache_dir = cargo_dirs.cache_dir().to_path_buf();
+        let mut config_dir = cargo_dirs.config_dir().to_path_buf();
+        let mut data_dir = cargo_dirs.data_dir().to_path_buf();
+        // fixme: executable_dir only available on Linux, use data_dir on macOS and Windows?


This should be figured out in the RFC thread perhaps?

matklad · 2018-03-17T10:15:01Z

src/cargo/util/config.rs

+
+        // 3. .cargo exists
+        let legacy_cargo_dir = home_dir.join(".cargo");
+        if cargo_home_env.is_none() && legacy_cargo_dir.exists() {


This should be strictly a fall-back perhaps? That is, if all variables are not set, we set all dirs from .cargo, in contrast with the current per-dir approach?

matklad · 2018-03-17T10:15:01Z

src/cargo/util/config.rs

-    for current in paths::ancestors(pwd) {
-        let possible = current.join(".cargo").join("config");
+    for current in paths::ancestors(&dirs.current_dir) {
+        let possible = current.join(".cargo").join("config"); // fixme: what to do about this?


Nothing to be done here? It's an explicit feture of Cargo that it looks for .cargo/config in call parent directories. This exists for per-project .cargo dirs. Or am I missing something here?

Yes, I think you are right. I just wanted to be extra careful by marking all changes where I wasn't absolutely sure.

soc · 2018-03-17T12:48:46Z

@matklad Yes, testing is also a concern for me.

I think as a first step it would really help to have a list of different configurations and scenarios, so it is at least possible to test everything in an organized fashion, even if it is done manually.

I'm not sure whether splitting things into two commits makes sense, I feel that having an additional transitory state would have some benefits, but would also add overhead that would outweigh the benefits.

Especially because a similar PR needs to be done for rustup. With changes in one commit we would only have to test 4 different setups, {cargo: pre-change, post-change} * {rustup pre-change, post-change}.

With an additional state in the middle, this would balloon up to 9 different setups, for which all configurations and scenarios need to be tested against.

matklad · 2018-04-09T20:29:09Z

@soc have you managed to prepare a similar PR for rustup as well? I think updating rusupt would be a next step here, because we really do want to land changes to rustup and cargo simultaneously :)

soc · 2018-04-10T20:38:51Z

@matklad Not yet, but here is my plan:

I could really need some review of the current code to get a better understanding if this is the way people want to go (especially in regard to Path vs. Filesystem).
I would be thankful if someone has some time to look into the remaining test failures with me. I can make tests pass, but sometimes I'm not sure my changes to the tests are testing what was originally intended.
Implement changes in rustup.
Write some documentation that explains the changes.
Fix the existing documentation.
- Fix the official documentation.
- Fix third-party tutorials, blogs, StackOverflow, etc.
- Ping book authors.

matklad · 2018-04-11T10:37:40Z

src/cargo/ops/cargo_install.rs

@@ -235,9 +256,9 @@ fn install_one(
    // We have to check this again afterwards, but may as well avoid building
    // anything if we're gonna throw it away anyway.
    {
-        let metadata = metadata(config, root)?;
+        let metadata = metadata(config, &Filesystem::new(dirs.config_dir.clone()))?;


It feels like metadata could perhaps be a method of CargoInstallDirs?

matklad · 2018-04-11T10:37:40Z

src/cargo/util/config.rs

+    pub cache_dir: Filesystem,
+    pub config_dir: Filesystem,
+    pub data_dir: PathBuf,
+    pub bin_dir: PathBuf,


It would be great to add short docstrings here, to explain which directory stores which data.

matklad · 2018-04-11T10:37:40Z

src/cargo/util/config.rs

@@ -32,22 +33,119 @@ use util::Filesystem;

 use self::ConfigValue as CV;

+#[derive(Clone, Debug)]
+pub struct CargoDirs {
+    pub home_dir: Filesystem,


We don't actually use this field I think?

matklad · 2018-04-11T10:37:40Z

src/cargo/util/config.rs

+        let home_dir = ::home::home_dir().ok_or_else(|| {
+            format_err!("Cargo couldn't find your home directory. \
+                         This probably means that $HOME was not set.")
+        })?;


So looks like we don't use home_dir for anything except fallback, so let's move this as far down as possible, so that we don't actually fail if HOME is not set, but explicit directories are! We might want to test this behavior as well: working without home directory is great for reproducible and isolated builds.

matklad · 2018-04-11T10:37:40Z

src/cargo/util/config.rs

+        #[cfg(target_os = "macos")]
+        let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin"));
+        #[cfg(target_os = "windows")]
+        let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin"));


Let's extract this into a function

#[cfg(os = )] fn bin_dir(dirs: &ProjectDirs) -> Option<PathBuf>

matklad · 2018-04-11T10:37:40Z

src/cargo/util/config.rs

        } else if let Some(val) = self.get_path("build.target-dir")? {
-            let val = self.cwd.join(val.val);
+            let val = self.dirs.current_dir.join(val.val);


Let's use self.cwd() here for future-proofing.

matklad · 2018-04-11T10:37:40Z

src/cargo/util/config.rs

-        let home_path = self.home_path.clone().into_path_unlocked();
-        let credentials = home_path.join("credentials");
+        let config_path = self.dirs.config_dir.clone().into_path_unlocked();
+        let credentials = config_path.join("credentials");


Hm, and what is the "correct" place for credentials? config might be not the right place, because people sometimes publish it... I suggest raising this question on the RFC thread, if it wasn't raised already.

matklad · 2018-04-11T10:37:40Z

src/cargo/util/config.rs

@@ -638,7 +736,7 @@ impl Config {
                None => false,
            };
            let path = if maybe_relative {
-                self.cwd.join(tool_path)


matklad · 2018-04-11T10:46:50Z

tests/testsuite/login.rs

@@ -163,7 +163,7 @@ fn new_credentials_is_used_instead_old() {
        execs().with_status(0),
    );

-    let config = Config::new(Shell::new(), cargo_home(), cargo_home());
+    let config = Config::new(Shell::new(), CargoDirs::new().unwrap());


Here I think we want to point CargoDirs to CARGO_HOME inside test root.

One option is to provide another constructor for CargoDirs which accepts cargo_home: PathBuf. Another option is to modify the test such that it reads the config file directly. I am slightly in favor of the second approach.

matklad · 2018-04-11T11:05:03Z

tests/testsuite/cross_compile.rs

@@ -910,7 +910,7 @@ fn build_script_needed_for_host_and_target() {
    );
 }

-#[test]
+//#[test]
 fn build_deps_for_the_right_arch() {


This actually passes for me locally

alexcrichton · 2018-04-11T15:39:14Z

r? @matklad

soc · 2018-06-23T05:50:46Z

Just a heads up that I still intend working on this, I'm just blocked at the moment by some necessary changes in the supporting layers:

home_dir is broken and needs to be fixed: rust-lang/rust#51656
the APIs of dirs and directories need to change home_dir's result type from PathBuf to Option<PathBuf>
dirs and directories need to be relicensed to MIT/APL2
dirs and directories need a 1.0 release

soc · 2018-07-05T14:09:21Z

I addressed the four issues mentioned above, so these blockers are gone.

There are some small cleanups in dirs/directories (factor out common code) that I want to ship, but apart from that I'm good to go now.

alexcrichton · 2018-07-13T15:27:16Z

ping @matklad do you have a chance to take a look at this again?

matklad · 2018-07-13T18:36:48Z

Sure! This looks good to me in general, and I would be comfortable with merging it behind an unstable flag, if the following is done:

new behavior is actually hidden behind an unstable flag :-) The flag should be added over here. The end result should be that running cargo foo behaves exactly as today, and cargo -Z xdg foo uses new logic, with fallbacks and such.
directories dependency in Cargo.toml is updated to 1.0
There exact behavior is documented here. I understand that this is an unstable feature, but I think in this specific case it is important to get documentation together with the implementation, given the amount of subtitles, platform-specific behavior and the fact that RFC text lags behind the implementation. The docs would be in part a copy-paste of directories documentation of course, but it is important to keep this crate an implementation detail.
The behavior without the feature flag matches current behavior exactly. I'd love to help with auditing this, but I would prefer to do this after we have docs for the new behavior in place :)

alexcrichton · 2018-10-12T17:31:05Z

I'm going to close this because it's been stale and quiet for quite some time now unfortunately. The Cargo team is pretty tied up until after the 2018 edition, but I think we can perhaps look to help out integrating this and fixing remaining issues after the edition release.

flying-sheep · 2019-01-08T14:06:50Z

Hi! Rust 2018 is here, it’s a fresh new year! Time to get this rolling 😄

flying-sheep · 2019-01-08T14:08:14Z

Uh, how did I manage to unassign @matklad just by commenting‽

joshtriplett · 2019-01-09T21:09:11Z

src/cargo/util/config.rs

+              && cargo_cache_env.is_none()
+              && cargo_config_env.is_none()
+              && cargo_data_env.is_none()
+              && cargo_bin_env.is_none() {


This doesn't seem to handle the case where some but not all of the environment variables are set.

joshtriplett · 2019-01-09T21:15:47Z

@soc Let's see if we can get this revived.

I posted one comment for a corner case this doesn't seem to handle.

Could you please update this to fix the conflicts, and ensure that it still passes tests?

And could you please add tests for the various configuration cases (existing legacy configuration, existing XDG configuration, both, no existing configuration), to make sure they all work as expected?

FranklinYu · 2019-03-31T02:57:38Z

I don’t like the solution for macOS. See dirs-dev/directories-rs#47.

soc · 2019-03-31T12:20:59Z

@joshtriplett Sorry for the late reply.

I think my changes so far are flawed in the sense that all the existing logic should be retained as-is, otherwise it becomes really really hard to make sure the current behavior is the same without flipping the hypothetical switch to the new structure.

Sadly, at the moment I have little time for this (and the website shenanigans didn't help either).

I can offer some advice, though: As a first step, identify each an every place in cargo, rustup, etc. that uses the current structure and add an explicit branch with a check for the proposed -Z flag there.

Then test that all tools reach the right branch when the flag is set/not set.

Only at this point it makes sense to even start working on the new structure

RalfJung · 2019-06-18T09:24:11Z

src/cargo/util/config.rs

+                bin_dir    = legacy_cargo_dir.join("bin");
+            // 4. ... otherwise follow platform conventions
+            } else {
+                let cargo_dirs = ProjectDirs::from("", "", "Cargo");


FWIW, in Miri we will likely use ProjectDirs::from("org", "rust-lang", "miri") to get more descriptive names on platforms that commonly do that. Might be a good idea to do the same here?

alexcrichton · 2019-07-19T18:50:50Z

I'm going to close this for now because it's been languishing for some time, but if someone is willing to take this up again and resubmit it the Cargo team would be interested in finding a reviewer for it!

This commit is a partial continuation and adaptation of rust-lang#5183, which aimed to make cargo no longer reliant on the `$HOME/.cargo` directory in user's home's, and instead uses the `directories` crate to get platform-defined standard directories for data, caches, and configs. Currently, only `XGD_DATA_HOME` (on Linux this is `$HOME/.local/share` by default) and `XDG_CACHE_HOME` (on Linux this is `$HOME/.cache` by default) are used at the moment. The priority of paths cargo will check is as follows: 1. Use `$CARGO_HOME`, if it is set 2. Use `$CARGO_CACHE_DIR`, `$CARGO_CONFIG_DIR`, etc, if they are set 3. If no environment variables are set, and `$HOME/.cargo` is present, use that 4. Finally, use the platform-default directory paths

This commit is a continuation and adaptation of rust-lang#5183, which aimed to make cargo no longer reliant on the `$HOME/.cargo` directory in user's home's, and instead uses the `directories` crate to get platform-defined standard directories for data, caches, and configs. The priority of paths cargo will check is as follows: 1. Use `$CARGO_HOME`, if it is set 2. Use `$CARGO_CACHE_DIR`, `$CARGO_CONFIG_DIR`, etc, if they are set 3. If no environment variables are set, and `$HOME/.cargo` is present, use that 4. Finally, use the platform-default directory paths

soredake · 2020-04-18T08:47:06Z

Any progress on this?

rust-highfive assigned alexcrichton Mar 15, 2018

soc force-pushed the soc:topic/standard-directories branch from 6065027 to 390edbb Mar 15, 2018

soc force-pushed the soc:topic/standard-directories branch from 390edbb to c192c7f Mar 15, 2018

soc reviewed Mar 15, 2018

View changes

soc force-pushed the soc:topic/standard-directories branch from c192c7f to a290016 Mar 15, 2018

soc force-pushed the soc:topic/standard-directories branch from a290016 to 0b350ad Mar 15, 2018

soc changed the title ~~[WIP] Add support for platform-defined standard directories~~ Add support for platform-defined standard directories Mar 17, 2018

matklad reviewed Mar 17, 2018

View changes

soc force-pushed the soc:topic/standard-directories branch 4 times, most recently from cc80517 to 3aca0cf Mar 18, 2018

Eh2406 mentioned this pull request Mar 28, 2018

Support the XDG Base Directory Specification. #1734

Open

matklad added the Rust-2018 label Mar 29, 2018

ehuss mentioned this pull request Mar 31, 2018

rfc: make cargo install extensible rust-lang/rfcs#2376

Closed

soc force-pushed the soc:topic/standard-directories branch 2 times, most recently from ff5e99a to bef5ad1 Apr 11, 2018

matklad reviewed Apr 11, 2018

View changes

tests/testsuite/cross_compile.rs Outdated

@@ -910,7 +910,7 @@ fn build_script_needed_for_host_and_target() {

);

}

#[test]

//#[test]

fn build_deps_for_the_right_arch() {

This comment has been minimized.

Sign in to view

matklad Apr 11, 2018
Member

This actually passes for me locally

soc force-pushed the soc:topic/standard-directories branch from 750edc1 to 5d807d8 Apr 11, 2018

reitermarkus mentioned this pull request Sep 4, 2018

rust/cargo: enable caching Homebrew/brew#4815

Merged

6 tasks done

alexcrichton closed this Oct 12, 2018

flying-sheep unassigned matklad Jan 8, 2019

joshtriplett reviewed Jan 9, 2019

View changes

joshtriplett reopened this Jan 9, 2019

dwijnand removed the Rust-2018 label Feb 11, 2019

nrc assigned joshtriplett Mar 6, 2019

RalfJung reviewed Jun 18, 2019

View changes

alexcrichton closed this Jul 19, 2019

alexarchambault mentioned this pull request Jan 5, 2020

Unify cache directory across all OSes coursier/coursier#1483

Closed

spacekookie mentioned this pull request Apr 1, 2020

Use platform-defined directories for cargo state #8063

Closed

soc mentioned this pull request Mar 24, 2021

Create 0000_cargo_standard_directories.md spacekookie/rfcs#1

Open

rust-lang / cargo

Support platform-defined standard directories #5183

Support platform-defined standard directories #5183

Conversation

soc commented Mar 15, 2018 • edited

rust-highfive commented Mar 15, 2018

soc commented Mar 15, 2018

soc commented Mar 15, 2018 • edited

This comment has been minimized.

soc Mar 15, 2018 Author

This comment has been minimized.

matklad Mar 17, 2018 Member

bors commented Mar 15, 2018

aturon commented Mar 16, 2018

matklad left a comment

This comment has been minimized.

matklad Mar 17, 2018 Member

This comment has been minimized.

matklad Mar 17, 2018 Member

This comment has been minimized.

matklad Mar 17, 2018 Member

This comment has been minimized.

matklad Mar 17, 2018 Member

This comment has been minimized.

matklad Mar 17, 2018 Member

This comment has been minimized.

matklad Mar 17, 2018 Member

This comment has been minimized.

soc Mar 17, 2018 Author

soc commented Mar 17, 2018 • edited

matklad commented Apr 9, 2018

soc commented Apr 10, 2018 • edited

This comment has been minimized.

matklad Apr 11, 2018 Member

This comment has been minimized.

matklad Apr 11, 2018 Member

This comment has been minimized.

matklad Apr 11, 2018 Member

This comment has been minimized.

matklad Apr 11, 2018 Member

This comment has been minimized.

matklad Apr 11, 2018 Member

This comment has been minimized.

matklad Apr 11, 2018 Member

This comment has been minimized.

matklad Apr 11, 2018 Member

This comment has been minimized.

matklad Apr 11, 2018 Member

This comment has been minimized.

matklad Apr 11, 2018 Member

This comment has been minimized.

matklad Apr 11, 2018 Member

alexcrichton commented Apr 11, 2018

soc commented Jun 23, 2018 • edited

soc commented Jul 5, 2018

alexcrichton commented Jul 13, 2018

matklad commented Jul 13, 2018

alexcrichton commented Oct 12, 2018

flying-sheep commented Jan 8, 2019

flying-sheep commented Jan 8, 2019

This comment has been minimized.

joshtriplett Jan 9, 2019 Member

joshtriplett commented Jan 9, 2019 • edited

FranklinYu commented Mar 31, 2019

soc commented Mar 31, 2019

This comment has been minimized.

RalfJung Jun 18, 2019 Member

alexcrichton commented Jul 19, 2019

soredake commented Apr 18, 2020

soc commented Mar 15, 2018 •

edited

soc commented Mar 15, 2018 •

edited

soc Mar 15, 2018
Author

matklad Mar 17, 2018
Member

matklad Mar 17, 2018
Member

matklad Mar 17, 2018
Member

matklad Mar 17, 2018
Member

matklad Mar 17, 2018
Member

matklad Mar 17, 2018
Member

matklad Mar 17, 2018
Member

soc Mar 17, 2018
Author

soc commented Mar 17, 2018 •

edited

soc commented Apr 10, 2018 •

edited

matklad Apr 11, 2018
Member

matklad Apr 11, 2018
Member

matklad Apr 11, 2018
Member

matklad Apr 11, 2018
Member

matklad Apr 11, 2018
Member

matklad Apr 11, 2018
Member

matklad Apr 11, 2018
Member

matklad Apr 11, 2018
Member

matklad Apr 11, 2018
Member

matklad Apr 11, 2018
Member

soc commented Jun 23, 2018 •

edited

joshtriplett Jan 9, 2019
Member

joshtriplett commented Jan 9, 2019 •

edited

RalfJung Jun 18, 2019
Member