Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support platform-defined standard directories #5183

Closed
wants to merge 1 commit into from

Conversation

@soc
Copy link

@soc soc commented Mar 15, 2018

This change stops cargo from violating the operating system rules
regarding the placement of config, cache, ... directories on Linux,
macOS and Windows.

Existing directories and overrides are retained.

The precedence is as follows:

  1. use the CARGO_HOME environment variable if it exists (legacy)
  2. use CARGO_CACHE_DIR, CARGO_CONFIG_DIR etc. env vars if they exist
  3. use the ~/.cargo directory if it exists (legacy)
  4. follow operating system standards

A new cargo command, dirs, is added, which can provide path
information to other command line tools.

Fixes:
#1734
#1976
rust-lang/rust#12725

Addresses:
rust-lang/rfcs#1615
#148,
#3981

@rust-highfive
Copy link

@rust-highfive rust-highfive commented Mar 15, 2018

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

@soc
Copy link
Author

@soc soc commented Mar 15, 2018

Hi everyone, I would love to get some feedback on this (and I couldn't reach anyone at #cargo).
There are probably still things that are wrong and need to be adapted, but I'd like to get some feedback early.

@soc soc force-pushed the soc:topic/standard-directories branch from 6065027 to 390edbb Mar 15, 2018
@soc
Copy link
Author

@soc soc commented Mar 15, 2018

New cargo dirs command

Examples:

  • with existing .cargo directory
$ target/debug/cargo dirs
CARGO_CACHE_DIR:  "/home/soc/.cargo"
CARGO_CONFIG_DIR: "/home/soc/.cargo"
CARGO_DATA_DIR:   "/home/soc/.cargo"
CARGO_BIN_DIR:    "/home/soc/.cargo/bin"
  • without existing .cargo directory
$ target/debug/cargo dirs
CARGO_CACHE_DIR:  "/home/soc/.cache/cargo"
CARGO_CONFIG_DIR: "/home/soc/.config/cargo"
CARGO_DATA_DIR:   "/home/soc/.local/share/cargo"
CARGO_BIN_DIR:    "/home/soc/.local/bin/"
@soc soc force-pushed the soc:topic/standard-directories branch from 390edbb to c192c7f Mar 15, 2018
// This is written in the most straight-forward way possible, because it is
// hard as-is to understand all the different options, without trying to
// save lines of code.
pub fn cargo_dirs() -> CargoDirs {

This comment has been minimized.

@soc

soc Mar 15, 2018
Author

Should Config::cargo_dirs be renamed/moved to CargoDirs::new?

This comment has been minimized.

@matklad

matklad Mar 17, 2018
Member

Yeah, new would looks slightly better I think, though current option is OK as well!

@soc soc force-pushed the soc:topic/standard-directories branch from c192c7f to a290016 Mar 15, 2018
@bors
Copy link
Contributor

@bors bors commented Mar 15, 2018

The latest upstream changes (presumably #5176) made this pull request unmergeable. Please resolve the merge conflicts.

@soc soc force-pushed the soc:topic/standard-directories branch from a290016 to 0b350ad Mar 15, 2018
@aturon
Copy link
Member

@aturon aturon commented Mar 16, 2018

cc @nrc

@soc soc changed the title [WIP] Add support for platform-defined standard directories Add support for platform-defined standard directories Mar 17, 2018
Copy link
Member

@matklad matklad left a comment

Overall, this looks great to me @soc!

One thing I am really worried about is how are we going to test this =/

  • we want to test it across at least mac/windows/linux (and probably on windows, there's also msvc/mingw/sigwin/linux subsystem axis?)
  • we want to test different fallback scenarios
  • we want to test this in conjunction with rustup

All this together implies to me that plain #[test] tests ain't gonna work here at all :(

I am thinking about a really heavy weight solution, like preparing docker images for different initial state of the machines, and then writing tests as bash scripts, which install rustup, create cargo project, etc. But, one does not simply create a docker image for windows I guess 🤷‍♂️ ?

It's also interesting that this PR actually does two things:

  • it refactors Cargo to support CargoDirs instead of monolithic CARGO_HOME.
  • it changes default locations for stuff, using directories and environment variables.

I wonder if it makes sense to split this over two pull request, and implement a refactoring first, while preserving current behavior fully. That way, we can separately check that the refactoring does not introduce regressions by itself, and then maximally concentrate on the fallback bits.

@rust-lang/cargo

@@ -122,7 +143,7 @@ pub fn install(
if installed_anything {
// Print a warning that if this directory isn't in PATH that they won't be
// able to run these commands.
let dst = metadata(opts.config, &root)?.parent().join("bin");
let dst = metadata(opts.config, &Filesystem::new(root.config_dir))?.parent().join("bin");

This comment has been minimized.

@matklad

matklad Mar 17, 2018
Member

This should be root.bin_dir I guess?

// This is written in the most straight-forward way possible, because it is
// hard as-is to understand all the different options, without trying to
// save lines of code.
pub fn cargo_dirs() -> CargoDirs {

This comment has been minimized.

@matklad

matklad Mar 17, 2018
Member

Yeah, new would looks slightly better I think, though current option is OK as well!

let home_dir = ::home::home_dir().ok_or_else(|| {
format_err!("Cargo couldn't find your home directory. \
This probably means that $HOME was not set.")
}).unwrap();

This comment has been minimized.

@matklad

matklad Mar 17, 2018
Member

I think unwraps may panic in some obscure, yet real world scenario, so we really need proper error handlng. Changing the return type to CargoResult<CargoDirs> and replacing .unwraps with ? should do the trick I think?

As for an example of weird scenario, there were bug in Cargo about ::std::env::current_exe call failing, because Cargo was executed in chroot without procfs :)

let mut cache_dir = cargo_dirs.cache_dir().to_path_buf();
let mut config_dir = cargo_dirs.config_dir().to_path_buf();
let mut data_dir = cargo_dirs.data_dir().to_path_buf();
// fixme: executable_dir only available on Linux, use data_dir on macOS and Windows?

This comment has been minimized.

@matklad

matklad Mar 17, 2018
Member

This should be figured out in the RFC thread perhaps?


// 3. .cargo exists
let legacy_cargo_dir = home_dir.join(".cargo");
if cargo_home_env.is_none() && legacy_cargo_dir.exists() {

This comment has been minimized.

@matklad

matklad Mar 17, 2018
Member

This should be strictly a fall-back perhaps? That is, if all variables are not set, we set all dirs from .cargo, in contrast with the current per-dir approach?

for current in paths::ancestors(pwd) {
let possible = current.join(".cargo").join("config");
for current in paths::ancestors(&dirs.current_dir) {
let possible = current.join(".cargo").join("config"); // fixme: what to do about this?

This comment has been minimized.

@matklad

matklad Mar 17, 2018
Member

Nothing to be done here? It's an explicit feture of Cargo that it looks for .cargo/config in call parent directories. This exists for per-project .cargo dirs. Or am I missing something here?

This comment has been minimized.

@soc

soc Mar 17, 2018
Author

Yes, I think you are right. I just wanted to be extra careful by marking all changes where I wasn't absolutely sure.

@soc
Copy link
Author

@soc soc commented Mar 17, 2018

@matklad Yes, testing is also a concern for me.

I think as a first step it would really help to have a list of different configurations and scenarios, so it is at least possible to test everything in an organized fashion, even if it is done manually.

I'm not sure whether splitting things into two commits makes sense, I feel that having an additional transitory state would have some benefits, but would also add overhead that would outweigh the benefits.

Especially because a similar PR needs to be done for rustup. With changes in one commit we would only have to test 4 different setups, {cargo: pre-change, post-change} * {rustup pre-change, post-change}.

With an additional state in the middle, this would balloon up to 9 different setups, for which all configurations and scenarios need to be tested against.

@soc soc force-pushed the soc:topic/standard-directories branch 4 times, most recently from cc80517 to 3aca0cf Mar 18, 2018
@matklad
Copy link
Member

@matklad matklad commented Apr 9, 2018

@soc have you managed to prepare a similar PR for rustup as well? I think updating rusupt would be a next step here, because we really do want to land changes to rustup and cargo simultaneously :)

@soc
Copy link
Author

@soc soc commented Apr 10, 2018

@matklad Not yet, but here is my plan:

  • I could really need some review of the current code to get a better understanding if this is the way people want to go (especially in regard to Path vs. Filesystem).
  • I would be thankful if someone has some time to look into the remaining test failures with me. I can make tests pass, but sometimes I'm not sure my changes to the tests are testing what was originally intended.
  • Implement changes in rustup.
  • Write some documentation that explains the changes.
  • Fix the existing documentation.
    • Fix the official documentation.
    • Fix third-party tutorials, blogs, StackOverflow, etc.
    • Ping book authors.
@soc soc force-pushed the soc:topic/standard-directories branch 2 times, most recently from ff5e99a to bef5ad1 Apr 11, 2018
@@ -235,9 +256,9 @@ fn install_one(
// We have to check this again afterwards, but may as well avoid building
// anything if we're gonna throw it away anyway.
{
let metadata = metadata(config, root)?;
let metadata = metadata(config, &Filesystem::new(dirs.config_dir.clone()))?;

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

It feels like metadata could perhaps be a method of CargoInstallDirs?

pub cache_dir: Filesystem,
pub config_dir: Filesystem,
pub data_dir: PathBuf,
pub bin_dir: PathBuf,

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

It would be great to add short docstrings here, to explain which directory stores which data.

@@ -32,22 +33,119 @@ use util::Filesystem;

use self::ConfigValue as CV;

#[derive(Clone, Debug)]
pub struct CargoDirs {
pub home_dir: Filesystem,

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

We don't actually use this field I think?

let home_dir = ::home::home_dir().ok_or_else(|| {
format_err!("Cargo couldn't find your home directory. \
This probably means that $HOME was not set.")
})?;

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

So looks like we don't use home_dir for anything except fallback, so let's move this as far down as possible, so that we don't actually fail if HOME is not set, but explicit directories are! We might want to test this behavior as well: working without home directory is great for reproducible and isolated builds.

#[cfg(target_os = "macos")]
let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin"));
#[cfg(target_os = "windows")]
let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin"));

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

Let's extract this into a function

#[cfg(os = )]
fn bin_dir(dirs: &ProjectDirs) -> Option<PathBuf>
} else if let Some(val) = self.get_path("build.target-dir")? {
let val = self.cwd.join(val.val);
let val = self.dirs.current_dir.join(val.val);

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

Let's use self.cwd() here for future-proofing.

let home_path = self.home_path.clone().into_path_unlocked();
let credentials = home_path.join("credentials");
let config_path = self.dirs.config_dir.clone().into_path_unlocked();
let credentials = config_path.join("credentials");

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

Hm, and what is the "correct" place for credentials? config might be not the right place, because people sometimes publish it... I suggest raising this question on the RFC thread, if it wasn't raised already.

@@ -638,7 +736,7 @@ impl Config {
None => false,
};
let path = if maybe_relative {
self.cwd.join(tool_path)

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

.cws()

@@ -163,7 +163,7 @@ fn new_credentials_is_used_instead_old() {
execs().with_status(0),
);

let config = Config::new(Shell::new(), cargo_home(), cargo_home());
let config = Config::new(Shell::new(), CargoDirs::new().unwrap());

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

Here I think we want to point CargoDirs to CARGO_HOME inside test root.

One option is to provide another constructor for CargoDirs which accepts cargo_home: PathBuf. Another option is to modify the test such that it reads the config file directly. I am slightly in favor of the second approach.

@@ -910,7 +910,7 @@ fn build_script_needed_for_host_and_target() {
);
}

#[test]
//#[test]
fn build_deps_for_the_right_arch() {

This comment has been minimized.

@matklad

matklad Apr 11, 2018
Member

This actually passes for me locally

@soc soc force-pushed the soc:topic/standard-directories branch from 750edc1 to 5d807d8 Apr 11, 2018
@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Apr 11, 2018

@soc
Copy link
Author

@soc soc commented Jun 23, 2018

Just a heads up that I still intend working on this, I'm just blocked at the moment by some necessary changes in the supporting layers:

  • home_dir is broken and needs to be fixed: rust-lang/rust#51656
  • the APIs of dirs and directories need to change home_dir's result type from PathBuf to Option<PathBuf>
  • dirs and directories need to be relicensed to MIT/APL2
  • dirs and directories need a 1.0 release
@soc
Copy link
Author

@soc soc commented Jul 5, 2018

I addressed the four issues mentioned above, so these blockers are gone.

There are some small cleanups in dirs/directories (factor out common code) that I want to ship, but apart from that I'm good to go now.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jul 13, 2018

ping @matklad do you have a chance to take a look at this again?

@matklad
Copy link
Member

@matklad matklad commented Jul 13, 2018

Sure! This looks good to me in general, and I would be comfortable with merging it behind an unstable flag, if the following is done:

  • new behavior is actually hidden behind an unstable flag :-) The flag should be added over here. The end result should be that running cargo foo behaves exactly as today, and cargo -Z xdg foo uses new logic, with fallbacks and such.

  • directories dependency in Cargo.toml is updated to 1.0

  • There exact behavior is documented here. I understand that this is an unstable feature, but I think in this specific case it is important to get documentation together with the implementation, given the amount of subtitles, platform-specific behavior and the fact that RFC text lags behind the implementation. The docs would be in part a copy-paste of directories documentation of course, but it is important to keep this crate an implementation detail.

  • The behavior without the feature flag matches current behavior exactly. I'd love to help with auditing this, but I would prefer to do this after we have docs for the new behavior in place :)

@reitermarkus reitermarkus mentioned this pull request Sep 4, 2018
6 tasks done
@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Oct 12, 2018

I'm going to close this because it's been stale and quiet for quite some time now unfortunately. The Cargo team is pretty tied up until after the 2018 edition, but I think we can perhaps look to help out integrating this and fixing remaining issues after the edition release.

@flying-sheep
Copy link

@flying-sheep flying-sheep commented Jan 8, 2019

Hi! Rust 2018 is here, it’s a fresh new year! Time to get this rolling 😄

@flying-sheep
Copy link

@flying-sheep flying-sheep commented Jan 8, 2019

Uh, how did I manage to unassign @matklad just by commenting‽

&& cargo_cache_env.is_none()
&& cargo_config_env.is_none()
&& cargo_data_env.is_none()
&& cargo_bin_env.is_none() {

This comment has been minimized.

@joshtriplett

joshtriplett Jan 9, 2019
Member

This doesn't seem to handle the case where some but not all of the environment variables are set.

@joshtriplett
Copy link
Member

@joshtriplett joshtriplett commented Jan 9, 2019

@soc Let's see if we can get this revived.

I posted one comment for a corner case this doesn't seem to handle.

Could you please update this to fix the conflicts, and ensure that it still passes tests?

And could you please add tests for the various configuration cases (existing legacy configuration, existing XDG configuration, both, no existing configuration), to make sure they all work as expected?

@FranklinYu
Copy link

@FranklinYu FranklinYu commented Mar 31, 2019

I don’t like the solution for macOS. See dirs-dev/directories-rs#47.

@soc
Copy link
Author

@soc soc commented Mar 31, 2019

@joshtriplett Sorry for the late reply.

I think my changes so far are flawed in the sense that all the existing logic should be retained as-is, otherwise it becomes really really hard to make sure the current behavior is the same without flipping the hypothetical switch to the new structure.

Sadly, at the moment I have little time for this (and the website shenanigans didn't help either).

I can offer some advice, though: As a first step, identify each an every place in cargo, rustup, etc. that uses the current structure and add an explicit branch with a check for the proposed -Z flag there.

Then test that all tools reach the right branch when the flag is set/not set.

Only at this point it makes sense to even start working on the new structure

bin_dir = legacy_cargo_dir.join("bin");
// 4. ... otherwise follow platform conventions
} else {
let cargo_dirs = ProjectDirs::from("", "", "Cargo");

This comment has been minimized.

@RalfJung

RalfJung Jun 18, 2019
Member

FWIW, in Miri we will likely use ProjectDirs::from("org", "rust-lang", "miri") to get more descriptive names on platforms that commonly do that. Might be a good idea to do the same here?

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jul 19, 2019

I'm going to close this for now because it's been languishing for some time, but if someone is willing to take this up again and resubmit it the Cargo team would be interested in finding a reviewer for it!

spacekookie added a commit to spacekookie/cargo that referenced this pull request Apr 1, 2020
This commit is a partial continuation and adaptation of rust-lang#5183, which
aimed to make cargo no longer reliant on the `$HOME/.cargo` directory
in user's home's, and instead uses the `directories` crate to get
platform-defined standard directories for data, caches, and configs.

Currently, only `XGD_DATA_HOME` (on Linux this is `$HOME/.local/share`
by default) and `XDG_CACHE_HOME` (on Linux this is `$HOME/.cache` by
default) are used at the moment.

The priority of paths cargo will check is as follows:

1. Use `$CARGO_HOME`, if it is set
2. Use `$CARGO_CACHE_DIR`, `$CARGO_CONFIG_DIR`, etc, if they are set
3. If no environment variables are set, and `$HOME/.cargo` is present,
   use that
4. Finally, use the platform-default directory paths
spacekookie added a commit to spacekookie/cargo that referenced this pull request Apr 1, 2020
This commit is a continuation and adaptation of rust-lang#5183, which aimed to
make cargo no longer reliant on the `$HOME/.cargo` directory in user's
home's, and instead uses the `directories` crate to get
platform-defined standard directories for data, caches, and configs.

The priority of paths cargo will check is as follows:

1. Use `$CARGO_HOME`, if it is set
2. Use `$CARGO_CACHE_DIR`, `$CARGO_CONFIG_DIR`, etc, if they are set
3. If no environment variables are set, and `$HOME/.cargo` is present,
   use that
4. Finally, use the platform-default directory paths
spacekookie added a commit to spacekookie/cargo that referenced this pull request Apr 1, 2020
This commit is a continuation and adaptation of rust-lang#5183, which aimed to
make cargo no longer reliant on the `$HOME/.cargo` directory in user's
home's, and instead uses the `directories` crate to get
platform-defined standard directories for data, caches, and configs.

The priority of paths cargo will check is as follows:

1. Use `$CARGO_HOME`, if it is set
2. Use `$CARGO_CACHE_DIR`, `$CARGO_CONFIG_DIR`, etc, if they are set
3. If no environment variables are set, and `$HOME/.cargo` is present,
   use that
4. Finally, use the platform-default directory paths
spacekookie added a commit to spacekookie/cargo that referenced this pull request Apr 1, 2020
This commit is a continuation and adaptation of rust-lang#5183, which aimed to
make cargo no longer reliant on the `$HOME/.cargo` directory in user's
home's, and instead uses the `directories` crate to get
platform-defined standard directories for data, caches, and configs.

The priority of paths cargo will check is as follows:

1. Use `$CARGO_HOME`, if it is set
2. Use `$CARGO_CACHE_DIR`, `$CARGO_CONFIG_DIR`, etc, if they are set
3. If no environment variables are set, and `$HOME/.cargo` is present,
   use that
4. Finally, use the platform-default directory paths
spacekookie added a commit to spacekookie/cargo that referenced this pull request Apr 1, 2020
This commit is a continuation and adaptation of rust-lang#5183, which aimed to
make cargo no longer reliant on the `$HOME/.cargo` directory in user's
home's, and instead uses the `directories` crate to get
platform-defined standard directories for data, caches, and configs.

The priority of paths cargo will check is as follows:

1. Use `$CARGO_HOME`, if it is set
2. Use `$CARGO_CACHE_DIR`, `$CARGO_CONFIG_DIR`, etc, if they are set
3. If no environment variables are set, and `$HOME/.cargo` is present,
   use that
4. Finally, use the platform-default directory paths
@soredake
Copy link

@soredake soredake commented Apr 18, 2020

Any progress on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet