New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support platform-defined standard directories #5183

Open
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
@soc
Copy link

soc commented Mar 15, 2018

This change stops cargo from violating the operating system rules
regarding the placement of config, cache, ... directories on Linux,
macOS and Windows.

Existing directories and overrides are retained.

The precedence is as follows:

  1. use the CARGO_HOME environment variable if it exists (legacy)
  2. use CARGO_CACHE_DIR, CARGO_CONFIG_DIR etc. env vars if they exist
  3. use the ~/.cargo directory if it exists (legacy)
  4. follow operating system standards

A new cargo command, dirs, is added, which can provide path
information to other command line tools.

Fixes:
#1734
#1976
rust-lang/rust#12725

Addresses:
rust-lang/rfcs#1615
#148,
#3981

@rust-highfive

This comment has been minimized.

Copy link

rust-highfive commented Mar 15, 2018

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

@soc

This comment has been minimized.

Copy link
Author

soc commented Mar 15, 2018

Hi everyone, I would love to get some feedback on this (and I couldn't reach anyone at #cargo).
There are probably still things that are wrong and need to be adapted, but I'd like to get some feedback early.

@soc soc force-pushed the soc:topic/standard-directories branch from 6065027 to 390edbb Mar 15, 2018

@soc

This comment has been minimized.

Copy link
Author

soc commented Mar 15, 2018

New cargo dirs command

Examples:

  • with existing .cargo directory
$ target/debug/cargo dirs
CARGO_CACHE_DIR:  "/home/soc/.cargo"
CARGO_CONFIG_DIR: "/home/soc/.cargo"
CARGO_DATA_DIR:   "/home/soc/.cargo"
CARGO_BIN_DIR:    "/home/soc/.cargo/bin"
  • without existing .cargo directory
$ target/debug/cargo dirs
CARGO_CACHE_DIR:  "/home/soc/.cache/cargo"
CARGO_CONFIG_DIR: "/home/soc/.config/cargo"
CARGO_DATA_DIR:   "/home/soc/.local/share/cargo"
CARGO_BIN_DIR:    "/home/soc/.local/bin/"

@soc soc force-pushed the soc:topic/standard-directories branch from 390edbb to c192c7f Mar 15, 2018

// This is written in the most straight-forward way possible, because it is
// hard as-is to understand all the different options, without trying to
// save lines of code.
pub fn cargo_dirs() -> CargoDirs {

This comment has been minimized.

@soc

soc Mar 15, 2018

Author

Should Config::cargo_dirs be renamed/moved to CargoDirs::new?

This comment has been minimized.

@matklad

matklad Mar 17, 2018

Member

Yeah, new would looks slightly better I think, though current option is OK as well!

@soc soc force-pushed the soc:topic/standard-directories branch from c192c7f to a290016 Mar 15, 2018

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Mar 15, 2018

☔️ The latest upstream changes (presumably #5176) made this pull request unmergeable. Please resolve the merge conflicts.

@soc soc force-pushed the soc:topic/standard-directories branch from a290016 to 0b350ad Mar 15, 2018

@aturon

This comment has been minimized.

Copy link
Member

aturon commented Mar 16, 2018

cc @nrc

@soc soc changed the title [WIP] Add support for platform-defined standard directories Add support for platform-defined standard directories Mar 17, 2018

@matklad
Copy link
Member

matklad left a comment

Overall, this looks great to me @soc!

One thing I am really worried about is how are we going to test this =/

  • we want to test it across at least mac/windows/linux (and probably on windows, there's also msvc/mingw/sigwin/linux subsystem axis?)
  • we want to test different fallback scenarios
  • we want to test this in conjunction with rustup

All this together implies to me that plain #[test] tests ain't gonna work here at all :(

I am thinking about a really heavy weight solution, like preparing docker images for different initial state of the machines, and then writing tests as bash scripts, which install rustup, create cargo project, etc. But, one does not simply create a docker image for windows I guess 🤷‍♂️ ?

It's also interesting that this PR actually does two things:

  • it refactors Cargo to support CargoDirs instead of monolithic CARGO_HOME.
  • it changes default locations for stuff, using directories and environment variables.

I wonder if it makes sense to split this over two pull request, and implement a refactoring first, while preserving current behavior fully. That way, we can separately check that the refactoring does not introduce regressions by itself, and then maximally concentrate on the fallback bits.

@rust-lang/cargo

@@ -122,7 +143,7 @@ pub fn install(
if installed_anything {
// Print a warning that if this directory isn't in PATH that they won't be
// able to run these commands.
let dst = metadata(opts.config, &root)?.parent().join("bin");
let dst = metadata(opts.config, &Filesystem::new(root.config_dir))?.parent().join("bin");

This comment has been minimized.

@matklad

matklad Mar 17, 2018

Member

This should be root.bin_dir I guess?

// This is written in the most straight-forward way possible, because it is
// hard as-is to understand all the different options, without trying to
// save lines of code.
pub fn cargo_dirs() -> CargoDirs {

This comment has been minimized.

@matklad

matklad Mar 17, 2018

Member

Yeah, new would looks slightly better I think, though current option is OK as well!

let home_dir = ::home::home_dir().ok_or_else(|| {
format_err!("Cargo couldn't find your home directory. \
This probably means that $HOME was not set.")
}).unwrap();

This comment has been minimized.

@matklad

matklad Mar 17, 2018

Member

I think unwraps may panic in some obscure, yet real world scenario, so we really need proper error handlng. Changing the return type to CargoResult<CargoDirs> and replacing .unwraps with ? should do the trick I think?

As for an example of weird scenario, there were bug in Cargo about ::std::env::current_exe call failing, because Cargo was executed in chroot without procfs :)

let mut cache_dir = cargo_dirs.cache_dir().to_path_buf();
let mut config_dir = cargo_dirs.config_dir().to_path_buf();
let mut data_dir = cargo_dirs.data_dir().to_path_buf();
// fixme: executable_dir only available on Linux, use data_dir on macOS and Windows?

This comment has been minimized.

@matklad

matklad Mar 17, 2018

Member

This should be figured out in the RFC thread perhaps?


// 3. .cargo exists
let legacy_cargo_dir = home_dir.join(".cargo");
if cargo_home_env.is_none() && legacy_cargo_dir.exists() {

This comment has been minimized.

@matklad

matklad Mar 17, 2018

Member

This should be strictly a fall-back perhaps? That is, if all variables are not set, we set all dirs from .cargo, in contrast with the current per-dir approach?

for current in paths::ancestors(pwd) {
let possible = current.join(".cargo").join("config");
for current in paths::ancestors(&dirs.current_dir) {
let possible = current.join(".cargo").join("config"); // fixme: what to do about this?

This comment has been minimized.

@matklad

matklad Mar 17, 2018

Member

Nothing to be done here? It's an explicit feture of Cargo that it looks for .cargo/config in call parent directories. This exists for per-project .cargo dirs. Or am I missing something here?

This comment has been minimized.

@soc

soc Mar 17, 2018

Author

Yes, I think you are right. I just wanted to be extra careful by marking all changes where I wasn't absolutely sure.

@soc

This comment has been minimized.

Copy link
Author

soc commented Mar 17, 2018

@matklad Yes, testing is also a concern for me.

I think as a first step it would really help to have a list of different configurations and scenarios, so it is at least possible to test everything in an organized fashion, even if it is done manually.

I'm not sure whether splitting things into two commits makes sense, I feel that having an additional transitory state would have some benefits, but would also add overhead that would outweigh the benefits.

Especially because a similar PR needs to be done for rustup. With changes in one commit we would only have to test 4 different setups, {cargo: pre-change, post-change} * {rustup pre-change, post-change}.

With an additional state in the middle, this would balloon up to 9 different setups, for which all configurations and scenarios need to be tested against.

@soc soc force-pushed the soc:topic/standard-directories branch 4 times, most recently from cc80517 to 3aca0cf Mar 18, 2018

@matklad matklad added the Rust-2018 label Mar 29, 2018

@matklad

This comment has been minimized.

Copy link
Member

matklad commented Apr 9, 2018

@soc have you managed to prepare a similar PR for rustup as well? I think updating rusupt would be a next step here, because we really do want to land changes to rustup and cargo simultaneously :)

@soc

This comment has been minimized.

Copy link
Author

soc commented Apr 10, 2018

@matklad Not yet, but here is my plan:

  • I could really need some review of the current code to get a better understanding if this is the way people want to go (especially in regard to Path vs. Filesystem).
  • I would be thankful if someone has some time to look into the remaining test failures with me. I can make tests pass, but sometimes I'm not sure my changes to the tests are testing what was originally intended.
  • Implement changes in rustup.
  • Write some documentation that explains the changes.
  • Fix the existing documentation.
    • Fix the official documentation.
    • Fix third-party tutorials, blogs, StackOverflow, etc.
    • Ping book authors.

@soc soc force-pushed the soc:topic/standard-directories branch 2 times, most recently from ff5e99a to bef5ad1 Apr 11, 2018

@@ -235,9 +256,9 @@ fn install_one(
// We have to check this again afterwards, but may as well avoid building
// anything if we're gonna throw it away anyway.
{
let metadata = metadata(config, root)?;
let metadata = metadata(config, &Filesystem::new(dirs.config_dir.clone()))?;

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

It feels like metadata could perhaps be a method of CargoInstallDirs?

pub cache_dir: Filesystem,
pub config_dir: Filesystem,
pub data_dir: PathBuf,
pub bin_dir: PathBuf,

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

It would be great to add short docstrings here, to explain which directory stores which data.

@@ -32,22 +33,119 @@ use util::Filesystem;

use self::ConfigValue as CV;

#[derive(Clone, Debug)]
pub struct CargoDirs {
pub home_dir: Filesystem,

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

We don't actually use this field I think?

let home_dir = ::home::home_dir().ok_or_else(|| {
format_err!("Cargo couldn't find your home directory. \
This probably means that $HOME was not set.")
})?;

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

So looks like we don't use home_dir for anything except fallback, so let's move this as far down as possible, so that we don't actually fail if HOME is not set, but explicit directories are! We might want to test this behavior as well: working without home directory is great for reproducible and isolated builds.

#[cfg(target_os = "macos")]
let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin"));
#[cfg(target_os = "windows")]
let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin"));

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

Let's extract this into a function

#[cfg(os = )]
fn bin_dir(dirs: &ProjectDirs) -> Option<PathBuf>
} else if let Some(val) = self.get_path("build.target-dir")? {
let val = self.cwd.join(val.val);
let val = self.dirs.current_dir.join(val.val);

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

Let's use self.cwd() here for future-proofing.

let home_path = self.home_path.clone().into_path_unlocked();
let credentials = home_path.join("credentials");
let config_path = self.dirs.config_dir.clone().into_path_unlocked();
let credentials = config_path.join("credentials");

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

Hm, and what is the "correct" place for credentials? config might be not the right place, because people sometimes publish it... I suggest raising this question on the RFC thread, if it wasn't raised already.

@@ -638,7 +736,7 @@ impl Config {
None => false,
};
let path = if maybe_relative {
self.cwd.join(tool_path)

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

.cws()

@@ -163,7 +163,7 @@ fn new_credentials_is_used_instead_old() {
execs().with_status(0),
);

let config = Config::new(Shell::new(), cargo_home(), cargo_home());
let config = Config::new(Shell::new(), CargoDirs::new().unwrap());

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

Here I think we want to point CargoDirs to CARGO_HOME inside test root.

One option is to provide another constructor for CargoDirs which accepts cargo_home: PathBuf. Another option is to modify the test such that it reads the config file directly. I am slightly in favor of the second approach.

@@ -910,7 +910,7 @@ fn build_script_needed_for_host_and_target() {
);
}

#[test]
//#[test]
fn build_deps_for_the_right_arch() {

This comment has been minimized.

@matklad

matklad Apr 11, 2018

Member

This actually passes for me locally

@soc soc force-pushed the soc:topic/standard-directories branch from 750edc1 to 5d807d8 Apr 11, 2018

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Apr 11, 2018

@ollie27

This comment has been minimized.

Copy link

ollie27 commented Apr 16, 2018

I think any kind of automatic migration for existing installs to the new directory scheme would be an unnecessary complication. If people already have .rustup and .cargo directories set up then rustup and cargo should continue to use them.

And, again, as a step toward merging cargo and rustup, you could just use reuse CargoDirs instead of creating a new RustupDirs.

I would propose the exact opposite, namely trying to separate rustup and cargo as much as possible. Currently rustup installs its binaries in cargo's bin directory but I don't know of any good reason for that. In the new scheme rustup should use its own bin directory (on Linux they would be the same anyway). It should be possible to use rustup without any kind of cargo home.

@brson

This comment has been minimized.

Copy link
Contributor

brson commented Apr 16, 2018

I believe that the intention is that CARGO_HOME env variable will always be the supported way to do just that

Hm, ok. That makes sense - rustup was probably going to have to figure out a place to stuff it's data under the cargo XDG directories anyway. Unfortunately, if CARGO_HOME indicates 'put all my data here', then there is no explicit signal for rustup to do that conversion. It also means the proposal I made (and that is reflected in this patch) that XDG takes precedence over CARGO_HOME is unworkable - CARGO_HOME must take ultimate precedence. Which also means the simple solution to make rustup work with legacy toolchains is also _probably unworkable.

@matklad

This comment has been minimized.

Copy link
Member

matklad commented Apr 16, 2018

Oh, at rust all hands we've decided that we don't want to merge rustup and cargo at least near/mid-term. However, we probably would move target management (i.e, downloading precompiled stdlib binaries for non-host plaftform) to Cargo, together with implementing an ability to cross-compile stdlib from sources on the spot.

@brson

This comment was marked as disruptive content.

Copy link
Contributor

brson commented Apr 16, 2018

So it's ok for rustup to disobey XDG even though everybody's all pissy that cargo doesn't? Doesn't really square.

@brson

This comment has been minimized.

Copy link
Contributor

brson commented Apr 16, 2018

I would propose the exact opposite, namely trying to separate rustup and cargo as much as possible. Currently rustup installs its binaries in cargo's bin directory but I don't know of any good reason for that.

Because that's where your rust binaries live and it's on the PATH. All Rust binaries in one place.

@brson

This comment has been minimized.

Copy link
Contributor

brson commented Apr 16, 2018

I'll think about it more, but if we really want CARGO_HOME to mean "everything here", then this patch has to change, and much of the strategy I wrote above, which I considered the simple way to modify rustup, is unworkable.

I also think you should go ahead and make rustup do the same thing with its directory structure while you are doing this - surely it'll just turn into another multi-year RFC fight for that if you don't.

@brson

This comment has been minimized.

Copy link
Contributor

brson commented Apr 16, 2018

FWIW the RFC prioritizes XDG dirs over CARGO_HOME. It does seem to be easier to deprecate CARGO_HOME - leave it for the legacy toolchains, merge cargo and rustup into the same directory structure, and introduce RUST_HOME to mean "everything here".

OK, I'll stop talking now.

@soc

This comment has been minimized.

Copy link
Author

soc commented Apr 16, 2018

@brson

I don't believe this is feasibile without seriously compromising what rustup is capable of. Would we really say "you can no longer use versions of Rust older than X?"; "you can't use new builds of Rust with an old rustup"?

But I'll be pretty disappointed if there's a big cutoff point where everything old breaks.

I think I worded it poorly, and believe we are on the same page considering the concerns you mentioned:

What I meant to say was that we have cut-off version, where everything before ends up in .cargo, and everything after uses the standard directories by default. I didn't want to imply that support for anything should be dropped, apologies if this caused some misunderstanding.

The idea was just to have that env var you suggested and keep testing things until everyone is confident that the changes are exactly what should be done, and have a separate switch date where the change is activated.

@brson

This comment has been minimized.

Copy link
Contributor

brson commented Apr 16, 2018

Oh, regarding needing a new RUST_HOME dir to create self-contained installs, that isn't actually necessary, if the rustup changes incorporate my suggestion of recording the directory layout at install time - selecting a "self-contained" can just mean create a directory somewhere and setting all the various directories to values inside of it.

The only downside is that you can't do it through a single environment variable.

On the other hand, if you were to migrate rustup to integrate closer with cargo, you could just put all the rustup data inside the cargo data dirs, and an install with CARGO_HOME set would put everything in CARGO_HOME (again assuming you record the directory layout on install).

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Apr 22, 2018

☔️ The latest upstream changes (presumably #5407) made this pull request unmergeable. Please resolve the merge conflicts.

@aturon

This comment has been minimized.

Copy link
Member

aturon commented Apr 28, 2018

@brson I was away on vacation during your recent comments on this thread, but wanted to circle back and address them from a procedural/community perspective.

The very long initial comment you made interspersed a lot of potentially useful technical detail with some unconstructive critiques and re-litigation of the RFC. That's not an appropriate way to engage on a PR thread, especially with the passive-aggressive notes dropped throughout.

I'm grateful that @soc was willing to take it all in stride, but it's important that we all put in the effort to keep discussion constructive and helpful -- especially when such commentary is coming from project leaders, past or present.

Add support for platform-defined standard directories
This change stops cargo from violating the operating system rules
regarding the placement of config, cache, ... directories on Linux,
macOS and Windows.

Existing directories and overrides are retained.

The precedence is as follows:

1) use the `CARGO_HOME` environment variable if it exists (legacy)
2) use `CARGO_CACHE_DIR`, `CARGO_CONFIG_DIR` etc. env vars if they exist
3) use the ~/.cargo directory if it exists (legacy)
4) follow operating system standards

A new cargo command, `dirs`,  is added, which can provide path
information to other command line tools.

Fixes:
  #1734
  #1976
  rust-lang/rust#12725

Addresses:
  rust-lang/rfcs#1615
  #148,
  #3981

@soc soc force-pushed the soc:topic/standard-directories branch from 4baa882 to d3ffb18 Apr 28, 2018

@soc

This comment has been minimized.

Copy link
Author

soc commented Jun 23, 2018

Just a heads up that I still intend working on this, I'm just blocked at the moment by some necessary changes in the supporting layers:

  • home_dir is broken and needs to be fixed: rust-lang/rust#51656
  • the APIs of dirs and directories need to change home_dir's result type from PathBuf to Option<PathBuf>
  • dirs and directories need to be relicensed to MIT/APL2
  • dirs and directories need a 1.0 release
@soc

This comment has been minimized.

Copy link
Author

soc commented Jul 5, 2018

I addressed the four issues mentioned above, so these blockers are gone.

There are some small cleanups in dirs/directories (factor out common code) that I want to ship, but apart from that I'm good to go now.

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jul 13, 2018

ping @matklad do you have a chance to take a look at this again?

@matklad

This comment has been minimized.

Copy link
Member

matklad commented Jul 13, 2018

Sure! This looks good to me in general, and I would be comfortable with merging it behind an unstable flag, if the following is done:

  • new behavior is actually hidden behind an unstable flag :-) The flag should be added over here. The end result should be that running cargo foo behaves exactly as today, and cargo -Z xdg foo uses new logic, with fallbacks and such.

  • directories dependency in Cargo.toml is updated to 1.0

  • There exact behavior is documented here. I understand that this is an unstable feature, but I think in this specific case it is important to get documentation together with the implementation, given the amount of subtitles, platform-specific behavior and the fact that RFC text lags behind the implementation. The docs would be in part a copy-paste of directories documentation of course, but it is important to keep this crate an implementation detail.

  • The behavior without the feature flag matches current behavior exactly. I'd love to help with auditing this, but I would prefer to do this after we have docs for the new behavior in place :)

@reitermarkus reitermarkus referenced this pull request Sep 4, 2018

Merged

rust/cargo: enable caching #4815

6 of 6 tasks complete
@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Oct 12, 2018

I'm going to close this because it's been stale and quiet for quite some time now unfortunately. The Cargo team is pretty tied up until after the 2018 edition, but I think we can perhaps look to help out integrating this and fixing remaining issues after the edition release.

@flying-sheep

This comment has been minimized.

Copy link

flying-sheep commented Jan 8, 2019

Hi! Rust 2018 is here, it’s a fresh new year! Time to get this rolling 😄

@flying-sheep

This comment has been minimized.

Copy link

flying-sheep commented Jan 8, 2019

Uh, how did I manage to unassign @matklad just by commenting‽

&& cargo_cache_env.is_none()
&& cargo_config_env.is_none()
&& cargo_data_env.is_none()
&& cargo_bin_env.is_none() {

This comment has been minimized.

@joshtriplett

joshtriplett Jan 9, 2019

Member

This doesn't seem to handle the case where some but not all of the environment variables are set.

@joshtriplett

This comment has been minimized.

Copy link
Member

joshtriplett commented Jan 9, 2019

@soc Let's see if we can get this revived.

I posted one comment for a corner case this doesn't seem to handle.

Could you please update this to fix the conflicts, and ensure that it still passes tests?

And could you please add tests for the various configuration cases (existing legacy configuration, existing XDG configuration, both, no existing configuration), to make sure they all work as expected?

@joshtriplett joshtriplett reopened this Jan 9, 2019

@dwijnand dwijnand removed the Rust-2018 label Feb 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment