Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.
Sign upSupport platform-defined standard directories #5183
Conversation
rust-highfive
assigned
alexcrichton
Mar 15, 2018
This comment has been minimized.
This comment has been minimized.
rust-highfive
commented
Mar 15, 2018
|
Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon. If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes. Please see the contribution instructions for more information. |
This comment has been minimized.
This comment has been minimized.
|
Hi everyone, I would love to get some feedback on this (and I couldn't reach anyone at #cargo). |
soc
force-pushed the
soc:topic/standard-directories
branch
from
6065027
to
390edbb
Mar 15, 2018
This comment has been minimized.
This comment has been minimized.
|
New Examples:
|
soc
force-pushed the
soc:topic/standard-directories
branch
from
390edbb
to
c192c7f
Mar 15, 2018
soc
reviewed
Mar 15, 2018
| // This is written in the most straight-forward way possible, because it is | ||
| // hard as-is to understand all the different options, without trying to | ||
| // save lines of code. | ||
| pub fn cargo_dirs() -> CargoDirs { |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
matklad
Mar 17, 2018
Member
Yeah, new would looks slightly better I think, though current option is OK as well!
soc
force-pushed the
soc:topic/standard-directories
branch
from
c192c7f
to
a290016
Mar 15, 2018
This comment has been minimized.
This comment has been minimized.
|
|
soc
force-pushed the
soc:topic/standard-directories
branch
from
a290016
to
0b350ad
Mar 15, 2018
This comment has been minimized.
This comment has been minimized.
|
cc @nrc |
soc
changed the title
[WIP] Add support for platform-defined standard directories
Add support for platform-defined standard directories
Mar 17, 2018
matklad
reviewed
Mar 17, 2018
|
Overall, this looks great to me @soc! One thing I am really worried about is how are we going to test this =/
All this together implies to me that plain I am thinking about a really heavy weight solution, like preparing docker images for different initial state of the machines, and then writing tests as bash scripts, which install rustup, create cargo project, etc. But, one does not simply create a docker image for windows I guess It's also interesting that this PR actually does two things:
I wonder if it makes sense to split this over two pull request, and implement a refactoring first, while preserving current behavior fully. That way, we can separately check that the refactoring does not introduce regressions by itself, and then maximally concentrate on the fallback bits. |
| @@ -122,7 +143,7 @@ pub fn install( | |||
| if installed_anything { | |||
| // Print a warning that if this directory isn't in PATH that they won't be | |||
| // able to run these commands. | |||
| let dst = metadata(opts.config, &root)?.parent().join("bin"); | |||
| let dst = metadata(opts.config, &Filesystem::new(root.config_dir))?.parent().join("bin"); | |||
This comment has been minimized.
This comment has been minimized.
| // This is written in the most straight-forward way possible, because it is | ||
| // hard as-is to understand all the different options, without trying to | ||
| // save lines of code. | ||
| pub fn cargo_dirs() -> CargoDirs { |
This comment has been minimized.
This comment has been minimized.
matklad
Mar 17, 2018
Member
Yeah, new would looks slightly better I think, though current option is OK as well!
| let home_dir = ::home::home_dir().ok_or_else(|| { | ||
| format_err!("Cargo couldn't find your home directory. \ | ||
| This probably means that $HOME was not set.") | ||
| }).unwrap(); |
This comment has been minimized.
This comment has been minimized.
matklad
Mar 17, 2018
Member
I think unwraps may panic in some obscure, yet real world scenario, so we really need proper error handlng. Changing the return type to CargoResult<CargoDirs> and replacing .unwraps with ? should do the trick I think?
As for an example of weird scenario, there were bug in Cargo about ::std::env::current_exe call failing, because Cargo was executed in chroot without procfs :)
| let mut cache_dir = cargo_dirs.cache_dir().to_path_buf(); | ||
| let mut config_dir = cargo_dirs.config_dir().to_path_buf(); | ||
| let mut data_dir = cargo_dirs.data_dir().to_path_buf(); | ||
| // fixme: executable_dir only available on Linux, use data_dir on macOS and Windows? |
This comment has been minimized.
This comment has been minimized.
|
|
||
| // 3. .cargo exists | ||
| let legacy_cargo_dir = home_dir.join(".cargo"); | ||
| if cargo_home_env.is_none() && legacy_cargo_dir.exists() { |
This comment has been minimized.
This comment has been minimized.
matklad
Mar 17, 2018
Member
This should be strictly a fall-back perhaps? That is, if all variables are not set, we set all dirs from .cargo, in contrast with the current per-dir approach?
| for current in paths::ancestors(pwd) { | ||
| let possible = current.join(".cargo").join("config"); | ||
| for current in paths::ancestors(&dirs.current_dir) { | ||
| let possible = current.join(".cargo").join("config"); // fixme: what to do about this? |
This comment has been minimized.
This comment has been minimized.
matklad
Mar 17, 2018
Member
Nothing to be done here? It's an explicit feture of Cargo that it looks for .cargo/config in call parent directories. This exists for per-project .cargo dirs. Or am I missing something here?
This comment has been minimized.
This comment has been minimized.
soc
Mar 17, 2018
Author
Yes, I think you are right. I just wanted to be extra careful by marking all changes where I wasn't absolutely sure.
This comment has been minimized.
This comment has been minimized.
|
@matklad Yes, testing is also a concern for me. I think as a first step it would really help to have a list of different configurations and scenarios, so it is at least possible to test everything in an organized fashion, even if it is done manually. I'm not sure whether splitting things into two commits makes sense, I feel that having an additional transitory state would have some benefits, but would also add overhead that would outweigh the benefits. Especially because a similar PR needs to be done for rustup. With changes in one commit we would only have to test 4 different setups, {cargo: pre-change, post-change} * {rustup pre-change, post-change}. With an additional state in the middle, this would balloon up to 9 different setups, for which all configurations and scenarios need to be tested against. |
soc
force-pushed the
soc:topic/standard-directories
branch
4 times, most recently
from
cc80517
to
3aca0cf
Mar 18, 2018
Eh2406
referenced this pull request
Mar 28, 2018
Open
Support the XDG Base Directory Specification. #1734
matklad
added
the
Rust-2018
label
Mar 29, 2018
This comment has been minimized.
This comment has been minimized.
|
@soc have you managed to prepare a similar PR for rustup as well? I think updating rusupt would be a next step here, because we really do want to land changes to rustup and cargo simultaneously :) |
This comment has been minimized.
This comment has been minimized.
|
@matklad Not yet, but here is my plan:
|
soc
force-pushed the
soc:topic/standard-directories
branch
2 times, most recently
from
ff5e99a
to
bef5ad1
Apr 11, 2018
matklad
reviewed
Apr 11, 2018
| @@ -235,9 +256,9 @@ fn install_one( | |||
| // We have to check this again afterwards, but may as well avoid building | |||
| // anything if we're gonna throw it away anyway. | |||
| { | |||
| let metadata = metadata(config, root)?; | |||
| let metadata = metadata(config, &Filesystem::new(dirs.config_dir.clone()))?; | |||
This comment has been minimized.
This comment has been minimized.
| pub cache_dir: Filesystem, | ||
| pub config_dir: Filesystem, | ||
| pub data_dir: PathBuf, | ||
| pub bin_dir: PathBuf, |
This comment has been minimized.
This comment has been minimized.
matklad
Apr 11, 2018
Member
It would be great to add short docstrings here, to explain which directory stores which data.
| @@ -32,22 +33,119 @@ use util::Filesystem; | |||
|
|
|||
| use self::ConfigValue as CV; | |||
|
|
|||
| #[derive(Clone, Debug)] | |||
| pub struct CargoDirs { | |||
| pub home_dir: Filesystem, | |||
This comment has been minimized.
This comment has been minimized.
| let home_dir = ::home::home_dir().ok_or_else(|| { | ||
| format_err!("Cargo couldn't find your home directory. \ | ||
| This probably means that $HOME was not set.") | ||
| })?; |
This comment has been minimized.
This comment has been minimized.
matklad
Apr 11, 2018
Member
So looks like we don't use home_dir for anything except fallback, so let's move this as far down as possible, so that we don't actually fail if HOME is not set, but explicit directories are! We might want to test this behavior as well: working without home directory is great for reproducible and isolated builds.
| #[cfg(target_os = "macos")] | ||
| let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin")); | ||
| #[cfg(target_os = "windows")] | ||
| let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin")); |
This comment has been minimized.
This comment has been minimized.
matklad
Apr 11, 2018
Member
Let's extract this into a function
#[cfg(os = )]
fn bin_dir(dirs: &ProjectDirs) -> Option<PathBuf>
| } else if let Some(val) = self.get_path("build.target-dir")? { | ||
| let val = self.cwd.join(val.val); | ||
| let val = self.dirs.current_dir.join(val.val); |
This comment has been minimized.
This comment has been minimized.
| let home_path = self.home_path.clone().into_path_unlocked(); | ||
| let credentials = home_path.join("credentials"); | ||
| let config_path = self.dirs.config_dir.clone().into_path_unlocked(); | ||
| let credentials = config_path.join("credentials"); |
This comment has been minimized.
This comment has been minimized.
matklad
Apr 11, 2018
Member
Hm, and what is the "correct" place for credentials? config might be not the right place, because people sometimes publish it... I suggest raising this question on the RFC thread, if it wasn't raised already.
| @@ -638,7 +736,7 @@ impl Config { | |||
| None => false, | |||
| }; | |||
| let path = if maybe_relative { | |||
| self.cwd.join(tool_path) | |||
This comment has been minimized.
This comment has been minimized.
matklad
reviewed
Apr 11, 2018
| @@ -163,7 +163,7 @@ fn new_credentials_is_used_instead_old() { | |||
| execs().with_status(0), | |||
| ); | |||
|
|
|||
| let config = Config::new(Shell::new(), cargo_home(), cargo_home()); | |||
| let config = Config::new(Shell::new(), CargoDirs::new().unwrap()); | |||
This comment has been minimized.
This comment has been minimized.
matklad
Apr 11, 2018
Member
Here I think we want to point CargoDirs to CARGO_HOME inside test root.
One option is to provide another constructor for CargoDirs which accepts cargo_home: PathBuf. Another option is to modify the test such that it reads the config file directly. I am slightly in favor of the second approach.
matklad
reviewed
Apr 11, 2018
| @@ -910,7 +910,7 @@ fn build_script_needed_for_host_and_target() { | |||
| ); | |||
| } | |||
|
|
|||
| #[test] | |||
| //#[test] | |||
| fn build_deps_for_the_right_arch() { | |||
This comment has been minimized.
This comment has been minimized.
soc
force-pushed the
soc:topic/standard-directories
branch
from
750edc1
to
5d807d8
Apr 11, 2018
This comment has been minimized.
This comment has been minimized.
|
r? @matklad |
This comment was marked as disruptive content.
This comment was marked as disruptive content.
|
So it's ok for rustup to disobey XDG even though everybody's all pissy that cargo doesn't? Doesn't really square. |
This comment has been minimized.
This comment has been minimized.
Because that's where your rust binaries live and it's on the PATH. All Rust binaries in one place. |
This comment has been minimized.
This comment has been minimized.
|
I'll think about it more, but if we really want I also think you should go ahead and make rustup do the same thing with its directory structure while you are doing this - surely it'll just turn into another multi-year RFC fight for that if you don't. |
This comment has been minimized.
This comment has been minimized.
|
FWIW the RFC prioritizes XDG dirs over OK, I'll stop talking now. |
This comment has been minimized.
This comment has been minimized.
I think I worded it poorly, and believe we are on the same page considering the concerns you mentioned: What I meant to say was that we have cut-off version, where everything before ends up in .cargo, and everything after uses the standard directories by default. I didn't want to imply that support for anything should be dropped, apologies if this caused some misunderstanding. The idea was just to have that env var you suggested and keep testing things until everyone is confident that the changes are exactly what should be done, and have a separate switch date where the change is activated. |
This comment has been minimized.
This comment has been minimized.
|
Oh, regarding needing a new The only downside is that you can't do it through a single environment variable. On the other hand, if you were to migrate rustup to integrate closer with cargo, you could just put all the rustup data inside the cargo data dirs, and an install with |
This comment has been minimized.
This comment has been minimized.
|
|
This comment has been minimized.
This comment has been minimized.
|
@brson I was away on vacation during your recent comments on this thread, but wanted to circle back and address them from a procedural/community perspective. The very long initial comment you made interspersed a lot of potentially useful technical detail with some unconstructive critiques and re-litigation of the RFC. That's not an appropriate way to engage on a PR thread, especially with the passive-aggressive notes dropped throughout. I'm grateful that @soc was willing to take it all in stride, but it's important that we all put in the effort to keep discussion constructive and helpful -- especially when such commentary is coming from project leaders, past or present. |
soc
force-pushed the
soc:topic/standard-directories
branch
from
4baa882
to
d3ffb18
Apr 28, 2018
soc
referenced this pull request
Jun 21, 2018
Merged
Deprecate `std::env::home_dir` and fix incorrect documentation #51656
This comment has been minimized.
This comment has been minimized.
|
Just a heads up that I still intend working on this, I'm just blocked at the moment by some necessary changes in the supporting layers:
|
This comment has been minimized.
This comment has been minimized.
|
I addressed the four issues mentioned above, so these blockers are gone. There are some small cleanups in |
This comment has been minimized.
This comment has been minimized.
|
ping @matklad do you have a chance to take a look at this again? |
This comment has been minimized.
This comment has been minimized.
|
Sure! This looks good to me in general, and I would be comfortable with merging it behind an unstable flag, if the following is done:
|
This comment has been minimized.
This comment has been minimized.
|
I'm going to close this because it's been stale and quiet for quite some time now unfortunately. The Cargo team is pretty tied up until after the 2018 edition, but I think we can perhaps look to help out integrating this and fixing remaining issues after the edition release. |
alexcrichton
closed this
Oct 12, 2018
This comment has been minimized.
This comment has been minimized.
flying-sheep
commented
Jan 8, 2019
|
Hi! Rust 2018 is here, it’s a fresh new year! Time to get this rolling |
flying-sheep
unassigned
matklad
Jan 8, 2019
This comment has been minimized.
This comment has been minimized.
flying-sheep
commented
Jan 8, 2019
|
Uh, how did I manage to unassign @matklad just by commenting‽ |
joshtriplett
reviewed
Jan 9, 2019
| && cargo_cache_env.is_none() | ||
| && cargo_config_env.is_none() | ||
| && cargo_data_env.is_none() | ||
| && cargo_bin_env.is_none() { |
This comment has been minimized.
This comment has been minimized.
joshtriplett
Jan 9, 2019
Member
This doesn't seem to handle the case where some but not all of the environment variables are set.
This comment has been minimized.
This comment has been minimized.
|
@soc Let's see if we can get this revived. I posted one comment for a corner case this doesn't seem to handle. Could you please update this to fix the conflicts, and ensure that it still passes tests? And could you please add tests for the various configuration cases (existing legacy configuration, existing XDG configuration, both, no existing configuration), to make sure they all work as expected? |
joshtriplett
reopened this
Jan 9, 2019
dwijnand
removed
the
Rust-2018
label
Feb 11, 2019
nrc
assigned
joshtriplett
Mar 6, 2019
This comment has been minimized.
This comment has been minimized.
FranklinYu
commented
Mar 31, 2019
|
I don’t like the solution for macOS. See soc/directories-rs#47. |
This comment has been minimized.
This comment has been minimized.
|
@joshtriplett Sorry for the late reply. I think my changes so far are flawed in the sense that all the existing logic should be retained as-is, otherwise it becomes really really hard to make sure the current behavior is the same without flipping the hypothetical switch to the new structure. Sadly, at the moment I have little time for this (and the website shenanigans didn't help either). I can offer some advice, though: As a first step, identify each an every place in cargo, rustup, etc. that uses the current structure and add an explicit branch with a check for the proposed Then test that all tools reach the right branch when the flag is set/not set. Only at this point it makes sense to even start working on the new structure |
soc commentedMar 15, 2018
•
edited
This change stops cargo from violating the operating system rules
regarding the placement of config, cache, ... directories on Linux,
macOS and Windows.
Existing directories and overrides are retained.
The precedence is as follows:
CARGO_HOMEenvironment variable if it exists (legacy)CARGO_CACHE_DIR,CARGO_CONFIG_DIRetc. env vars if they existA new cargo command,
dirs, is added, which can provide pathinformation to other command line tools.
Fixes:
#1734
#1976
rust-lang/rust#12725
Addresses:
rust-lang/rfcs#1615
#148,
#3981