-
Notifications
You must be signed in to change notification settings - Fork 104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rework planning to be based off the resolve tree and not directly pkgs #282
Conversation
@UebelAndre sorry if this stomps on anything you have in progress |
It undoubtably will 😢. While I do like the changes here, I would greatly appreciate it if I could get #276 through first. I've had to rebase and resolve conflicts a few times already If I could just get it reviewed, I can promise to have it updated as fast as I possibly can to make sure it gets closed as soon as possible 😃 |
435f71c
to
cbf3327
Compare
cbf3327
to
37d6cac
Compare
impl/src/planning/subplanners.rs
Outdated
@@ -279,6 +267,8 @@ impl<'planner> WorkspaceSubplanner<'planner> { | |||
} | |||
} | |||
|
|||
let mut root_ctx = None; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This bit, and its devil children I dont fully like, it would be nice maybe to just grab the root contect without having to discover it through resolving deps.
I could not think of or find a way to do it, its probably ok for now, and maybe a better future approach might be to move to guppy as a dep
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider including this as a comment inline (I'm sure as heck not going to remember this, and I think it's good guidance worth remembering)
impl/src/planning/subplanners.rs
Outdated
.into_iter() | ||
.map(|(target, deps)| { | ||
let target = target.unwrap().to_string(); | ||
let platform_targets = get_matching_bazel_triples(&target, &self.settings.targets)? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I noticed in working with this, the targetting functions are a bit scattered and not really that reused.
As such I removed a bunch of the additional functions, smushing things down towards a single function, and in avoiding allocation-o-rama (which we do a bit frequently in cargo-raze) made it return an iterator.
The part for the generation of bazel conditions I moved into the template itself, as it seemed like a lot of code was written just to essentially prepend a string onto the platform targets.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
allocation-o-rama? You mean making copies of stuff?
Yeah I'm usually too lazy to bother with reducing copies because (i've come to believe, perhaps erroneously that) it almost never matters.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I kinda ignore it in cargo-raze because the tool runs for me a few times a week, but when calling three consecutive functions seems to produce vector and string of each other it gets a bit much ;)
|
||
let all_skipped_deps = self | ||
.crate_settings | ||
.iter() | ||
.flat_map(|pkg| pkg.skipped_deps.iter()) | ||
.collect::<HashSet<_>>(); | ||
|
||
for dep_id in &self.node.dependencies { | ||
for dep in &self.node.deps { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This wonderful part of the metadata gives us both renames and targets. Irritatingly its still a little tricky to detect renames fully (we need this to finally deduce the aliases - see is_renamed
)
If rust-lang/cargo#7289 gets solved then a lot of the left-over rename detection code can get murdered
.push(buildable_dependency.clone()); | ||
} else { | ||
dep_set.build_deps.push(buildable_dependency.clone()); | ||
// TODO(GregBowyer): Reimplement what cargo does to detect bad renames |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The problem manifests as this
[dependencies]
bytes_new = { version = "0.3.0", package = "bytes" }
bytes_old = { version = "0.3.0", package = "bytes" }
Strictly this is an error. Right now cargo metadata will basically lose both bytes
deps from the resolve graph :gruntle: but a cargo build
will scream about having the same basic package with multiple names.
I dont have a good solution right now (I dont think we should go back to trying to essentially emulate the resolve graph like before), so I guess a todo :/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe better to reference a github issue here, but I don't feel strongly about this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 to TODO(#cargo-raze-issue): referencing this error.
impl/src/planning/subplanners.rs
Outdated
|
||
fn is_dep_targetted(&self, target: Option<&str>) -> bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NIT: Maybe add a comment in keeping with the other functions?
37d6cac
to
dc97c23
Compare
dc97c23
to
40d9daa
Compare
I can hold on your changes and work with that, I think we get #127 in then #276 and then see how bad that breaks this. |
Yeah, I can rebase on top of that when that goes in. |
40d9daa
to
3d5b94e
Compare
3d5b94e
to
ce72b01
Compare
@acmcarther thoughts? |
Was holding off on review because the other PR is in review, but that's taking a while (because I'm picky or something). I'll review this now. |
Nah we are all picky ;P I am sure this one is going to need more work anyhow there are things I cant think of a better way for that I dont like |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not ideal that the formatter changes are co-mingled with the behavior changes. A bit harder to review.
I am so unqualified to review these PRs at this point. I don't remember much about the implementation and anyway it has changed a lot since I wrote significant code here.
Oh well
@@ -36,8 +36,13 @@ use subplanners::WorkspaceSubplanner; | |||
/** A ready-to-be-rendered build, containing renderable context for each crate. */ | |||
#[derive(Debug)] | |||
pub struct PlannedBuild { | |||
/// The overall context for this workspace |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does "///" have any meaning beyond "//"? Is this like a rustdoc thing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rustdoc thing, but I was noticing in the other review most comments in carg-raze are f the form /** comment */
and so I should make the comments that I think (for the consistency)
impl/src/planning/subplanners.rs
Outdated
@@ -279,6 +267,8 @@ impl<'planner> WorkspaceSubplanner<'planner> { | |||
} | |||
} | |||
|
|||
let mut root_ctx = None; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider including this as a comment inline (I'm sure as heck not going to remember this, and I think it's good guidance worth remembering)
impl/src/planning/subplanners.rs
Outdated
.iter() | ||
.filter(|to_alias| to_alias.is_root_dependency && to_alias.lib_target_name.is_some()) | ||
.flat_map(|to_alias| { | ||
let pkg_name = sanitize_ident(&to_alias.pkg_name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these huge lambdas good rust style? I've been griping about them in @UebelAndre's PRs, but maybe I need to get over it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From what I have seen it typically is providing its not too crazy but this is pretty much at the limit and might need to just be a function directly.
impl/src/planning/subplanners.rs
Outdated
.into_iter() | ||
.map(|(target, deps)| { | ||
let target = target.unwrap().to_string(); | ||
let platform_targets = get_matching_bazel_triples(&target, &self.settings.targets)? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
allocation-o-rama? You mean making copies of stuff?
Yeah I'm usually too lazy to bother with reducing copies because (i've come to believe, perhaps erroneously that) it almost never matters.
.push(buildable_dependency.clone()); | ||
} else { | ||
dep_set.build_deps.push(buildable_dependency.clone()); | ||
// TODO(GregBowyer): Reimplement what cargo does to detect bad renames |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe better to reference a github issue here, but I don't feel strongly about this.
/// | ||
/// Currently cargo-metadata provides rename detection in a few places, we take the names | ||
/// from the resolution of a package | ||
fn is_renamed(&self, dep_package: &Package) -> bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it make sense to precompute some of these new properties? I'm lost in the details here, but part of the idea with the crate catalog was to make this type of lookup more efficient.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Humm yeah might work, and might be a good way to make the find the root sillyness go away.
I would only bother with the first commit in the PR, I can just pull the second commit (the reformatting) and we can do a round of that after all crazy PRs get merged |
Crazy PR has been merged. No more crazy PRs from me for the foreseeable future 😅 |
In hindsight, this should have been merged before any crazy PRs 😞 Sorry about pushing for mine over this one. @GregBowyer do you have bandwidth in the foreseeable future to revive this effort 😅? edit: Attempted to do the rebase myself and it's become quite hairy. I think the best course of action would be to more or less start from a clean slate. I think if you drop all changes made to tests then the merge isn't so bad. I've since updated how metadata is mocked for tests to make them deterministic and not require internet. That seems to be the biggest source of conflicts. |
I have time to pick this up again |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple of comments and suggestions but otherwise sounds pretty good. I'm going to need to take another day to page some of the nuances here back into my brain and do another pass but still pretty excited for the direction!
impl/src/planning/subplanners.rs
Outdated
.iter() | ||
.filter(|to_alias| to_alias.lib_target_name.is_some()) | ||
.flat_map(|to_alias| { | ||
let pkg_name = util::sanitize_ident(&to_alias.pkg_name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel like the sanitize is worth keeping but wanted to share 9d74090 here in case that's relevant. I think sanatize_indent
might need to be updated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops I thought I factored that in when I rebased so good catch.
Since migrating most things to Bazel, ensuring the examples are up to date has not been added to CI. Would you be able to run |
I will pull in the comments (all sane I think) make sure that test failure is not something sinister. I think I should also take another scan through the bug list to see if there are any other bugs that this solves. |
8edc09d
to
2063fd5
Compare
No need it's not going towards the templates so won't serialise
On Fri, Apr 23, 2021, at 3:08 PM, UebelAndre wrote:
***@***.**** commented on this pull request.
In impl/src/util.rs <#282 (comment)>:
> @@ -49,20 +51,33 @@ static SUPPORTED_PLATFORM_TRIPLES: &[&str] = &[
"x86_64-unknown-freebsd",
];
+#[derive(Debug, PartialEq, Eq, Copy, Clone)]
…
Specifically asking in regards to #415 <#415> but that's likely out of scope for that change and something I'll go update 😄
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#282 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA6X3YKL6ITTTIRWFU5P5TTKHVVLANCNFSM4TNHIF4A>.
|
There are a range of subtle and irritating bugs that stem from attempting to work with renames from the package set provided by cargo metadata. Fortunately, recent versions of cargo metadata provide a limited version of the resolve nodes, both as the original node ids and as a newer set providing both rename and targeting information. As such we can (hopefully) simplify crate -> dep node resolution, remove a bunch of subtle aliasing bugs. Fixes google#241, fixes google#269, fixes google#270, resolves google#144, resolves google#187
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just another small pass. I'll take some time over the weekend to do a thorough review. The thing I want to look into is the test coverage for this. What's your confidence level that there's enough here to prevent a regression? (again, need to look, there might be enough already but I don't have that context paged in my brain right now 😅)
if let Some(platform_details) = &self.platform_details { | ||
if let Some(settings_target) = &self.settings.target { | ||
let platform = Platform::from_str(&target_str)?; | ||
fn is_dep_targetted(&self, target: Option<&String>) -> bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Can you add a small doc for this?
|
||
Ok((self._produce_deps(&default_deps)?, targeted_set)) | ||
} | ||
fn process_dep( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: add a doc here as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, a few more nits and one or two questions. But otherwise I think this is looking really good! 😄
@@ -438,7 +438,7 @@ impl RazeMetadataFetcher { | |||
let version = info.req(); | |||
let src_dir = self.fetch_crate_src(cargo_dir.as_ref(), &name, version)?; | |||
checksums.insert( | |||
package_ident(name, version), | |||
package_ident(name, &version), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was there an issue with this before?
@@ -155,7 +128,7 @@ impl<'planner> WorkspaceSubplanner<'planner> { | |||
&self, | |||
node: &Node, | |||
catalog: &CrateCatalog, | |||
) -> Option<Result<CrateContext>> { | |||
) -> Option<Result<(CrateContext, bool)>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this not information that already lives (or could already live) in CrateContext
? Admittedly that struct could use some cleaning but whether or not the crate is a workspace member seems like it makes sense to store in the context.
cargo-raze/impl/src/context.rs
Lines 116 to 153 in ee2effe
#[derive(Debug, Clone, Serialize)] | |
pub struct CrateContext { | |
pub pkg_name: String, | |
pub pkg_version: Version, | |
pub edition: String, | |
pub raze_settings: CrateSettings, | |
pub canonical_additional_build_file: Option<PathBuf>, | |
pub default_deps: CrateDependencyContext, | |
pub targeted_deps: Vec<CrateTargetedDepContext>, | |
pub license: LicenseData, | |
pub features: Vec<String>, | |
pub workspace_path_to_crate: String, | |
pub workspace_member_dependents: Vec<PathBuf>, | |
pub workspace_member_dev_dependents: Vec<PathBuf>, | |
pub workspace_member_build_dependents: Vec<PathBuf>, | |
pub is_workspace_member_dependency: bool, | |
pub is_binary_dependency: bool, | |
pub targets: Vec<BuildableTarget>, | |
pub build_script_target: Option<BuildableTarget>, | |
pub links: Option<String>, | |
pub source_details: SourceDetails, | |
pub sha256: Option<String>, | |
pub registry_url: String, | |
// TODO(acmcarther): This is used internally by renderer to know where to put the build file. It | |
// probably should live somewhere else. Renderer params (separate from context) should live | |
// somewhere more explicit. | |
// | |
// I'm punting on this now because this requires a more serious look at the renderer code. | |
pub expected_build_path: String, | |
// The name of the main lib target for this crate (if present). | |
// Currently only one such lib can exist per crate. | |
pub lib_target_name: Option<String>, | |
// This field tracks whether or not the lib target of `lib_target_name` | |
// is a proc_macro library or not. | |
pub is_proc_macro: bool, | |
} |
{%- for condition in targeted_dep.conditions %} | ||
"{{ condition }}", | ||
{%- for platform_target in targeted_dep.platform_targets %} | ||
"@rules_rust//rust/platform:{{platform_target}}", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"@rules_rust//rust/platform:{{platform_target}}", | |
"@{{ rust_rules_workspace_name }}//rust/platform:{{platform_target}}", |
{%- for condition in targeted_dep.conditions %} | ||
"{{ condition }}", | ||
{%- for platform_target in targeted_dep.platform_targets %} | ||
"@rules_rust//rust/platform:{{platform_target}}", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"@rules_rust//rust/platform:{{platform_target}}", | |
"@{{ rust_rules_workspace_name }}//rust/platform:{{platform_target}}", |
@@ -225,7 +225,7 @@ pub fn mock_remote_crate<'server>( | |||
let enc = flate2::write::GzEncoder::new(tar_gz, Compression::default()); | |||
let mut tar = tar::Builder::new(enc); | |||
tar | |||
.append_dir_all(package_ident(name, version), dir.as_ref().join("archive")) | |||
.append_dir_all(package_ident(name, &version), dir.as_ref().join("archive")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another question about using a reference. Just curious.
@acmcarther this is looking near completion. Don't know if you're around to review it but it'd be great to get in. If not, maybe @dfreese can take a look? It'd probably be good to get your eyes on it too just so you're aware of the changes (sorry for the ping otherwise 😅) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haven't had a chance to make a full pass at this, but had one major question, based on past experience.
pub workspace_context: WorkspaceContext, | ||
/// The creates to build for |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// The creates to build for | |
/// The crates to build for |
// We only want equality for the first member as it can have multiple names pointing at it. | ||
impl Hash for DependencyAlias { | ||
fn hash<H: Hasher>(&self, state: &mut H) { | ||
self.target.hash(state); | ||
} | ||
} | ||
|
||
impl PartialEq for DependencyAlias { | ||
fn eq(&self, other: &Self) -> bool { | ||
self.target == other.target | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've been bitten by some nasty bugs where hash or comparison functions don't cover the all of the data in the struct.
Is there another way to structure this, perhaps by using map from target string to alias (or &DependencyAlias)?
.push(buildable_dependency.clone()); | ||
} else { | ||
dep_set.build_deps.push(buildable_dependency.clone()); | ||
// TODO(GregBowyer): Reimplement what cargo does to detect bad renames |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 to TODO(#cargo-raze-issue): referencing this error.
@GregBowyer friendly ping. I feel this is so close to going in 😄 |
@GregBowyer @dfreese another friendly ping 😅 Would love to get this merged and am wondering what outstanding action items are. Happy to help move this forward however I can. |
Friendly ping again. Happy to help however I can to get this in but would love to see this merged 😄 |
.find(|target| target.kind == "lib" || target.kind == "proc-macro") | ||
.map(|target| target.name.clone()); | ||
|
||
let is_proc_macro = targets.iter().any(|target| target.kind == "proc_macro"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let is_proc_macro = targets.iter().any(|target| target.kind == "proc_macro"); | |
let is_proc_macro = targets.iter().any(|target| target.kind == "proc-macro"); |
Aren't string literals fun? 😄
@dfreese is this not something you could apply the suggestions for and accept as is? I think it's in a good spot. |
Not having all fields included as a part of a hash/comparison function causes some incredibly hard bugs to figure out, so I'd want to see if there's an alternative solution to that first. |
Is that not something you or I could patch in this PR or in a follow up? I think @GregBowyer has put fourth a huge amount of effort implementing this PR at least 3 times. I'd hate to have this reimplemented a 4th time and there are other changes I think should go in. |
This fixes `crates.bzl` never having any crates defined in the `_PROC_MACRO_DEPENDENCIES` map. This fixes a regression from #425 (or #282) and regenerates cargo-raze outputs (which should have been done on the merged commit). This also fixes a regression in how BUILD files were rendered, resulting in buildifier defects. I cherry picked a change I had an another branch which is ahead of `main` and contains the fix. This change includes #393 Finally, this fixes another regression where `extra_aliased_targets` was being ignored.
There are a range of subtle and irritating bugs that stem from
attempting to work with renames from the package set provided by cargo
metadata.
Fortunately, recent versions of cargo metadata provide a limited version
of the resolve nodes, both as the original node ids and as a newer set
providing both rename and targetting information.
As such we can (hopefully) simplify crate -> dep node resolution, remove
a bunch of subtle aliasing bugs.
Fixes #241, fixes #269, fixes #270, resolves #144, resolves #187