Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework planning to be based off the resolve tree and not directly pkgs #282

Closed
wants to merge 1 commit into from

Conversation

GregBowyer
Copy link
Contributor

There are a range of subtle and irritating bugs that stem from
attempting to work with renames from the package set provided by cargo
metadata.

Fortunately, recent versions of cargo metadata provide a limited version
of the resolve nodes, both as the original node ids and as a newer set
providing both rename and targetting information.

As such we can (hopefully) simplify crate -> dep node resolution, remove
a bunch of subtle aliasing bugs.

Fixes #241, fixes #269, fixes #270, resolves #144, resolves #187

@GregBowyer
Copy link
Contributor Author

@UebelAndre sorry if this stomps on anything you have in progress

@UebelAndre
Copy link
Collaborator

It undoubtably will 😢. While I do like the changes here, I would greatly appreciate it if I could get #276 through first. I've had to rebase and resolve conflicts a few times already ☹️

If I could just get it reviewed, I can promise to have it updated as fast as I possibly can to make sure it gets closed as soon as possible 😃

@@ -279,6 +267,8 @@ impl<'planner> WorkspaceSubplanner<'planner> {
}
}

let mut root_ctx = None;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bit, and its devil children I dont fully like, it would be nice maybe to just grab the root contect without having to discover it through resolving deps.

I could not think of or find a way to do it, its probably ok for now, and maybe a better future approach might be to move to guppy as a dep

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider including this as a comment inline (I'm sure as heck not going to remember this, and I think it's good guidance worth remembering)

.into_iter()
.map(|(target, deps)| {
let target = target.unwrap().to_string();
let platform_targets = get_matching_bazel_triples(&target, &self.settings.targets)?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed in working with this, the targetting functions are a bit scattered and not really that reused.

As such I removed a bunch of the additional functions, smushing things down towards a single function, and in avoiding allocation-o-rama (which we do a bit frequently in cargo-raze) made it return an iterator.

The part for the generation of bazel conditions I moved into the template itself, as it seemed like a lot of code was written just to essentially prepend a string onto the platform targets.

Copy link
Member

@acmcarther acmcarther Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

allocation-o-rama? You mean making copies of stuff?

Yeah I'm usually too lazy to bother with reducing copies because (i've come to believe, perhaps erroneously that) it almost never matters.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I kinda ignore it in cargo-raze because the tool runs for me a few times a week, but when calling three consecutive functions seems to produce vector and string of each other it gets a bit much ;)


let all_skipped_deps = self
.crate_settings
.iter()
.flat_map(|pkg| pkg.skipped_deps.iter())
.collect::<HashSet<_>>();

for dep_id in &self.node.dependencies {
for dep in &self.node.deps {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wonderful part of the metadata gives us both renames and targets. Irritatingly its still a little tricky to detect renames fully (we need this to finally deduce the aliases - see is_renamed)

If rust-lang/cargo#7289 gets solved then a lot of the left-over rename detection code can get murdered

.push(buildable_dependency.clone());
} else {
dep_set.build_deps.push(buildable_dependency.clone());
// TODO(GregBowyer): Reimplement what cargo does to detect bad renames
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem manifests as this

[dependencies]
bytes_new = { version = "0.3.0", package = "bytes" }
bytes_old = { version = "0.3.0", package = "bytes" }

Strictly this is an error. Right now cargo metadata will basically lose both bytes deps from the resolve graph :gruntle: but a cargo build will scream about having the same basic package with multiple names.

I dont have a good solution right now (I dont think we should go back to trying to essentially emulate the resolve graph like before), so I guess a todo :/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better to reference a github issue here, but I don't feel strongly about this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to TODO(#cargo-raze-issue): referencing this error.


fn is_dep_targetted(&self, target: Option<&str>) -> bool {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Maybe add a comment in keeping with the other functions?

@GregBowyer
Copy link
Contributor Author

It undoubtably will cry. While I do like the changes here, I would greatly appreciate it if I could get #276 through first. I've had to rebase and resolve conflicts a few times already frowning_face

If I could just get it reviewed, I can promise to have it updated as fast as I possibly can to make sure it gets closed as soon as possible smiley

I can hold on your changes and work with that, I think we get #127 in then #276 and then see how bad that breaks this.

@UebelAndre
Copy link
Collaborator

Yeah, I can rebase on top of that when that goes in.

@UebelAndre
Copy link
Collaborator

#276 is rebased on top of #127 now

@GregBowyer
Copy link
Contributor Author

@acmcarther thoughts?

@acmcarther
Copy link
Member

Was holding off on review because the other PR is in review, but that's taking a while (because I'm picky or something).

I'll review this now.

@GregBowyer
Copy link
Contributor Author

Nah we are all picky ;P

I am sure this one is going to need more work anyhow there are things I cant think of a better way for that I dont like

Copy link
Member

@acmcarther acmcarther left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not ideal that the formatter changes are co-mingled with the behavior changes. A bit harder to review.

I am so unqualified to review these PRs at this point. I don't remember much about the implementation and anyway it has changed a lot since I wrote significant code here.

Oh well

@@ -36,8 +36,13 @@ use subplanners::WorkspaceSubplanner;
/** A ready-to-be-rendered build, containing renderable context for each crate. */
#[derive(Debug)]
pub struct PlannedBuild {
/// The overall context for this workspace
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does "///" have any meaning beyond "//"? Is this like a rustdoc thing?

Copy link
Contributor Author

@GregBowyer GregBowyer Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rustdoc thing, but I was noticing in the other review most comments in carg-raze are f the form /** comment */ and so I should make the comments that I think (for the consistency)

@@ -279,6 +267,8 @@ impl<'planner> WorkspaceSubplanner<'planner> {
}
}

let mut root_ctx = None;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider including this as a comment inline (I'm sure as heck not going to remember this, and I think it's good guidance worth remembering)

.iter()
.filter(|to_alias| to_alias.is_root_dependency && to_alias.lib_target_name.is_some())
.flat_map(|to_alias| {
let pkg_name = sanitize_ident(&to_alias.pkg_name);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these huge lambdas good rust style? I've been griping about them in @UebelAndre's PRs, but maybe I need to get over it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I have seen it typically is providing its not too crazy but this is pretty much at the limit and might need to just be a function directly.

.into_iter()
.map(|(target, deps)| {
let target = target.unwrap().to_string();
let platform_targets = get_matching_bazel_triples(&target, &self.settings.targets)?
Copy link
Member

@acmcarther acmcarther Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

allocation-o-rama? You mean making copies of stuff?

Yeah I'm usually too lazy to bother with reducing copies because (i've come to believe, perhaps erroneously that) it almost never matters.

.push(buildable_dependency.clone());
} else {
dep_set.build_deps.push(buildable_dependency.clone());
// TODO(GregBowyer): Reimplement what cargo does to detect bad renames
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better to reference a github issue here, but I don't feel strongly about this.

///
/// Currently cargo-metadata provides rename detection in a few places, we take the names
/// from the resolution of a package
fn is_renamed(&self, dep_package: &Package) -> bool {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to precompute some of these new properties? I'm lost in the details here, but part of the idea with the crate catalog was to make this type of lookup more efficient.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Humm yeah might work, and might be a good way to make the find the root sillyness go away.

@GregBowyer
Copy link
Contributor Author

Not ideal that the formatter changes are co-mingled with the behavior changes. A bit harder to review.

I am so unqualified to review these PRs at this point. I don't remember much about the implementation and anyway it has changed a lot since I wrote significant code here.

Oh well

I would only bother with the first commit in the PR, I can just pull the second commit (the reformatting) and we can do a round of that after all crazy PRs get merged

@UebelAndre
Copy link
Collaborator

Crazy PR has been merged. No more crazy PRs from me for the foreseeable future 😅

@UebelAndre
Copy link
Collaborator

UebelAndre commented Jan 3, 2021

In hindsight, this should have been merged before any crazy PRs 😞 Sorry about pushing for mine over this one.

@GregBowyer do you have bandwidth in the foreseeable future to revive this effort 😅?

edit: Attempted to do the rebase myself and it's become quite hairy. I think the best course of action would be to more or less start from a clean slate. I think if you drop all changes made to tests then the merge isn't so bad. I've since updated how metadata is mocked for tests to make them deterministic and not require internet. That seems to be the biggest source of conflicts.

@GregBowyer
Copy link
Contributor Author

I have time to pick this up again

Copy link
Collaborator

@UebelAndre UebelAndre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of comments and suggestions but otherwise sounds pretty good. I'm going to need to take another day to page some of the nuances here back into my brain and do another pass but still pretty excited for the direction!

impl/src/context.rs Outdated Show resolved Hide resolved
impl/src/util.rs Outdated Show resolved Hide resolved
impl/src/util.rs Show resolved Hide resolved
impl/src/planning.rs Outdated Show resolved Hide resolved
.iter()
.filter(|to_alias| to_alias.lib_target_name.is_some())
.flat_map(|to_alias| {
let pkg_name = util::sanitize_ident(&to_alias.pkg_name);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the sanitize is worth keeping but wanted to share 9d74090 here in case that's relevant. I think sanatize_indent might need to be updated

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops I thought I factored that in when I rebased so good catch.

impl/src/planning/subplanners.rs Outdated Show resolved Hide resolved
impl/src/planning/subplanners.rs Outdated Show resolved Hide resolved
impl/src/planning/subplanners.rs Outdated Show resolved Hide resolved
impl/src/planning/subplanners.rs Outdated Show resolved Hide resolved
impl/src/util.rs Outdated Show resolved Hide resolved
@UebelAndre
Copy link
Collaborator

Since migrating most things to Bazel, ensuring the examples are up to date has not been added to CI. Would you be able to run bazel run //tools:examples_raze and commit the results? I suspect a few things might change based these changes.

@GregBowyer
Copy link
Contributor Author

I will pull in the comments (all sane I think) make sure that test failure is not something sinister.

I think I should also take another scan through the bug list to see if there are any other bugs that this solves.

@GregBowyer GregBowyer force-pushed the alias-bugs branch 2 times, most recently from 8edc09d to 2063fd5 Compare April 23, 2021 20:58
@GregBowyer
Copy link
Contributor Author

GregBowyer commented Apr 23, 2021 via email

There are a range of subtle and irritating bugs that stem from
attempting to work with renames from the package set provided by cargo
metadata.

Fortunately, recent versions of cargo metadata provide a limited version
of the resolve nodes, both as the original node ids and as a newer set
providing both rename and targeting information.

As such we can (hopefully) simplify crate -> dep node resolution, remove
a bunch of subtle aliasing bugs.

Fixes google#241, fixes google#269, fixes google#270, resolves google#144, resolves google#187
Copy link
Collaborator

@UebelAndre UebelAndre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just another small pass. I'll take some time over the weekend to do a thorough review. The thing I want to look into is the test coverage for this. What's your confidence level that there's enough here to prevent a regression? (again, need to look, there might be enough already but I don't have that context paged in my brain right now 😅)

if let Some(platform_details) = &self.platform_details {
if let Some(settings_target) = &self.settings.target {
let platform = Platform::from_str(&target_str)?;
fn is_dep_targetted(&self, target: Option<&String>) -> bool {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can you add a small doc for this?


Ok((self._produce_deps(&default_deps)?, targeted_set))
}
fn process_dep(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add a doc here as well

Copy link
Collaborator

@UebelAndre UebelAndre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, a few more nits and one or two questions. But otherwise I think this is looking really good! 😄

@@ -438,7 +438,7 @@ impl RazeMetadataFetcher {
let version = info.req();
let src_dir = self.fetch_crate_src(cargo_dir.as_ref(), &name, version)?;
checksums.insert(
package_ident(name, version),
package_ident(name, &version),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was there an issue with this before?

@@ -155,7 +128,7 @@ impl<'planner> WorkspaceSubplanner<'planner> {
&self,
node: &Node,
catalog: &CrateCatalog,
) -> Option<Result<CrateContext>> {
) -> Option<Result<(CrateContext, bool)>> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this not information that already lives (or could already live) in CrateContext? Admittedly that struct could use some cleaning but whether or not the crate is a workspace member seems like it makes sense to store in the context.

#[derive(Debug, Clone, Serialize)]
pub struct CrateContext {
pub pkg_name: String,
pub pkg_version: Version,
pub edition: String,
pub raze_settings: CrateSettings,
pub canonical_additional_build_file: Option<PathBuf>,
pub default_deps: CrateDependencyContext,
pub targeted_deps: Vec<CrateTargetedDepContext>,
pub license: LicenseData,
pub features: Vec<String>,
pub workspace_path_to_crate: String,
pub workspace_member_dependents: Vec<PathBuf>,
pub workspace_member_dev_dependents: Vec<PathBuf>,
pub workspace_member_build_dependents: Vec<PathBuf>,
pub is_workspace_member_dependency: bool,
pub is_binary_dependency: bool,
pub targets: Vec<BuildableTarget>,
pub build_script_target: Option<BuildableTarget>,
pub links: Option<String>,
pub source_details: SourceDetails,
pub sha256: Option<String>,
pub registry_url: String,
// TODO(acmcarther): This is used internally by renderer to know where to put the build file. It
// probably should live somewhere else. Renderer params (separate from context) should live
// somewhere more explicit.
//
// I'm punting on this now because this requires a more serious look at the renderer code.
pub expected_build_path: String,
// The name of the main lib target for this crate (if present).
// Currently only one such lib can exist per crate.
pub lib_target_name: Option<String>,
// This field tracks whether or not the lib target of `lib_target_name`
// is a proc_macro library or not.
pub is_proc_macro: bool,
}

{%- for condition in targeted_dep.conditions %}
"{{ condition }}",
{%- for platform_target in targeted_dep.platform_targets %}
"@rules_rust//rust/platform:{{platform_target}}",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"@rules_rust//rust/platform:{{platform_target}}",
"@{{ rust_rules_workspace_name }}//rust/platform:{{platform_target}}",

{%- for condition in targeted_dep.conditions %}
"{{ condition }}",
{%- for platform_target in targeted_dep.platform_targets %}
"@rules_rust//rust/platform:{{platform_target}}",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"@rules_rust//rust/platform:{{platform_target}}",
"@{{ rust_rules_workspace_name }}//rust/platform:{{platform_target}}",

@@ -225,7 +225,7 @@ pub fn mock_remote_crate<'server>(
let enc = flate2::write::GzEncoder::new(tar_gz, Compression::default());
let mut tar = tar::Builder::new(enc);
tar
.append_dir_all(package_ident(name, version), dir.as_ref().join("archive"))
.append_dir_all(package_ident(name, &version), dir.as_ref().join("archive"))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another question about using a reference. Just curious.

@UebelAndre
Copy link
Collaborator

@acmcarther this is looking near completion. Don't know if you're around to review it but it'd be great to get in. If not, maybe @dfreese can take a look? It'd probably be good to get your eyes on it too just so you're aware of the changes (sorry for the ping otherwise 😅)

@dfreese dfreese self-assigned this Apr 26, 2021
Copy link
Collaborator

@dfreese dfreese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't had a chance to make a full pass at this, but had one major question, based on past experience.

pub workspace_context: WorkspaceContext,
/// The creates to build for
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// The creates to build for
/// The crates to build for

Comment on lines +42 to +53
// We only want equality for the first member as it can have multiple names pointing at it.
impl Hash for DependencyAlias {
fn hash<H: Hasher>(&self, state: &mut H) {
self.target.hash(state);
}
}

impl PartialEq for DependencyAlias {
fn eq(&self, other: &Self) -> bool {
self.target == other.target
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been bitten by some nasty bugs where hash or comparison functions don't cover the all of the data in the struct.

Is there another way to structure this, perhaps by using map from target string to alias (or &DependencyAlias)?

.push(buildable_dependency.clone());
} else {
dep_set.build_deps.push(buildable_dependency.clone());
// TODO(GregBowyer): Reimplement what cargo does to detect bad renames
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to TODO(#cargo-raze-issue): referencing this error.

@UebelAndre
Copy link
Collaborator

@GregBowyer friendly ping. I feel this is so close to going in 😄

@UebelAndre
Copy link
Collaborator

@GregBowyer @dfreese another friendly ping 😅 Would love to get this merged and am wondering what outstanding action items are. Happy to help move this forward however I can.

@UebelAndre
Copy link
Collaborator

Friendly ping again. Happy to help however I can to get this in but would love to see this merged 😄

.find(|target| target.kind == "lib" || target.kind == "proc-macro")
.map(|target| target.name.clone());

let is_proc_macro = targets.iter().any(|target| target.kind == "proc_macro");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let is_proc_macro = targets.iter().any(|target| target.kind == "proc_macro");
let is_proc_macro = targets.iter().any(|target| target.kind == "proc-macro");

Aren't string literals fun? 😄

@UebelAndre
Copy link
Collaborator

@dfreese is this not something you could apply the suggestions for and accept as is? I think it's in a good spot.

@dfreese
Copy link
Collaborator

dfreese commented May 21, 2021

@dfreese is this not something you could apply the suggestions for and accept as is? I think it's in a good spot.

#282 (comment)

Not having all fields included as a part of a hash/comparison function causes some incredibly hard bugs to figure out, so I'd want to see if there's an alternative solution to that first.

@UebelAndre
Copy link
Collaborator

@dfreese is this not something you could apply the suggestions for and accept as is? I think it's in a good spot.

#282 (comment)

Not having all fields included as a part of a hash/comparison function causes some incredibly hard bugs to figure out, so I'd want to see if there's an alternative solution to that first.

Is that not something you or I could patch in this PR or in a follow up? I think @GregBowyer has put fourth a huge amount of effort implementing this PR at least 3 times. I'd hate to have this reimplemented a 4th time and there are other changes I think should go in.

@GregBowyer GregBowyer closed this Aug 11, 2021
illicitonion pushed a commit that referenced this pull request Aug 25, 2021
This fixes `crates.bzl` never having any crates defined in the `_PROC_MACRO_DEPENDENCIES` map.

This fixes a regression from #425 (or #282) and regenerates cargo-raze outputs (which should have been done on the merged commit). 

This also fixes a regression in how BUILD files were rendered, resulting in buildifier defects. I cherry picked a change I had an another branch which is ahead of `main` and contains the fix. This change includes #393

Finally, this fixes another regression where `extra_aliased_targets` was being ignored.
@GregBowyer GregBowyer deleted the alias-bugs branch September 28, 2021 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants