-
Notifications
You must be signed in to change notification settings - Fork 688
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate MSBuild property that has the path to a package #2271
Conversation
FYI @AndyGerlicher |
How about writing out the path to every package under As long as this is conditioned on TFM there can only be a single version of each id, so there wouldn't be conflicts. I'm glad to see work being done in this area, I have long wanted this feature, and had planned at one time to write an external package that would write this out after restore. A few issues I see with the PropertyName attribute:
|
I was initially writing out a property for every package but observed two issues:
Forcing the user to opt into this behavior feels like the right solution to convey that its not supposed to be necessary. We want all packages to conform so you don't need to know where they are. Also, by requiring the user to specify the property name, NuGet doesn't have to worry about the complicated algorithm to generating the property name.
I really don't forsee the need to generate properties for them. If needed, a user could add an explicit package reference to get a property.
I chose
At the moment, its last-one-wins just like everything else in MSBuild. We could throw an exception instead. Lastly, to get a property for all package references, a user can simply do this: <ItemDefinitionGroup>
<PackageReference>
<!-- Generate a property like PkgFooBar for all PackageReferences -->
<PropertyName>Pkg%(ItemSpec.Replace(".", ""))</PropertyName>
</PackageReference>
</ItemDefinitionGroup> But we wouldn't really want to encourage people to use this feature, we want to make it available for migration purposes to further the adoption of |
How about writing all package paths, but only if the user sets a flag to opt in then? You could also set a default prefix for the property names, or opt to not have a prefix at all. I would say that not having paths for transitive packages will be much more painful than collisions on We have seen a lot of complaints in the past about the need for adding top level references for things like this. When adding the feature and testing it on small solutions it seems fine, but with real world solutions it falls apart and becomes painful fast. We need to avoid these problems. Does I don't see this as a migration feature. There are many scenarios where users need to override how NuGet references dlls from a package. For example with externing a namespace, if for example you needed to do that on a transitive reference you would want to find the current references and modify them through MSBuild. If you had all package paths automatically I think it would be much easier. |
Can you describe more what you mean? Do you mean a property for every single package using an algorithm that comes up with the property name? Or just writing to a file every path? I want to understand your idea more.
Do you not think its enough to just have an explicit reference? Or to say that packages that are malformed can't be used transitively in non-traditional ways?
No, but my position is still that we don't need to provide properties for indirect dependencies since I am of the opinion that people can just have an explicit dependency on anything that they need that stock NuGet doesn't provide. The purpose of this feature in my view is a workaround to consume non-conformant packages. I believe if we wanted more functionality with PackageReference we should just add them as first-class features. Externing namespaces for example should be something that's just built in at some point. And using property path hints is the temporary workaround.
In my experience of onboarding small, medium, and large repos, only a very small percentage of the packages can't be used out-of-the-box with PackageReference. When a team needs to know the path, there's usually 1 to 5 packages that a repository is using that no one owns and can't be updated. 90% of the time its a command-line tool that they are executing during their build and the package doesn't have a |
Before even going into specifics of design, I'd recommend to first discuss the requirement and understand what do we have today? @rrelyea already mentioned on the original issue about already having something on the similar lines. Even I'm not really able to understand how it's more painful to consume some tool from a package with PackageReference compare to packages.config? With packages.config, you'd guess the path something like And lastly, even I agree with Justin's point to covering the transitive dependencies scenario as well because IMO that's the real difference between packages.config vs PackageReference. |
Here is how I see the problem:
For an actual tool package I would expect the download API to be used since these package are not used by a project. The best experience for tools so far is
I don't think anything like this exists already. Restore will write out the package folders, from there you could find the package. The problem is similar to packages.config, but it is a bit more tricky since the packages folder can change with the environment, and it is hard to take advantage of new features with PR such as floating versions. In the NuGet.Client build we workaround the problem by downloading many packages to a known location and then referencing them directly. If this problem were fixed you could get rid of this file completely: https://github.com/NuGet/NuGet.Client/blob/dev/.nuget/packages.config and the build wouldn't have to download nuget.exe at all to bootstrap 😄 @jeffkl Changing LibraryDependency here is equivalent to changing ItemSpec in MSBuild. NuGet can't 1st class everything for the same reasons that MSBuild can't. NuGet is used for C++ and 3rd party project systems which aren't part of VS. Users need a generic way to add custom functionality around packages.
|
@rrelyea there is no current feature to do what we are requesting. There seems to be two options:
Which of these should we go after? |
Samples of generating a property for every package: https://gist.github.com/jeffkl/6acdda0b4116b98244700685927a7cac |
Thanks @emgarten for the explanation. I agree that as a NuGet user, I should always be able to access my packages and consume them in build or publish workflow. This issue existed with |
@jainaashish and @nkolev92 please review, I've updated it according to our discussion. |
Ping please @jainaashish and @nkolev92 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like the right approach, but I'd rather see it follow the same pattern as the rest of the props generation code.
Please add tests in https://github.com/NuGet/NuGet.Client/blob/dev/test/NuGet.Core.Tests/NuGet.Commands.Test/BuildAssetsUtilsTests.cs
.Select(e => pkg.Value.GetAbsolutePath(e))) | ||
.Select(path => GetPathWithMacros(path, repositoryRoot)) | ||
.Select(GenerateImport)); | ||
pkg.Key.Build.WithExtension(TargetsExtension) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the indentation of this still looks off. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did not mean to touch the whitespace so I'll revert all that
.Select(e => pkg.Value.GetAbsolutePath(e))) | ||
.Select(path => GetPathWithMacros(path, repositoryRoot)) | ||
.Select(GenerateImport)); | ||
pkg.Key.Build.WithExtension(PropsExtension) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: same as above.
// Create an empty PropertyGroup for package properties | ||
var packagePathsPropertyGroup = MSBuildRestoreItemGroup.Create("PropertyGroup", Enumerable.Empty<XElement>(), 1000, isMultiTargeting ? frameworkConditions : Enumerable.Empty<string>()); | ||
|
||
var projectGraph = targetGraph.Graphs.FirstOrDefault(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not use the flattened graph?
targetGraph.Flattened?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Under the debugger, I found that the first graph was the project itself which has its direct dependencies. Would targetGraph.Flattened
give all transitive dependencies as well? Technically the GeneratePathProperty
wouldn't be set on any transitive dependencies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Under the debugger, I found that the first graph was the project itself which has its direct dependencies.
I'm not sure if that's the right statement, I'd double check it. Or filter it based on GraphItem
|
||
props.AddRange(GenerateGroupsWithConditions(buildPropsGroup, isMultiTargeting, frameworkConditions)); | ||
|
||
// Create an empty PropertyGroup for package properties | ||
var packagePathsPropertyGroup = MSBuildRestoreItemGroup.Create("PropertyGroup", Enumerable.Empty<XElement>(), 1000, isMultiTargeting ? frameworkConditions : Enumerable.Empty<string>()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Anything preventing you from using the same pattern as the build props code?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one is a little different in that its a property group with conditions vs an import group with no conditions containing imports with conditions. But if you feel strongly I can try to follow a pattern
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if I understand correctly but we already add PropertyGroup with condition in nuget.g.props file. wouldn't this be similar to that? We can have a dict of packages path and add them in one shot.
@nkolev92 let me know what you think of the unit test I wrote and I'll add a few more. My unit test tests the whole thing from end to end. |
@nkolev92 does this look ready? Can you initiate the CI tests please? |
@jeffkl |
var packagePathProperties = packagesWithTools.Union(projectGraph.Item.Data.Dependencies.Where(i => i.GeneratePathProperty).Select(i => i.Name)).Distinct(StringComparer.OrdinalIgnoreCase) | ||
.Select(i => sortedPackages.Cast<KeyValuePair<LockFileTargetLibrary, Lazy<LocalPackageSourceInfo>>?>().FirstOrDefault(x => x.Value.Key.Name.Equals(i, StringComparison.OrdinalIgnoreCase))) | ||
.Where(i => i != null && i.Value.Value.Exists()) | ||
.Select(i => GeneratePackagePathProperty(i.Value.Key.Name, i.Value.Value.GetAbsolutePath(i.Value.Value.Value.Package.ExpandedPath))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please simply this logic? these complex linq queries are never easy to read and understand. besides, i also dont understand this i.Value.Value.Value.Package.ExpandedPath
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, the Linq statements in this file really tested my understanding. I'll break them up a little.
@@ -493,6 +495,24 @@ public static string GetMSBuildFilePath(PackageSpec project, RestoreRequest requ | |||
|
|||
props.AddRange(GenerateGroupsWithConditions(buildPropsGroup, isMultiTargeting, frameworkConditions)); | |||
|
|||
// Create an empty PropertyGroup for package properties | |||
var packagePathsPropertyGroup = MSBuildRestoreItemGroup.Create("PropertyGroup", Enumerable.Empty<XElement>(), 1000, isMultiTargeting ? frameworkConditions : Enumerable.Empty<string>()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where are you adding PropertyGroup condition for ExcludeRestorePackageImports
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Those are added on line 573
.
|
||
private static XElement GeneratePackagePathProperty(string packageName, string packagePath) | ||
{ | ||
return GenerateProperty($"Pkg{packageName.Replace(".", "_")}", packagePath); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where are we adding condition for Pkg<package_path>
already exists or not on this property?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are added by GenerateProperty
on line 191
@@ -611,5 +619,108 @@ public void BuildAssetsUtils_VerifyPositionAndSortOrder() | |||
Assert.Equal("x", targetItemGroups[3].Attribute(XName.Get("Condition")).Value.Trim()); | |||
} | |||
} | |||
|
|||
[Fact] | |||
public async Task BuildAssetsUtils_GeneratePathProperty() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we also add test for a package with tool folder and verify that it does generate path property implicitly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall looks good, i just have few concerns regarding readability and missing conditions on PropertyGroup.
@nkolev92 I've rebased, please relaunch the CI build |
ahh it wasn't implicit, we have to update https://github.com/jainaashish/project-system/blob/a8cf44c3948757b7cb261532834ce546bb5fb47a/src/Microsoft.VisualStudio.ProjectSystem.Managed/ProjectSystem/Rules/PackageReference.xaml in order to get this metadata as part of nomination. I can raise the PR to fix it, but it's upto SDK Project System team about the timeline. @Pilchie |
@radical do you know if or when this would be available in Mono? I just tried |
Depends on the nuget.exe version the mono isntallation is carrying. This change is in 4.9, which I'm pretty sure mono 5.8.0 does not have. |
I'm guessing that @KirillOsenkov meant 5.18.x . We recently bumped to nuget 4.8.1 (mono/mono#12109). Could you open an issue under mono for this? |
@radical yes, 5.18, thanks. Filed mono/mono#12393. |
|
||
// Find the packages with matching IDs in the list of sorted packages, filtering out ones that there was no match for or that don't exist | ||
var packagePathProperties = localPackages | ||
.Where(pkg => pkg?.Value?.Package != null && packageIdsToCreatePropertiesFor.Contains(pkg.Value.Package.Id) && pkg.Exists()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hit a case-sensitivity issue here. I believe this contains call is the cause since I don't see how it handles if the winner of the distinct call above had different case than the value in Package.Id.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I've hit this too but wasn't able to figure out the problem. I think you're right that its casing. I'll submit a PR
I've been using VS 2019 RC with SDK-style projects, and I've noticed that some packages seem to be getting path entries in If it is supposed to be happening by default, then there's still a problem because I'm not seeing all of the packages in a given project have the entries, just some of them. |
I have observed the same behaviour, did the default change? |
The default was from the beginning to generate a property for any package that contains a |
Well that does explain the behavior I'm seeing, but I have to say that it being conditional like that is super confusing. I'd much rather just have it be there always or never. |
I think the argument could be made for either automatic or never/always. Maybe the best course of action would be to add a property you can set to disable the automatic creation of property for packages with tools. The default would still have to be automatic though... |
That's fine, make it automatic for all package references then, not just ones with a certain internal structure. I have scenarios where having the package path automatically created is going to be useful, and none of them involve the |
@bording I think you should open an issue and start a discussion. I probably don't fully understand your scenarios so having more info would help a lot. And that would also allow others to comment and give us an idea of whether or not to change the feature. |
@bording I realize this isn't a complete solution but it might help for your situation for now. You can add a <Project>
<ItemDefinitionGroup>
<PackageReference GeneratePathProperty="true" />
</ItemDefinitionGroup>
</Project> And all your package references will get that automatically without having to specify it each time. |
@AndyGerlicher Thanks for the suggestion. I would probably end up doing something like that for my repos. However, as you say, that doesn't address the general concerns about the usability/discoverability of the feature, so I will raise a new issue as @jeffkl suggested. |
I remember we did discuss the all or nothing approach (I actually preferred it over the extra property), but there were some concerns about adding lots of potentially unnecessary evaluations from msbuild side. Let's continue the discussion on the new issue but I just want to bring up that question again. |
Here's the new issue: NuGet/Home#7851 |
@jeffkl was this sample moved somewhere else. thanks |
Fixes: NuGet/Home#6949
Some packages have a weird layout. They are in use via
packages.config
because you can guess the path with something like..\packages\Some.Bad.Package.1.0.0\something.exe
. When migrating toPackageReference
, these packages are very hard to consume. Not all of these packages can be updated so there's a need to add a feature to NuGet so you can use them.This change adds a
GeneratePathProperty
metadata item to<PackageReference />
and then generates a property in the.g.props
.Example:
The resulting
nuget.g.props
contains:This change also generates a path property for any package that has a
tools
folder.Example: