Repeatable build using Project lock file implementation #2342

jainaashish · 2018-07-12T22:03:53Z

This is Repeatable build using Project lock file implementation and contain following parts:

generating/ updating project lock file
consuming project lock file during restore
Validation of package's sha512 and added error messages as appropriate
Test infra for legacy PackageReference func tests
Unit tests for lock file
Basic workflow is completed

Spec - https://github.com/NuGet/Home/wiki/Enable-repeatable-package-restore-using-lock-file
Technical Spec - https://github.com/NuGet/Home/wiki/Repeatable-build-using-lock-file-implementation

jainaashish · 2018-07-31T00:17:32Z

@NuGet/nuget-client 🔔 This is ready to be reviewed... Just skip the test part which i'm still working on.

jainaashish · 2018-08-08T23:21:00Z

@DoRonMotter suggested to add the team alias but explicitly add only those individuals who should really take a look. So I've updated the reviewers list accordingly.

@NuGet/nuget-client @rrelyea 🔔

PatoBeltran

Overall it looks good. Bunch of comments, most of them to improve readability and have a consistent code style. I'm concern that we have a LockFile class that refers to the assets file and a NuGetLockFile that refers to the actual lock file, this will make the code REALLY confusing. I think we should rename the current LockFile to have a better naming (i.e. AssetsFile) and then call this new class LockFile.

PatoBeltran · 2018-08-09T18:21:02Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

+            _lockFileVersion = lockFileVersion;
+        }
+
+        public NuGetLockFile CreateNuGetLockFile(LockFile assetsFile)


I feel like keeping the current assestsFile class as LockFile and creating a new NuGetLockFile class for this is REALLY confusing. I would argue in favor of renaming LockFile to AssetsFile in another PR first and then naming this LockFile instead of NuGetLockFile. Therefore also this class can be renamed to LockFileBuilder because we don't need to prepend NuGet to our class names.

can we call NuGetLockFile RepeatableBuildLockFile or something more descriptive? I do agree that it's incredibly confusing to see LockFile at someplace and NuGetLockFile in others.

It'll be difficult to rename the current lock file classes, as they are used heavily in the build targets.

Maybe in 5.0, but even then we'd likely have to take the initiative to fix the build targets ourselves.

I think we should discuss the naming concerns offline.
We have the same issue with the config changes.
We need to address both at an uber level.

@nkolev92 so you are suggesting that we should just keep piling on the tech debt? Honestly, I can see how someone looking at this for the first time get really confused about the fact that there is a LockFile class that has nothing to do with the repeatable build lock file. I would prefer to fix that first before adding more debt that will later be more difficult to fix.

I'm suggesting that we should have a naming concerns discussion and that this specific class LockFile is unlikely to change in 15.9 because the build targets are using it.

I'd be up for cleaning up the APIs in 5.0/16.0, but we have to commit to doing it rather than postpone it into perpetuity.

I agree with @nkolev92 that we can discuss the naming concerns offline and come to a conclusion.

But just to add my point of view here, we have some shitty naming conventions across our code base and I certainly wouldn't want to pile on top of that but we also have to realize that we're not in a re-factoring phase here and have the liberty to spend weeks and months on that. What we can do here is have less confusing names and better class summaries which clearly differentiates.

Now about this new lock file, then may I propose PackagesLockFile class name since the file name itself is packages.lock.json so it should be convenient to understand and have the clear summary to indicate that this class represents repeatable build packages.lock.json file.

As discussed offline individually with most of you, I'll go ahead with PackagesLockFile with appropriate summary description and for all other subsequent classes. And also logged a tracking issue# NuGet/Home#7207 to do some refactoring which we can discuss when we plan for dev16.

PatoBeltran · 2018-08-09T18:24:07Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

+
+            foreach (var target in assetsFile.Targets)
+            {
+                var nuGettarget = new NuGetLockFileTarget()


Same as comment above, there shouldn't be a need to prepend NuGet to the class name.

Also nit: rename nuGettarget to nuGetTarget for proper casing.

I'm not een sure what the proper casing is there lol.
NuGet*bla is confusing for sure.

As mentioned in the above comment, I'd prefer an offline pros/cons discussion for our naming strategy.

please refer the above comment

PatoBeltran · 2018-08-09T18:24:43Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

+                var framework = assetsFile.PackageSpec.TargetFrameworks.FirstOrDefault(
+                    f => EqualityUtility.EqualsWithNullCheck(f.FrameworkName, target.TargetFramework));
+
+                var lilbraries = target.Libraries;


nit: libraries

PatoBeltran · 2018-08-09T18:26:53Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

+                {
+                    var onlyTFM = assetsFile.Targets.First(t => EqualityUtility.EqualsWithNullCheck(t.TargetFramework, target.TargetFramework));
+
+                    lilbraries = target.Libraries.Where(lib => !onlyTFM.Libraries.Any(tfmLib => tfmLib.Equals(lib))).ToList();


nit: consider setting var libraries to null and then doing an if\else statement to avoid possibly loading the libraries variable twice.

we're not actually loading it twice, first is the assignment statement similar to assigning to null and then update it if required. Do you see anything wrong with that?

PatoBeltran · 2018-08-09T18:27:46Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

+                        Dependencies = library.Dependencies
+                    };
+
+                    var framework_dep = framework.Dependencies.FirstOrDefault(


Given that you used FirstOrDefault, framework could be null and this would give an exception.

Also nit: rename framework_dep to frameworkDependency, we should not be mixing CamelCase and snake_case in our code, and c# uses CamelCase.

framework cannot be null since we can't have a framework inside lock file targets which doesn't exists in packageSpec. But if we still want to be safeguard then i can handle that as well.

And agree with naming convention thing, but personally i dont care too much about block level local variables, rather be more concerned about class or method level variables. But I'll change this.

If it cannot be null then use First which will throw an exception if the value doesn't exist.

PatoBeltran · 2018-08-09T20:33:00Z

...kageManagement.VisualStudio.Test/ProjectSystems/LegacyPackageReferenceRestoreUtilityTests.cs

+            {
+                // Arrange
+                var sourceRepositoryProvider = TestSourceRepositoryUtility.CreateSourceRepositoryProvider(
+                new List<Configuration.PackageSource>()


nit: Please add a tab for correct indentation.

PatoBeltran · 2018-08-09T20:33:32Z

...kageManagement.VisualStudio.Test/ProjectSystems/LegacyPackageReferenceRestoreUtilityTests.cs

+            {
+                // Arrange
+                var sourceRepositoryProvider = TestSourceRepositoryUtility.CreateSourceRepositoryProvider(
+                new List<Configuration.PackageSource>()


nit: Please add a tab for correct indentation.

PatoBeltran · 2018-08-09T20:33:42Z

...kageManagement.VisualStudio.Test/ProjectSystems/LegacyPackageReferenceRestoreUtilityTests.cs

+            {
+                // Arrange
+                var sourceRepositoryProvider = TestSourceRepositoryUtility.CreateSourceRepositoryProvider(
+                new List<Configuration.PackageSource>()


nit: Please add a tab for correct indentation.

PatoBeltran · 2018-08-09T20:34:12Z

...kageManagement.VisualStudio.Test/ProjectSystems/LegacyPackageReferenceRestoreUtilityTests.cs

+            {
+                // Arrange
+                var sourceRepositoryProvider = TestSourceRepositoryUtility.CreateSourceRepositoryProvider(
+                new List<Configuration.PackageSource>()


nit: Please add a tab for correct indentation.

PatoBeltran · 2018-08-09T21:00:53Z

...ients.Tests/NuGet.PackageManagement.VisualStudio.Test/ProjectSystems/TestVSProjectAdapter.cs

+
+namespace NuGet.PackageManagement.VisualStudio.Test
+{
+    public class TestVSProjectAdapter : IVsProjectAdapter


nit: consider using => in every one-line method in this class, to create inline function declarations.

Also, should this class be internal?

yes to internal class and no to inline function

rohit21agrawal · 2018-08-13T18:22:45Z

src/NuGet.Clients/NuGet.SolutionRestoreManager/VsSolutionRestoreService.cs

@@ -57,6 +57,9 @@ public sealed class VsSolutionRestoreService : IVsSolutionRestoreService, IVsSol
        private const string TreatWarningsAsErrors = nameof(TreatWarningsAsErrors);
        private const string WarningsAsErrors = nameof(WarningsAsErrors);
        private const string NoWarn = nameof(NoWarn);
+        private const string RestorePackagesWithLockFile = nameof(RestorePackagesWithLockFile);


should these 3 properties be encapsulated by a single class like RestoreLockFileProperties?

hmm good suggestion.

rohit21agrawal · 2018-08-13T18:35:14Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

+                    };
+
+                    var framework_dep = framework.Dependencies.FirstOrDefault(
+                        dep => PathUtility.GetStringComparerBasedOnOS().Equals(dep.Name, library.Name));


is framework dependency an actual path? shouldn't it always be a case insensitive comparison?

No, this is dependency name and would be case sensitive for linux machines.

nkolev92

I'm halfway through the PR.

I will continue tomorrow, here's the initial feedback.

nkolev92 · 2018-08-13T23:31:00Z

src/NuGet.Clients/NuGet.SolutionRestoreManager/VsSolutionRestoreService.cs

@@ -57,6 +57,9 @@ public sealed class VsSolutionRestoreService : IVsSolutionRestoreService, IVsSol
        private const string TreatWarningsAsErrors = nameof(TreatWarningsAsErrors);
        private const string WarningsAsErrors = nameof(WarningsAsErrors);
        private const string NoWarn = nameof(NoWarn);
+        private const string RestorePackagesWithLockFile = nameof(RestorePackagesWithLockFile);
+        private const string NuGetLockFilePath = nameof(NuGetLockFilePath);
+        private const string FreezeLockFileOnRestore = nameof(FreezeLockFileOnRestore);


Can you please unify the properties/switches in msbuild in both specs?
The code matches https://github.com/NuGet/Engineering/blob/master/Client.Specs/RepeatableBuildProjectLockFile.md, but is different from https://github.com/NuGet/Home/wiki/Enable-repeatable-package-restore-using-lock-file#extensibility

@anangaur is working on unifying it in his spec. And if he change anything then i'll reflect that.

nkolev92 · 2018-08-13T23:46:52Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

+    public class NuGetLockFileBuilder
+    {
+        private readonly int _lockFileVersion;
+        public NuGetLockFileBuilder(int lockFileVersion)


format: add a new between the field and the constructor.

nkolev92 · 2018-08-13T23:48:03Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

+            _lockFileVersion = lockFileVersion;
+        }
+
+        public NuGetLockFile CreateNuGetLockFile(LockFile assetsFile)


I think we should discuss the naming concerns offline.
We have the same issue with the config changes.
We need to address both at an uber level.

nkolev92 · 2018-08-13T23:49:59Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

+
+            foreach (var target in assetsFile.Targets)
+            {
+                var nuGettarget = new NuGetLockFileTarget()


I'm not een sure what the proper casing is there lol.
NuGet*bla is confusing for sure.

As mentioned in the above comment, I'd prefer an offline pros/cons discussion for our naming strategy.

nkolev92 · 2018-08-13T23:53:06Z

src/NuGet.Core/NuGet.Commands/NuGetLockFileBuilder.cs

@@ -0,0 +1,102 @@
+using System;


Fix the usings (pretty sure some are unused, like the nuget.frameworks one) and add a header.

nkolev92 · 2018-08-14T00:58:06Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

+                            if (_request.ExistingLockFile != null)
+                            {
+                                newAssetsFile = _request.ExistingLockFile.Clone();
+                                newAssetsFile.LogMessages = logs;


Same here, I think you need to replay the old warnings.

let me check with Ankit on this.

You're replaying the warning only to replace them again, why?
Maybe I'm missing something.

nkolev92 · 2018-08-14T00:58:24Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

+                                lockFile: newAssetsFile,
+                                previousLockFile: _request.ExistingLockFile,
+                                lockFilePath: _request.ExistingLockFile?.Path,
+                                cacheFile: null,


same as above, you have the cache file already.

responded above, would like to keep it this way until there is a valid scenario instead to pass cache file details

Responded above for the reasoning.

nkolev92 · 2018-08-14T01:01:25Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

@@ -313,12 +438,36 @@ private bool VerifyAssetsFileMatchesProject()
            return (_request.ExistingLockFile != null && pathComparer.Equals( _request.ExistingLockFile.PackageSpec.FilePath , _request.Project.FilePath));
        }

-        private KeyValuePair<CacheFile, bool> EvaluateCacheFile()
+        private bool ValidatePackages(NuGetLockFile lockFile, LockFile assetsFile)


This should have a better name.

I thought it was self-explanatory but open for suggestions. How about ValidatePackagesSha or ValidatePackagesSha512 etc...

Yeah that might work. imho the current name is too simple.

I'm not against making it even more verbose such as ValidateLockFilePackagesShaMatchesAssetsFile

nah that's toooooooo verbose. I'll go with ValidatePackagesSha

nkolev92 · 2018-08-14T01:09:37Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreResult.cs

@@ -157,23 +172,26 @@ public virtual async Task CommitAsync(ILogger log, CancellationToken token)

            BuildAssetsUtils.WriteFiles(buildFilesToWrite, log);

+            if (result.LockFile == null || result.LockFilePath ==  null)


This is adding a check that wasn't there before.
Why?

err because that was missing and there was no test covering that... we had that for tool restore but not for actual restore.

What's the benefit of this?

This never happens because otherwise we would have had a runtime error.

Maybe the correct approach is to null check in the constructor of the result.
Here it just hides potential problems.

Ahh never mind.
I finally got this because of the comment.

Basically you're covering this case.

https://github.com/NuGet/NuGet.Client/pull/2342/files#diff-ae3ea71e74915c8f0634a376599d6b87R249

Similar to what I recommended in other places.
I'd rather have an additional flag that indicates the scenario rather than rely on nulls as valid scenarios.

Having LockFile as null when we've bailed out the restore completely make sense to me and the right approach. Adding an additional flag for it, is what doesn't make sense to me.

My problem with this is that it promotes bad practice, where you don't know if things are null and when they are null and what does it mean for them to be null.

I'm out of arguments for this... I sincerely dont follow your concerns over bad practice for this. It's as simple as any other null check in an api to before consuming anything.

Lets talk about this offline and finalize it

discussed offline, we're settled on this.

nkolev92 · 2018-08-14T01:10:39Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreRunner.cs

@@ -293,7 +293,8 @@ public static async Task<RestoreSummary> CommitAsync(RestoreResultPair restoreRe
            }

            // Remote the summary messages from the assets file. This will be removed later.
-            var messages = restoreResult.Result.LockFile.LogMessages.Select(e => new RestoreLogMessage(e.Level, e.Code, e.Message));
+            var messages = restoreResult.Result.LockFile?.LogMessages


Can you elaborate on why this is necessary?

because with packages lock file, there could be a case where there was no existing assets file and we fail restore because packages lock file is different than project file and user can denied updating packages lock file with restore.

Refer to the above comment

jainaashish

Thanks to all of you for taking your time to review this. I've tried to respond to each and every comment and will follow up with offline discussion if required to clarify some of the doubts.

jainaashish · 2018-08-14T17:14:39Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

+                                lockFile: newAssetsFile,
+                                previousLockFile: _request.ExistingLockFile,
+                                lockFilePath: _request.ExistingLockFile?.Path,
+                                cacheFile: null,


responded above, would like to keep it this way until there is a valid scenario instead to pass cache file details

jainaashish · 2018-08-14T17:16:26Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

@@ -313,12 +438,36 @@ private bool VerifyAssetsFileMatchesProject()
            return (_request.ExistingLockFile != null && pathComparer.Equals( _request.ExistingLockFile.PackageSpec.FilePath , _request.Project.FilePath));
        }

-        private KeyValuePair<CacheFile, bool> EvaluateCacheFile()
+        private bool ValidatePackages(NuGetLockFile lockFile, LockFile assetsFile)


I thought it was self-explanatory but open for suggestions. How about ValidatePackagesSha or ValidatePackagesSha512 etc...

jainaashish · 2018-08-14T17:19:07Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

        {
            CacheFile cacheFile;
            var noOp = false;

-            var newDgSpecHash = NoOpRestoreUtilities.GetHash(_request);
+            var newDgSpecHash = dgSpec.GetHash();


No it isn't... come out of NoOpRestoreUtilities and then think, we're generating dgSpec which can and should be used down further if required... Generating dgSpec isn't entirely tied up with NoOp check... it's just that NoOp check is the first which needs to generate it.

jainaashish · 2018-08-14T17:21:17Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreRequest.cs

@@ -163,5 +163,7 @@ public class RestoreRequest
        public SignedPackageVerifierSettings SignedPackageVerifierSettings { get; set; } = SignedPackageVerifierSettings.GetDefault();

        public Guid ParentId { get; set;}
+
+        public bool IsRestore { get; set; } = true;


For the current problem, We only need to differentiate between "restore" and install/ update hence a boolean works fine. I'll talk to Zhi if he wanted to do something similar and how that can be clubbed here.

jainaashish · 2018-08-14T17:21:51Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreResult.cs

+        /// <summary>
+        /// NuGet lock file path
+        /// </summary>
+        public string NuGetLockFilePath { get; }


As suggested above, how about PackagesLockFilePath?

jainaashish · 2018-08-14T18:23:54Z

src/NuGet.Core/NuGet.ProjectModel/ProjectLockFile/NuGetLockFileFormat.cs

+            return target;
+        }
+
+        private static LockFileDependency ReadTargetDependency(string property, JToken json)


Same as I responded above

jainaashish · 2018-08-14T18:30:12Z

src/NuGet.Core/NuGet.ProjectModel/ProjectLockFile/NuGetLockFileUtilities.cs

+            {
+                path = Path.Combine(project.BaseDirectory, "packages." + project.RestoreMetadata.ProjectName.Replace(' ', '_') + ".lock.json");
+
+                if (!File.Exists(path))


File.Exists is case insensitive and it can't be confusing.
I didn't understand what you mean by this?

Also, why should we use ProjectPath when there is already ProjectName? Are there cases where ProjectPath is set but not ProjectName in restore metadata? if not, then I'll lean towards using ProjectName and may be fixed that at other places going forward...

jainaashish · 2018-08-14T18:34:48Z

src/NuGet.Core/NuGet.ProjectModel/ProjectRestoreMetadata.cs

+        /// <summary>
+        /// Set when customer wants to opt into packages lock file
+        /// </summary>
+        public string RestorePackagesWithLockFile { get; set; }


there are actually three cases of this property - Null, true, and false. Most cases Null and false are treated same except when you also have an explicit packages.lock.json file... in that case null will be accepted but false is an error case. This is explained in the spec when someone set these properties.

jainaashish · 2018-08-14T18:36:40Z

...Get.PackageManagement.VisualStudio.Test/ProjectSystems/LegacyPackageReferenceProjectTests.cs

-            public IProjectSystemCapabilities Capabilities { get; } = Mock.Of<IProjectSystemCapabilities>();
+        public T GetGlobalService<T>() where T : class
+        {
+            throw new NotImplementedException();


As mentioned above I'm not a fan of inline functions... I haven't tried to change anything in existing apis but new apis have proper method signature

jainaashish · 2018-08-14T18:37:43Z

...ients.Tests/NuGet.PackageManagement.VisualStudio.Test/ProjectSystems/TestVSProjectAdapter.cs

+
+namespace NuGet.PackageManagement.VisualStudio.Test
+{
+    public class TestVSProjectAdapter : IVsProjectAdapter


yes to internal class and no to inline function

nkolev92 · 2018-08-16T05:04:11Z

...NuGet.Clients/NuGet.PackageManagement.VisualStudio/Projects/LegacyPackageReferenceProject.cs

+                        warningsAsErrors: _vsProjectAdapter.WarningsAsErrors),
+                    RestoreLockProperties = new RestoreLockProperties(
+                        await _vsProjectAdapter.GetRestorePackagesWithLockFileAsync(),
+                        await _vsProjectAdapter.GetNuGetLockFilePathAsync(),


Are you going to change these as well to align with the eventual NuGetLockFile name change?

No, these are fine since the MSBuild property name itself is NuGetLockFilePath

rohit21agrawal · 2018-08-16T05:58:54Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RequestFactory/RestoreArgs.cs

@@ -62,6 +62,8 @@ public class RestoreArgs

        public Guid ParentId { get; set; }

+        public bool IsRestore { get; set; } = true;


make this an enum which actually identifies what operation it is.

As it was discussed in earlier comments conversation for the same topic with @nkolev92, currently I only need to differentiate between restore n rest which is why a simple bool. There is no other specific ask to differentiate between other operations at this time. Also, just adding a enum here wouldn't do anything since we'll need to plump it through all the way from UI context to differentiate between Install or Update...

@zhili1208 was looking at this while working for NU1605 issue but realized this will be a bigger ask so we'll add enum or operation types when take up that work which is not planned yet and could take years to become reality.

rohit21agrawal · 2018-08-16T06:01:51Z

src/NuGet.Core/NuGet.Commands/PackagesLockFileBuilder.cs

+            };
+
+            var libraryLookup = assetsFile.Libraries.Where(e => e.Type == LibraryType.Package)
+                .ToDictionary(e => new PackageIdentity(e.Name, e.Version));


why do you need a dictionary if there is no key value pair? convert the IEnumerable to a HashSet instead.

Look closely, that is actually a dictionary where key is PackageIdentity and value is LockFIleLibrary

nkolev92

The 2nd of the first pass.

nkolev92 · 2018-08-16T05:46:46Z

src/NuGet.Core/NuGet.ProjectModel/ProjectLockFile/PackagesLockFileUtilities.cs

+
+            var path = project.RestoreMetadata.RestoreLockProperties.NuGetLockFilePath;
+
+            if (string.IsNullOrEmpty(path))


In whcih case will this be null?

If it's only tests, i'd really try to make sure that the tests populate it as often as possible.

That way you case you can just call project.RestoreMetadata?.RestoreLockProperties.NuGetLockFilePath

When user doesn't set this NuGetLockFilePath property explicitly and instead wants to use the default file name n path then it will be null. This is not for tests. You can find more details in specs.

If it's restore metadata, you should generate it at the beginning similar to what we do with the assets and cache files.

again not able to understand your concerns here as well... may be we speak different language. its to know the lock file path when we start to evaluate it now i dont understand what you mean by at the beginning...?

Again lets discuss offline

nkolev92 · 2018-08-16T05:51:10Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

+
+                var restorePackagesWithLockFile = _request.Project.RestoreMetadata?.RestoreLockProperties.RestorePackagesWithLockFile;
+
+                if (!MSBuildStringUtility.IsTrueOrEmpty(restorePackagesWithLockFile) && File.Exists(nuGetLockFilePath))


style:

The culprit isn't this PR solely, but this method is now around ~400 lines.
You can refactor out some smaller methods to make it more readable.
Additionally I think there are too many exit points in this method (~4 different places that return, at least one can be minimized)

I'll try to break it down further, but the problem was this method already had NoOp and actual restore. And this new code has be in between some where using or forwarding most of the existing stuff.

nkolev92 · 2018-08-16T06:03:12Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

+                        // validate package's SHA512
+                        _success &= ValidatePackagesSha(nuGetLockFile, assetsFile);
+
+                        // clear out the existing lock file so that we don't over-write the same file


I don't like this pattern.
The restore result should contain the up to date latest value of the lock file not a null if it wasn't updated.

If you're worried about the comparison (and I'd even say that's the right approach), I'd add another flag saying if the lock file has been updated and whether it needs writing.

The restore result should contain the up to date latest value of the lock file not a null if it wasn't updated.

Restore result is only used to commit files afterwards and in this case we've used the existing file to perform restore. There is no reason to keep persisting this value. But may be i'm missing something so please enlighten me with your concerns here or we can talk about this offline.

Think about it from an API perspective.
I run 2 consecutive restores.
Nothing get overriden, no new cache, no new assets file, no new lock file.
Yet the API returns me 2 different results.

One that contains the lock file another one that doesn't.
That's bad.

You run 2 restores -

first restore:
a. fresh restore - so nothing already exists n a new cache file, assets file and lock file gets created and returned.
b. noop restore - everything was there so new lock file and neither was returned.
c. restore changed - there was cache file n assets file but no lock file. So this restore generated a new cache file, assets file and lock file which are being returned.
……………. similarly try any other scenarios

second restore: there are always these files so no new cache file, no assets file n no lock file so nothing should be returned. But unfortunately we still return cache file and assets file and I don't want that unfortunate thing with lock file as well.

It's not an unfortunate thing.
It's by design.
Let's add this to the list of topics to discuss offline :)

nkolev92 · 2018-08-16T06:04:53Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

        {
            CacheFile cacheFile;
            var noOp = false;

-            var newDgSpecHash = NoOpRestoreUtilities.GetHash(_request);
+            var newDgSpecHash = dgSpec.GetHash();


Refer to https://github.com/NuGet/NuGet.Client/pull/2342/files#r210479950.

You already have the dg spec on the requst. Use that one.
You don't need to generate it.
That dg spec is being cloned there because it's being modified for the no-op dg spec hashing.

nkolev92 · 2018-08-16T06:11:20Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreResult.cs

@@ -157,23 +172,26 @@ public virtual async Task CommitAsync(ILogger log, CancellationToken token)

            BuildAssetsUtils.WriteFiles(buildFilesToWrite, log);

+            if (result.LockFile == null || result.LockFilePath ==  null)


What's the benefit of this?

This never happens because otherwise we would have had a runtime error.

Maybe the correct approach is to null check in the constructor of the result.
Here it just hides potential problems.

nkolev92 · 2018-08-16T07:27:24Z

src/NuGet.Core/NuGet.ProjectModel/ProjectLockFile/LockFileDependency.cs

+                return true;
+            }
+
+            return PathUtility.GetStringComparerBasedOnOS().Equals(Id, other.Id) &&


The package id is case insensitive.

not for linux... packageA and packagea are treated different packages

Take my words back, it's case intensive

nkolev92 · 2018-08-16T07:36:37Z

src/NuGet.Core/NuGet.ProjectModel/ProjectRestoreMetadata.cs

@@ -196,7 +200,8 @@ public ProjectRestoreMetadata Clone()
                Sources = Sources?.Select(c => c.Clone()).ToList(),
                TargetFrameworks = TargetFrameworks?.Select(c => c.Clone()).ToList(),
                Files = Files?.Select(c => c.Clone()).ToList(),
-                ProjectWideWarningProperties = ProjectWideWarningProperties?.Clone()
+                ProjectWideWarningProperties = ProjectWideWarningProperties?.Clone(),
+                RestoreLockProperties = RestoreLockProperties?.Clone()


Update the tests for this.

Ping.
Add tests for the cloning.

nkolev92 · 2018-08-16T07:43:40Z

src/NuGet.Core/NuGet.Commands/PackagesLockFileBuilder.cs

+                    };
+
+                    var framework_dep = framework?.Dependencies.FirstOrDefault(
+                        dep => PathUtility.GetStringComparerBasedOnOS().Equals(dep.Name, library.Name));


should this be case insensitive?

If you look at where this is coming from, it's from LibraryRange.
LibraryRange equals compares the name as case insensitive.

As i mentioned above, package name is case sensitive for linux and when we're enabled cross platforms we should take care of that. Currently we dont have consistent code across classes which is why we end up in comparison issues as well. But most people don't create multiple packages only difference in casings

Just checked the origin, and indeed it's case insensitive.

nkolev92 · 2018-08-16T07:51:17Z

src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs

+                                    .ToList();
+
+                                // add lock file libraries into RemoteWalkContext so that it can be used during restore graph generation
+                                contextForProject.LockFileLibraries.Add(new LockFileCacheKey(target.TargetFramework, target.RuntimeIdentifier), libraries);


I don't see a reason why you need a mapping for the libraries.
You only need a set of all the libraries. You never make use of the mapping.

We need this mapping to resolve dependencies which happens in ResolverUtility

nkolev92 · 2018-08-16T07:52:41Z

src/NuGet.Core/NuGet.DependencyResolver.Core/ResolverUtility.cs

        }

        public static async Task<GraphItem<RemoteResolveResult>> FindLibraryEntryAsync(
            LibraryRange libraryRange,
            NuGetFramework framework,
+            string runtimeIdentifier,


I don't think you need the runtime identifier per the other comment.

We do because we're being smart enough to not just copy all target graphs across tfm and tfm+rid in new packages.lock.json file rather tfm+rid will only have the changed dependencies in tfm+rid.

So when we come here to resolve all dependencies for tfm+rid, we need to understand the rid in order to get the corresponding dependency.

Go through the code changes of ResolverUtility if you still dont understand then let me know i'll try to add more comments there to make it understandable.

I was under the impression that we don't need them and you never truly use them.

I'll bug you offline if I reach the same conclusion when I go over it again.

I still think this can be changed.
Let's discuss offline.

nkolev92 · 2018-08-16T07:59:19Z

src/NuGet.Clients/NuGet.PackageManagement.VisualStudio/Projects/VsProjectAdapter.cs

+            return MSBuildStringUtility.IsTrue(value);
+        }
+
+        public async Task<bool> IsReevaluateNuGetLockFileAsync()


If I'm reading this correctly...this would allow users to overwrite perpetually correct?
Is that something we really want in Visual Studio?
Furthermore I don't remember us discussing and agreeing on this at any point, nor is it in the spec.

This is to support --reevaluate flag with dotnet.exe this is not going to be an explicit property available to be set which is why its not documented in the spec but you can read about --reevaluate there.

And just to explain, this flag is to reevaluate restore graph even if you had a lock file in order to support floating versions or ranges.

I get that, this flag should be similar to RestoreForce.

Used in commandline and the restore task but not in VS.

Then why are you reading it in Visual Studio? This is my biggest concern.

because it will be there in visual studio as well in form of a UI gesture n i dont want two different code paths for the same option as we're doing it for force...

we can discuss this offline as well.

jainaashish · 2018-08-17T17:14:05Z

@NuGet/nuget-client addressed all the comments so far and tried to make it more readable n understandable.

nkolev92 · 2018-08-17T18:32:47Z

@jainaashish
It's looking good so far.

There are couple of comments from the previous review that I followed up on.
The biggest one is reading the reevaluate property in VS.

Let me know when you've gone through those and I can do another top to bottom pass.

jainaashish · 2018-08-17T18:41:18Z

@nkolev92 it seems right thing would be to discuss some of your concerns offline and resolve. I'm WFH today so lets do that on monday.

jainaashish · 2018-08-21T17:15:34Z

@NuGet/nuget-client Team, there is one open item regarding - https://github.com/NuGet/NuGet.Client/pull/2342/files#diff-ae3ea71e74915c8f0634a376599d6b87R290 which needs to be discussed and finalize.

I don't agree to return the packages lock file instance as part of RestoreResult when we didn't change anything in existing instance which is logical and also prevents us from unnecessary over-writing of the same file. But Nikolche think otherwise and recommending to return the same instance. So would love to understand what others think sothat we can proceed to merge PR.

nkolev92 · 2018-08-22T19:04:08Z

@jainaashish can the telemetry and any future additions be in a separate PR?
Feels like playing catch-up here :)

jainaashish · 2018-08-22T19:30:36Z

@nkolev92 No more :)

actually telemetry change was pretty small and made sense to include as part of this PR to let basic telemetries flow...

jainaashish · 2018-09-07T00:33:24Z

@NuGet/nuget-client 🔔

nkolev92 · 2018-09-10T18:07:44Z

...NuGet.Clients/NuGet.PackageManagement.VisualStudio/Projects/LegacyPackageReferenceProject.cs

+                    RestoreLockProperties = new RestoreLockProperties(
+                        await _vsProjectAdapter.GetRestorePackagesWithLockFileAsync(),
+                        await _vsProjectAdapter.GetNuGetLockFilePathAsync(),
+                        await _vsProjectAdapter.IsRestoreLockedAsync())


We might be repeating the same conversation that we've had before, but do we really want this?

What's the precedence order if you use a reevaluate switch from the UI?

I'm not arguing for or against yet, I'm just trying to understand the behavior.

This is RestoreLockedMode property which is to restrict restore updating lock file. This will primarily be for CI machines so that restore doesn't update lock file but dev boxes should always allow which is why default is allowed. But if this property is set even on dev boxes then we have to respect that, hence this code.

This flag is mentioned in the spec with explanation. If you think this is not clear then may be we can ask Anand to make it more clear. - https://github.com/NuGet/Home/wiki/Enable-repeatable-package-restore-using-lock-file

nkolev92 · 2018-09-10T18:12:08Z

src/NuGet.Core/NuGet.Build.Tasks/NuGet.targets

@@ -119,7 +119,8 @@ Copyright (c) .NET Foundation. All rights reserved.
      RestoreRecursive="$(RestoreRecursive)"
      RestoreForce="$(RestoreForce)"
      HideWarningsAndErrors="$(HideWarningsAndErrors)"
-      Interactive="$(NuGetInteractive)"/>
+      Interactive="$(NuGetInteractive)"
+      ReevaluateRestoreGraph="$(ReevaluateRestoreGraph)"/>


spec: This is not documented anywhere in the specs.

This is the reevaluate lock file, hose everything (well almost) right?

this is the implementation part of --reevaluate flag passed with dotnet.exe restore Since this is not explicitly allowed or needed for msbuild hence not mentioned in the spec.

And yes, it just skip existing lock file and re-evaluate all the dependencies.

nkolev92 · 2018-09-10T18:14:46Z

...uGet.Core/NuGet.Commands/RestoreCommand/RequestFactory/DependencyGraphSpecRequestProvider.cs

@@ -154,8 +154,7 @@ private static ExternalProjectReference GetExternalProject(PackageSpec rootProje
                //  Project.json is special cased to put assets file and generated .props and targets in the project folder
                RestoreOutputPath = project.PackageSpec.RestoreMetadata.ProjectStyle == ProjectStyle.ProjectJson ? rootPath : project.PackageSpec.RestoreMetadata.OutputPath,
                DependencyGraphSpec = projectDgSpec,
-                MSBuildProjectExtensionsPath = projectPackageSpec.RestoreMetadata.OutputPath,
-                ParentId = restoreArgs.ParentId


Why is the ParentId removed?

This is being done through ApplyStandardProperties

nkolev92 · 2018-09-10T18:24:31Z

src/NuGet.Core/NuGet.Commands/Strings.resx

+    <comment>{0} - package id and version</comment>
+  </data>
+  <data name="Error_InvalidLockFileInput" xml:space="preserve">
+    <value>Invalid restore input where RestorePackagesWithLockFile property is set to false and also exist a packages lock file at {0}.</value>


nkolev92 · 2018-09-10T18:28:34Z

src/NuGet.Core/NuGet.ProjectModel/JsonPackageSpecReader.cs

@@ -329,6 +329,17 @@ private static ProjectRestoreMetadata GetMSBuildMetadata(PackageSpec packageSpec
                msbuildMetadata.ProjectWideWarningProperties = new WarningProperties(warnAsError, noWarn, allWarningsAsErrors);
            }

+            // read NuGet lock file msbuild properties
+            var restoreLockProperties = rawMSBuildMetadata.GetValue<JObject>("restoreLockProperties");


nit: I know you are following a pattern here, but having a constant would help avoiding weird bugs like case inconsistency.

It's up to the author who is adding new property needs to make sure to use the correct casing, even with constant, we can fault into that issue. Also, it will be little weird with constant since then it will be something like:

public static readonly string RestoreLockProperties = "restoreLockProperties";

nkolev92 · 2018-09-10T18:32:36Z

test/TestUtilities/Test.Utility/PackageManagement/TestDependencyInfoProvider.cs

@@ -0,0 +1,75 @@
+using System;


Missing header

nkolev92 · 2018-09-10T18:32:47Z

test/TestUtilities/Test.Utility/PackageManagement/TestMetadataProvider.cs

@@ -0,0 +1,90 @@
+using System;


missing header

nkolev92 · 2018-09-10T18:33:52Z

test/TestUtilities/Test.Utility/Test.Utility.csproj

@@ -19,6 +19,21 @@
    <ProjectReference Include="$(NuGetClientsSrcDirectory)NuGet.CommandLine\NuGet.CommandLine.csproj" />
  </ItemGroup>

+  <Choose>


I don't get what this change achieves.

Seems unnecessary as I don't see any functional differences.

i dont know why it changed this. But will revert as part of squash n merge.

nkolev92 · 2018-09-10T18:34:29Z

src/NuGet.Core/NuGet.ProjectModel/ProjectRestoreMetadata.cs

@@ -196,7 +200,8 @@ public ProjectRestoreMetadata Clone()
                Sources = Sources?.Select(c => c.Clone()).ToList(),
                TargetFrameworks = TargetFrameworks?.Select(c => c.Clone()).ToList(),
                Files = Files?.Select(c => c.Clone()).ToList(),
-                ProjectWideWarningProperties = ProjectWideWarningProperties?.Clone()
+                ProjectWideWarningProperties = ProjectWideWarningProperties?.Clone(),
+                RestoreLockProperties = RestoreLockProperties?.Clone()


Ping.
Add tests for the cloning.

nkolev92 · 2018-09-10T18:46:43Z

src/NuGet.Core/NuGet.DependencyResolver.Core/ResolverUtility.cs

        }

        public static async Task<GraphItem<RemoteResolveResult>> FindLibraryEntryAsync(
            LibraryRange libraryRange,
            NuGetFramework framework,
+            string runtimeIdentifier,


I still think this can be changed.
Let's discuss offline.

jainaashish · 2018-09-10T23:28:15Z

@nkolev92 Added missing headers and a cloning test.

generating/ updating project lock file consuming project lock file during restore in VS Validation of package's sha512 and added error messages as appropriate Test infra for legacy PackageReference func tests Base workflow is completed Added telemetry events for project lock file feature

jainaashish requested review from mishra14, zhili1208, nkolev92, rohit21agrawal and PatoBeltran July 19, 2018 06:20

jainaashish changed the title ~~[wip] Repeatable build using Project lock file implementation~~ Repeatable build using Project lock file implementation Jul 19, 2018

jainaashish force-pushed the dev-asja-projectLockFile branch from 74b458c to e19ac3d Compare July 29, 2018 09:27

jainaashish force-pushed the dev-asja-projectLockFile branch from e19ac3d to 2fc37b3 Compare July 31, 2018 06:59

jainaashish force-pushed the dev-asja-projectLockFile branch 3 times, most recently from df760da to 0b0e8bd Compare August 8, 2018 23:19

jainaashish removed request for rohit21agrawal, PatoBeltran and mishra14 August 8, 2018 23:21

PatoBeltran reviewed Aug 9, 2018

View reviewed changes

rohit21agrawal reviewed Aug 13, 2018

View reviewed changes

jainaashish force-pushed the dev-asja-projectLockFile branch from e5b2e9e to 2469687 Compare August 14, 2018 00:13

nkolev92 reviewed Aug 14, 2018

View reviewed changes

jainaashish commented Aug 14, 2018

View reviewed changes

nkolev92 reviewed Aug 16, 2018

View reviewed changes

rohit21agrawal reviewed Aug 16, 2018

View reviewed changes

nkolev92 reviewed Aug 16, 2018

View reviewed changes

jainaashish force-pushed the dev-asja-projectLockFile branch from 0239ec2 to 22054ce Compare August 17, 2018 06:54

jainaashish force-pushed the dev-asja-projectLockFile branch from 9148c22 to 63e2d23 Compare August 31, 2018 19:04

nkolev92 reviewed Sep 10, 2018

View reviewed changes

jainaashish force-pushed the dev-asja-projectLockFile branch from 744b763 to 75ea2fc Compare September 10, 2018 23:16

nkolev92 approved these changes Sep 10, 2018

View reviewed changes

jainaashish force-pushed the dev-asja-projectLockFile branch from 75ea2fc to bb6c399 Compare September 11, 2018 23:15

jainaashish merged commit a933f23 into dev Sep 12, 2018

jainaashish mentioned this pull request Sep 14, 2018

[PackageReference] Enable repeatable build using project lock file NuGet/Home#7139

Closed

jainaashish deleted the dev-asja-projectLockFile branch October 1, 2018 22:10

jerriep mentioned this pull request Feb 20, 2020

Support for NuGet package lock files dependabot/dependabot-core#1303

Open

		@@ -157,23 +172,26 @@ public virtual async Task CommitAsync(ILogger log, CancellationToken token)

		BuildAssetsUtils.WriteFiles(buildFilesToWrite, log);

		if (result.LockFile == null \|\| result.LockFilePath == null)

		@@ -62,6 +62,8 @@ public class RestoreArgs

		public Guid ParentId { get; set; }

		public bool IsRestore { get; set; } = true;


		var path = project.RestoreMetadata.RestoreLockProperties.NuGetLockFilePath;

		if (string.IsNullOrEmpty(path))


		var restorePackagesWithLockFile = _request.Project.RestoreMetadata?.RestoreLockProperties.RestorePackagesWithLockFile;

		if (!MSBuildStringUtility.IsTrueOrEmpty(restorePackagesWithLockFile) && File.Exists(nuGetLockFilePath))

Repeatable build using Project lock file implementation #2342

Repeatable build using Project lock file implementation #2342

Conversation

jainaashish commented Jul 12, 2018 • edited

jainaashish commented Jul 31, 2018

jainaashish commented Aug 8, 2018 • edited

PatoBeltran left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkolev92 Aug 14, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkolev92 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jainaashish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jainaashish commented Jul 12, 2018 •

edited

jainaashish commented Aug 8, 2018 •

edited

nkolev92 Aug 14, 2018 •

edited