build failure: error NETSDK1004: Assets file '...' not found. Run a NuGet package restore to generate this file. #45755
Comments
|
Do you have the text log of what the initial build.sh command printed? Maybe you also have a binlog? Restore should have taken care of restoring the |
|
I'm curious if this actually complains for |
|
master is always failing for me. This is the console output: cout.txt. |
|
If you need it, I can provide you with a binlog if you tell me how to generate it and where I can find it. I don't use them because they are not very usable on Linux. |
|
So yes, this is a restore issue: We have been such cases from time to time where NuGet fails to restore certain package but doesn't fail the build. |
Sure, just add the
There's a webview available for it now: https://live.msbuildlog.com/ :) |
I've run this 5-ish times. It's weird that it's always missing exactly the same packages.
I don't feel it scales for the logs I throw at it.
Running gave me: Is this what you expect? |
The binlog should be located in the CWD.
Fair.
I don't understand that either. I would recommend to wait a few hours and try it again and let us please know. |
|
Tagging subscribers to this area: @ViktorHofer Issue DetailsWhen I build this repo using: My build fails with: @ViktorHofer do you have an idea what may be causing this?
|
|
I just hit this when running I believe this is happening due to the 5.0.0 SDK update on global.json. The problem seems to be manifesting whenever you don't have runtime/Directory.Build.targets Lines 31 to 40 in 10381a2 Because
I think we should update |
|
Yeah I just confirmed that if I update |
|
Actually, do we need this ref at all for our libraries projects? I believe we should remove this KnownFrameworkReference, we should be building against live references for |
|
It looks like we disable it already but only for inbox libraries: runtime/src/libraries/Directory.Build.targets Lines 23 to 24 in 5432a66 Note that when building libraries src projects, i.e: |
|
From the logs, I see a bunch of different restores are running.
|
|
Yes, we invoke restore three times:
Would that cause a problem? In the past we had 30+ restore invocations... |
For which configuration? For runtime/eng/targetingpacks.targets Line 6 in dcbf6e9 |
I don't see a |
Is someone looking into this? |
|
I just replied to this thread 30min ago, so yes, we are looking into this :) |
|
Multiple invocations would not cause a problem, but it makes it difficult to understand which operation is actually failing. |
ghost
added
the
in-pr
ghost
removed
the
in-pr
|
I still can't build. I hit a different error, also restore related. @safern @ViktorHofer does a build work on a fresh machine for you? |
|
Taking a closer look now. |
|
I haven't seen this binlog corruption before, it fails immediately when trying to read the first buffer from the stream: I don't have the expertise to dig deeper unfortunately. This is where the binlog creation logic lives in MSBuild: |
Looking at https://github.com/dotnet/arcade/blob/7d638bb082f374f7c8ddbada79f757662df4a246/src/Microsoft.DotNet.Build.Tasks.Packaging/src/GetLastStablePackage.cs#L139, the task doesn't log an error. It's possible that restore is skipped, rather than failed. We just don't have enough logging right now to be able to confirm that. Have you been able to narrow it down to a repro? |
Unfortunately not yet. I'm trying to reproduce the error via #46134 but haven't hit the error yet... That PR includes generating the static graph restore evaluation binlog that would tell us more.
I think we should definitely do that. It's not clear to me if the static graph restore nuget console app a) failed, b) succeeded but actually swallowed an exception and didn't do anything (my guess) or c) succeeded. I believe we can rule out the latter as the build later fails because of the missing project.assets.json and other nuget generated files.
What would skipped mean? Even if nothing is restored, wouldn't I still see some of the logging when projects are evaluated and the graph is constructed that I see in other binlogs in incremental restore operations?
Right. It's unclear to me if something inside that task failed. That message is only an informational logging. |
Not every project type supports restore, so when run on a solution, NuGet might skip some projects. That's a good behavior in many cases. Recently we investigated an issue where I'm making this guess because we have not had issues with restore swallowing exceptions before, but skipping projects has happened in both static and non-static graph restores. |
|
Wouldn't static graph restore skipping projects indicate a deterministic behavior? What I was seeing in CI was a one-off failure. The issue that @tmds reported originally in this issue is a timeout and probably a different cause (you responded offline). |
One would hope :) We can add some extra logs in static graph restore, but I'm not sure how quickly that propagates. |
|
@nkolev92 would you mind filing a bug to add better logging to RestoreTaskEx? |
|
I added a comment to expand NuGet/Home#10363. |
|
I still can't build the runtime repo. I'm still seeing errors from #45755 (comment). I sent you a mail with logs. It's weird no one else is seeing this. I created a new user on my system to rule out something in my HOME causes the problem. Also for this user, I get the same errors. |
tmds
changed the title
I thought about what still might be specific about my machine, so ran I have then added back the versions I had installed: So, the issue I have gets caused by the @ViktorHofer can we change the build script so it prefers cc @omajid |
@ViktorHofer is this an appropriate change? I can look into making it. |
I believe this was designed that way so that builds were faster if we found a compatible SDK installed machine-wide to the version we specify in global.json. |
|
Hey Tom, sorry for getting back to you so late. We intentionally allow globally installed sdks to avoid the unnecessary download which can be quite big. I think we should diagnose why this is happening with mixed globally installed sdks as this could also happen in any other customer project... |
|
Also show we show the binlogs sent by Tom to an expert in compression to understand why a small binlog is fine, but large binlogs don't extract properly? |
|
cc: @carlossanlop for the compression question. |
|
When I go to @tmds links above, I just get the RedHat logo. It sounds like we need a bug opened with an example of the file that was created with .NET, but cannot be opened by .NET (but can be opened by gzip) |
|
@danmosemsft I have the binlogs locally, I'll open a bug and provide a link to internal-only binlog downloads |
|
I've filed #47024 |
Is there something that verifies the build works if you meet the requirement in
Is there an option to not use globally installed sdks? If not, can we add one? |
It's currently a direct match. If the installed SDK matches the specified version in global.json, it's a match. There's no support for patch/minor/major roll forward: dotnet/arcade#6636. This means that the SDK is fetched ie if you have a 5.0.102 installed. The corresponding logic can be found here: https://github.com/dotnet/arcade/blob/8ca877b54001e682f9642d8995f2a01616ac3fd4/eng/common/tools.sh#L153.
Me and many others still have 5.0.100 installed (which on Windows is the case if you don't update your VS installation often or Windows Update isn't triggered) and it works fine. Based on what you described earlier this seems to be connected to have multiple SDKs installed which interfere with each other somehow.
@marcpopMSFT @vitek-karas @dsplaisted have we heard of such issues in the past where installing previous shipped SDKs change the behavior of the latest one installed and causes the discussed restore issues?
Yes, that's possible but we won't be able to check the code in as with that we would loose the optimization. Here's how you would do it: https://github.com/dotnet/coreclr/blob/9fb9458bed5f812b88a69a02f795b738a5a8f167/eng/configure-toolset.sh#L1-L5. You need that file in the eng folder containing the first 5 lines. |
|
My system sdk is now 5.0.102 and no longer the same version as required by dotnet/runtime. When I change: diff --git a/global.json b/global.json
index f271ad70ec6..a5dba62247e 100644
--- a/global.json
+++ b/global.json
@@ -1,11 +1,11 @@
{
"sdk": {
- "version": "5.0.100",
+ "version": "5.0.102",
"allowPrerelease": true,
"rollForward": "major"
},
"tools": {
- "dotnet": "5.0.100"
+ "dotnet": "5.0.102"
},
"native-tools": {
"cmake": "3.16.4",the build also works. I won't try to reproduce and investigate further. |
ghost
locked as resolved and limited conversation to collaborators
and others
When I build this repo using:
My build fails with:
@ViktorHofer do you have an idea what may be causing this?
The text was updated successfully, but these errors were encountered: