New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build fails due to locked file which it locks itself (concurrent build problem) #9585
Comments
maybe @rainersigwald can help here. In the meantime, it would really help if you'd be able to get a build log ( |
Yes, a binary log (or a diagnostic log) would be helpful here. It would probably be helpful even from a build that did not fail with this problem--I suspect that there is a race condition in your build such that some projects are building twice and sometimes they're concurrent. Also good to know: when it occurs, is the error always in the same project ( |
Here's the binlog with failure. So far the problem seems to occur only with Integration.Kardex.csproj, but with different files (In the attached log it is Integration.Kardex.csprojAssemblyReference.cache). On CI machine the error happens much more frequently (50% of times) than on my dev laptop. There is no other tooling running, the CI machine even has no VS installed. |
I guess I'll be going back to having one project collecting all the dependencies and copying them to common folder. Why oh why is msbuild so dumb about these things? |
From that log:
That indicates that As shown in that image, both of the racing instances of the project actually have the same TF and RID, which is where I expected to see the global property change. Unfortunately, the binary log wasn't captured with an MSBuild that has dotnet/msbuild#3252 (fixed in 15.8 that's not yet released), so it's pretty hard to figure out what properties are different. Based on testing on a simplified project (one mvc project referencing one But that doesn't seem to be what's happening here, because the query-TFs call didn't collapse into the build-single-TF call. I can't tell why from the logging; I'm getting lost in the various different properties set in different projects. There's definitely a problem here. At the moment I would suggest running |
This is the second issue about users getting into problems with Does it seem reasonable if the set of scenarios where |
(I continue to be amazed by the work @rainersigwald puts into explaining build issues. now its full-blown diagrams, I still remember some hand-drawn diagrams about RID conflicts ^^) |
I believe the problem with slns was first clearly identified with those famous diagrams. :) Yes, we should warn. In 3.0, we should either disallow it or actually make it work somehow. |
Thanks for suggestions! Is there somewhere this unreleased version? I could try that to see if I can get better logs for you to find the cause. And I may add that, from an outside perspective, this sounds reeeally scary. After being told in the past that you can't ship what "dotnet build" produces, now looking at "dotnet publish" not working for solutions I see just lots of question marks. I'm building a product, which consists of many (C#) projects, having to write a nontrivial script which would publish all the projects, while still using my 16 cores sounds like... writing what a modern build system should do flawlessly. Maybe it's time to rearchitect msbuild to make it smart. Something like bazel. ;) |
@rainersigwald the problem started appearing during normal build when specifying 2 Logs attached, 1 failed, 1 successful. Can I get the 15.8 version somewhere so I can get better logs? |
I second the vote for "make it work somehow". Telling people they have to manually build every project separately is ridiculous. People expect to just build a solution and for the build engine to work. |
So.. Is there a solution to this? It disrupts our automated build processes.. |
Any updates on this ? |
I have encountered this issue in our build/CI environment, but I encountered it when upgrading the VM doing the builds to 2 CPU cores rather than 1. Since having working builds is generally more useful than faster builds, I've worked around this by downgrading the VM back to a single core, so it doesn't try to build projects in parallel... so that's one "solution" at least where you have that much control over your build environment. I never get this problem on my local machine whilst building for debug/test, but I never try to publish locally either. The builds I have trouble with are being run by TeamCity's ".NET CLI (dotnet)" build step, building .NET core 2.2 solutions. Having read this, I may attempt to change this to build projects rather than solutions, but this seems a bit painful to have to do... A proper fix would be appreciated. |
So, i can repro this parallel build issue extremely simple, and publish is not needed, but Reproduction:
Probably works with different setups equally easy.
|
@Structed The leftover process is expected. You can shut down such processes with If the leftover process is locking files causing subsequent builds to fail, then that is an issue. |
Yes, exactly this is the case: files are being locked.
Can't currently provide logs/repro project. If you need, let me know and I
can provide them Monday.
Am 20. September 2019 18:47:52 schrieb Daniel Plaisted
<notifications@github.com>:
… @Structed The leftover process is expected. You can shut down such
processes with dotnet build-server shutdown.
If the leftover process is locking files causing subsequent builds to fail,
then that is an issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
@Structed Yes, please provide the logs and repro project if possible. |
Repo: https://github.com/burnasheva/mstest_dotnet3.git Cmd: Error:
|
As a sidenode: |
@smaktacular's reccomendation works to fix the build of the actual project, but (for me) breaks the build of the related Docker/docker-compose. Error for the project containing the
Workaround |
Has anything ever come out of this? If I could pass in a list of projects (like when building the solution) that may be helpful, even if have to list them in build order. Possibly tell it to ignore locked files (because these files packages ARE there), |
How are you building the solution? Are you passing any command line arguments? I ask because issues like this can be caused by things like specifying the output path on the command line, and then multiple projects try to build to the same output path and can get file conflicts. |
Yep! I am specifying the output path because I want them all in a single folder when I'm done. dotnet publish mysolution.sln --output /app |
So my VERY temporary workaround... Thanks dsplaisted for your comment it was helpful! |
This is still an issue with .Net 5... is this ever going to get fixed? |
My workaround for this is to disable concurrent build like this:
But a proper fix would certainly be nice :) |
still an issue here with .net5 in November 2021....any updates? |
@southrivertech It looks to me like the issue here has to do with running Is that what you're encountering? If so, what do you expect the |
Doesn't make sense to me because documentation is saying: Specifies the maximum number of concurrent processes to use when building. If you don't include this switch, the default value is 1. For us the fix was to disable antivirus. |
Presumably the msbuild server feature will fix this issue as with it, there will only be a single process doing the work (with multiple threads)? |
@ViktorHofer I don't think that's necessarily true; it'll still be possible to create race conditions where an external process like the C# compiler is trying to write different content to the same file concurrently. |
When invoking |
The default is "there is a single |
Commenting to say we experienced this issue with |
I'm getting this error on Github, during a CI/CD "GitHub Actions". Localhost is fine.
Notice that:
so it looks like all of the projects are copying the build output-assets to the same destination folder .. which is fighting. Not sure why the msbuild-retries all fail. But is this an issue? shouldn't the outputs be a single folder per project, inside the destination folder?
|
For me, removing With |
Still getting this. It becomes a struggle. I have to try 5-6 times (sometimes maybe more) every time I want to publish something. |
Same for me. I want/need to add |
@rainersigwald Hi! |
We moved from .net core 2.0 to 2.1. In this move, we retargeted several projects which were from netstandard2.0 to multitarget netcoreapp2.1 and netstandard2.0. Now we sometimes experience failed builds, both on CI servers and dev machines.
Q: I'd like to provide as much info as I can, so I could try collecting diag build logs and see if I can get one from failed build, would that help?
Steps to reproduce
No stable repro. Start a build using:
dotnet publish -c Release -r win-x64 -f netcoreapp2.1
Expected behavior
Build works
Actual behavior
Sometimes the build stops with something like this:
build 11-Jul-2018 22:23:55 C:\Program Files\dotnet\sdk\2.1.301\Microsoft.Common.CurrentVersion.targets(4364,5): error MSB3371: The file "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-AC-JOB1\test\Integration.Kardex\obj\Release\netcoreapp2.1\win-x64\Integration.Kardex.csproj.CopyComplete" cannot be created. The process cannot access the file 'D:\Atlassian\Bamboo\xml-data\build-dir\TIC-AC-JOB1\test\Integration.Kardex\obj\Release\netcoreapp2.1\win-x64\Integration.Kardex.csproj.CopyComplete' because it is being used by another process. [D:\Atlassian\Bamboo\xml-data\build-dir\TIC-AC-JOB1\test\Integration.Kardex\Integration.Kardex.csproj]
Environment data
dotnet --info
output:.NET Core SDK (reflecting any global.json):
Version: 2.1.301
Commit: 5952487
Runtime Environment:
OS Name: Windows
OS Version: 6.3.9600
OS Platform: Windows
RID: win81-x64
Base Path: C:\Program Files\dotnet\sdk\2.1.301\
Host (useful for support):
Version: 2.1.1
Commit: 6985b9f684
.NET Core SDKs installed:
1.0.4 [C:\Program Files\dotnet\sdk]
1.1.0 [C:\Program Files\dotnet\sdk]
2.0.0 [C:\Program Files\dotnet\sdk]
2.0.2 [C:\Program Files\dotnet\sdk]
2.0.3 [C:\Program Files\dotnet\sdk]
2.1.101 [C:\Program Files\dotnet\sdk]
2.1.201 [C:\Program Files\dotnet\sdk]
2.1.301 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
The text was updated successfully, but these errors were encountered: