New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building WiX projects that generate messages may deadlock in VSTS vNext builds #5486

Closed
dwhearn opened this Issue Feb 3, 2017 · 9 comments

Comments

Projects
None yet
4 participants
@dwhearn

dwhearn commented Feb 3, 2017

Please provide answers to the following questions to help us narrow down, reproduce, and fix the problem. Fill out one section and delete the others.

Bugs

  • Which version of WiX are you building with?

3.10.3.3007

  • Which version of Visual Studio are you building with (if any)?

Visual Studio 2015 Update 2 from VSTS vNext build definition.

  • Which version of .NET are you building with?

.NET 4.5.2

  • If the problem occurs when installing your packages built with WiX, what is the version of Windows the package is running on?

NA

  • Describe the problem and the steps to reproduce it.

Using Visual Studio Team Services new vNext build definition, add a step to build a Visual Studio solution containing a WiX project. If the WiX linker generates any warning or error messages it can deadlock the build.

  • Describe the behavior you expected and how it differed from the actual behavior.

There was a report of the problem and some suggested workarounds on Microsoft's website that has been removed. One of the suggested workarounds was to specify the '‘/p:RunWixToolsOutOfProc=true’ as an MSBuild argument to the build step. This does fix the problem but is not easy to find this as a solution to the build hanging.

This was brought up as an issue with product manager VSTS build at Microsoft and their response was that they had concerns about adding this string to every single project build command and advised opening up a bug report here. The suggestion is for WiX to always run out of process so that it does not deadlock.

@barnson

This comment has been minimized.

Show comment
Hide comment
@barnson

barnson Feb 3, 2017

Member

Is this discussion with VSTS folks public somewhere? It would be useful to understand what's causing the deadlock.

Member

barnson commented Feb 3, 2017

Is this discussion with VSTS folks public somewhere? It would be useful to understand what's causing the deadlock.

@barnson

This comment has been minimized.

Show comment
Hide comment
@barnson

barnson Feb 14, 2017

Member

If you can provide more details, we'll take a look at our next online meeting on 28-Feb.

Member

barnson commented Feb 14, 2017

If you can provide more details, we'll take a look at our next online meeting on 28-Feb.

@dwhearn

This comment has been minimized.

Show comment
Hide comment
@dwhearn

dwhearn Feb 14, 2017

The Microsoft product manager who is our contact with this issue said they would have someone address your questions. But the basic problem is if the WiX linker generates a warning or error message and is running in-process with the vNext build it will normally deadlock and the build will eventually time out.

dwhearn commented Feb 14, 2017

The Microsoft product manager who is our contact with this issue said they would have someone address your questions. But the basic problem is if the WiX linker generates a warning or error message and is running in-process with the vNext build it will normally deadlock and the build will eventually time out.

@barnson

This comment has been minimized.

Show comment
Hide comment
@barnson

barnson Feb 28, 2017

Member

We haven't heard back from Microsoft and aren't experiencing it in the WiX build itself.

Member

barnson commented Feb 28, 2017

We haven't heard back from Microsoft and aren't experiencing it in the WiX build itself.

@ericsciple

This comment has been minimized.

Show comment
Hide comment
@ericsciple

ericsciple Mar 1, 2017

@barnson in VSTS we hook up a custom forwarding logger to msbuild so we can convert errors/warnings into issues that are attached to the build. The logger formats the error with special syntax, and then prints to STDOUT (for the agent to intercept and convert into an issue associated with the build).

So the logger calls Console.WriteLine(string), which is hanging sometimes. Console.WriteLine(string) ends up calling System.IO.TextWriter.SyncTextWriter.WriteLine(string value), which calls this._out.WriteLine(char). From a dump, I saw the local variable state _out is an instance of Microsoft.Tools.WindowsInstallerXml.Build.Tasks.WixToolTask.WixToolTaskLogger and it appears WixToolTaskLogger.Write(char) was deadlocking. We saw this in WixTasks.dll 3.10.1.

ericsciple commented Mar 1, 2017

@barnson in VSTS we hook up a custom forwarding logger to msbuild so we can convert errors/warnings into issues that are attached to the build. The logger formats the error with special syntax, and then prints to STDOUT (for the agent to intercept and convert into an issue associated with the build).

So the logger calls Console.WriteLine(string), which is hanging sometimes. Console.WriteLine(string) ends up calling System.IO.TextWriter.SyncTextWriter.WriteLine(string value), which calls this._out.WriteLine(char). From a dump, I saw the local variable state _out is an instance of Microsoft.Tools.WindowsInstallerXml.Build.Tasks.WixToolTask.WixToolTaskLogger and it appears WixToolTaskLogger.Write(char) was deadlocking. We saw this in WixTasks.dll 3.10.1.

@barnson

This comment has been minimized.

Show comment
Hide comment
@barnson

barnson Mar 14, 2017

Member

https://github.com/wixtoolset/wix3/blob/develop/src/tools/WixTasks/WixToolTask.cs#L382 is how we redirect console output. Can you get more detail about the deadlock? Also, try WiX v3.10.3 (or the new v3.11 RC).

Member

barnson commented Mar 14, 2017

https://github.com/wixtoolset/wix3/blob/develop/src/tools/WixTasks/WixToolTask.cs#L382 is how we redirect console output. Can you get more detail about the deadlock? Also, try WiX v3.10.3 (or the new v3.11 RC).

@ericsciple

This comment has been minimized.

Show comment
Hide comment
@ericsciple

ericsciple Mar 14, 2017

Unfortunately I don't have a repro. A customer ran into the issue and we were able to figure out a workaround.

However, here is a thread that talks about the issue. In that thread someone describes a potential theoretical repro... although the custom logger we use in TFS simply calls Console.WriteLine, so I'm not sure.

ericsciple commented Mar 14, 2017

Unfortunately I don't have a repro. A customer ran into the issue and we were able to figure out a workaround.

However, here is a thread that talks about the issue. In that thread someone describes a potential theoretical repro... although the custom logger we use in TFS simply calls Console.WriteLine, so I'm not sure.

@barnson

This comment has been minimized.

Show comment
Hide comment
@barnson

barnson Mar 14, 2017

Member

Thanks, @ericsciple; that thread should be useful (though I wish someone had reported it back then).

Member

barnson commented Mar 14, 2017

Thanks, @ericsciple; that thread should be useful (though I wish someone had reported it back then).

@barnson barnson added this to the v4.0 milestone Mar 28, 2017

@laedit

This comment has been minimized.

Show comment
Hide comment
@laedit

laedit May 3, 2017

My two cents on this issue: I had a Wix build hanging on VSTS and it appear that building the wixproj on x86 platform with warnings like

warning CNDL1044: The File/@shortname attribute's value 'SYSTEM~1.DLL' is an ambiguous short name because it ends with a '~' character followed by a number. Under some circumstances, this name could resolve to more than one file or directory name and lead to unpredictable results (for example 'MICROS~1' may correspond to 'Microsoft Shared' or 'Microsoft Foo' or literally 'Micros~1').

was the cause.

The RunWixToolsOutOfProc property, setted manually to true in my project, had allow me to unblock the build on VSTS. Once the bad code gone I was able to remove the property without blocking the build again.

I tried to reproduce it on a new wixproj, but the warning doesn't block the build.
That said it appears twice on the log, I don't know if it is relevant:

d:\a\1\s\SetupProject1\Product.wxs(17): error CNDL0026: The Directory/@ShortSourceName attribute's value, 'INSTALL~1', is not a valid 8.3-compliant name. Legal names contain no more than 8 non-period characters followed by an optional period and extension of no more than 3 non-period characters. Any character except for the follow may be used: \ ? | > < : / * " + , ; = [ ] (space). [d:\a\1\s\SetupProject1\SetupProject1.wixproj]
SetupProject1\Product.wxs(17,0): Error CNDL0026: The Directory/@ShortSourceName attribute's value, 'INSTALL~1', is not a valid 8.3-compliant name. Legal names contain no more than 8 non-period characters followed by an optional period and extension of no more than 3 non-period characters. Any character except for the follow may be used: \ ? | > < : / * " + , ; = [ ] (space).

laedit commented May 3, 2017

My two cents on this issue: I had a Wix build hanging on VSTS and it appear that building the wixproj on x86 platform with warnings like

warning CNDL1044: The File/@shortname attribute's value 'SYSTEM~1.DLL' is an ambiguous short name because it ends with a '~' character followed by a number. Under some circumstances, this name could resolve to more than one file or directory name and lead to unpredictable results (for example 'MICROS~1' may correspond to 'Microsoft Shared' or 'Microsoft Foo' or literally 'Micros~1').

was the cause.

The RunWixToolsOutOfProc property, setted manually to true in my project, had allow me to unblock the build on VSTS. Once the bad code gone I was able to remove the property without blocking the build again.

I tried to reproduce it on a new wixproj, but the warning doesn't block the build.
That said it appears twice on the log, I don't know if it is relevant:

d:\a\1\s\SetupProject1\Product.wxs(17): error CNDL0026: The Directory/@ShortSourceName attribute's value, 'INSTALL~1', is not a valid 8.3-compliant name. Legal names contain no more than 8 non-period characters followed by an optional period and extension of no more than 3 non-period characters. Any character except for the follow may be used: \ ? | > < : / * " + , ; = [ ] (space). [d:\a\1\s\SetupProject1\SetupProject1.wixproj]
SetupProject1\Product.wxs(17,0): Error CNDL0026: The Directory/@ShortSourceName attribute's value, 'INSTALL~1', is not a valid 8.3-compliant name. Legal names contain no more than 8 non-period characters followed by an optional period and extension of no more than 3 non-period characters. Any character except for the follow may be used: \ ? | > < : / * " + , ; = [ ] (space).

robmen added a commit to wixtoolset/wix3 that referenced this issue May 3, 2018

Fix potential deadlock with VSTS logger
Avoid deadlock with reentrant Console.WriteLine() calls and our
specialized handling of logging when tools are loaded in-proc.

Fixes wixtoolset/issues#5486

robmen added a commit to firegiant/wix3 that referenced this issue Jun 13, 2018

Fix potential deadlock with VSTS logger
Avoid deadlock with reentrant Console.WriteLine() calls and our
specialized handling of logging when tools are loaded in-proc.

Fixes wixtoolset/issues#5486
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment