OutputType does not set Nt headers correctly on Linux

[NicolasDorier]

Publishing my app on mcr.microsoft.com/dotnet/core/sdk:3.0.101-buster for win-x64, it seems that the OutputType is completely ignored from the resulting executable. (not using PublishSingleFile)

The NT Headers value for subsystem is console instead of Windows GUI.
Workaround: Binary modification of the executable.

[NicolasDorier]

My workaround is here for those interested:

https://github.com/btcpayserver/BTCPayServer.Vault/blob/bc7e80aff45fe703e8164ded5d02e6015ba5b7b7/Build/win-x64/Dockerfile#L41-L48

[wli3]

@swaroop-sridhar @nguerrera it is possible/make sense to have a cross platform impl to set Windows headers?

[swaroop-sridhar]

@wli3 If the target generated is a PE Image, the GUI/CUI bit should be set correctly, regardless of the host OS.

I think the HostModel can handle this when generating PE files, if the information passed in from the SDK. https://github.com/dotnet/runtime/blob/4f9ae42d861fcb4be2fcd5d3d55d5f227d30e723/src/installer/managed/Microsoft.NET.HostModel/AppHost/HostWriter.cs#L60-L69

[wli3]

@swaroop-sridhar so we can remove the check for host machine OS?

[swaroop-sridhar]

@wli3 The resource updater makes actual Win32 calls, so it only supports Windows hosts for now.

But we should be able to set the GUI/CUI bit correctly regardless of whether we can update the resources.

[swaroop-sridhar]

I expected that this bit should be set properly -- because the SDK's computation of the WindowsGraphicalUserInterface property or the HostModel's setting the GUI/CUI bit is independent of the host (but rather depends on the RID).

So, we need to investigate what went wrong here.

[NicolasDorier]

Probably fixed by dotnet/runtime#89303 in 8.0, untested though.

[NicolasDorier]

Fixed in 8.0