Linux build fixes

[sgtatham]

This series of patches enables me to build the Core library using .Net Core on Linux (specifically Ubuntu 18.04). See wixtoolset/issues#4381.

[rseanhall]

Thanks for starting to work on this.

I updated Nerdbank.GitVersioning and should have addressed the cross platform issues with CompileCoreTextExtensionWixlib in #166.

For .NET Core, compiling to an executable requires that you target a specific platform. We don't want to do that. So if you want to run a local build, then run dotnet wix.dll and then add the options after that. If you look at my changes for CompileCoreTextExtensionWixlib, you'll see that's how I'm running it as a .NET Core app.

[sgtatham]

I've reworked the target framework changes as you suggested in wixtoolset/Tools#44.

Your update to GitVersioning is working fine on Ubuntu, and so is the revised version of CompileCoreTextExtensionWixlib, so I've dropped both my patches for those.

That just leaves the change to command-line syntax, which I think is probably unavoidable for a port to any Unixlike VFS in which non-switch arguments are very likely to be absolute paths starting with a /.

[rseanhall]

Thanks for helping us start to be better at cross-platform. We're not going to take the change for conditioning the TargetFrameworks off of the OS. It's not worth the added complexity for us since we'll always do official builds on Windows. Also, all of the cross-platform work is intended be in the Core.Native repo. We are willing to be much more flexible about build configurations in that repo. FYI, we are hoping we can do the official build there on Windows with Visual Studio's C++ cross-platform support.

[sgtatham]

Thanks for getting back to this! I've been busy with other things as well, naturally, but I'm still interested in pushing Linux support in Wix 4 forward.

We're not going to take the change for conditioning the TargetFrameworks off of the OS. It's not worth the added complexity for us since we'll always do official builds on Windows.

Ah yes, now I remind myself, I saw that comment on one of the other PRs. So the idea is that I can do development on the C# parts by running debug builds in the Linux version of .Net Core, but then your official release builds happen on Windows and produce .exe files that Linux users can run via Mono as before, providing a Linux version of the native-code executable helper?

That works for me as a general strategy (although I see that the invocations of msi.dll in src/WixToolset.Core.WindowsInstaller/Msi/MsiInterop.cs are still there, and haven't been moved into Core.Native).

I'll resubmit this PR with only the remaining patch for newline fixes.

[rseanhall]

We don't have any plans to remove the dll dependency on msi.dll. That would be a massive undertaking. There are many more invocations of msi.dll in Dtf. I think the typical way of providing msi.dll on Linux is to use Wine. I guess if someone really wanted to reimplement msi.dll in Core.Native for Linux, they could do that.

but then your official release builds happen on Windows and produce .exe files that Linux users can run via Mono as before, providing a Linux version of the native-code executable helper?

Yes, the official WiX build would include a Linux version of the native-code executable helper. It would run on .NET Core though, not Mono. Although, there is .NET consolidation happening in .NET 6 so it may be able to run on Mono at that point.

[sgtatham]

We don't have any plans to remove the dll dependency on msi.dll.

Oh! That's a change of policy from @robmen's previous comment. I thought the plan of record was that OS dependencies would be moved into a .exe helper subprogram, and that those direct imports from msi.dll were going to be (at some point, as other priorities permitted) replaced with wrappers that called out to that subprogram.

I guess the plan might have changed somewhere I wasn't watching?

I think the typical way of providing msi.dll on Linux is to use Wine.

Not as far as I know: Wine doesn't provide any reimplementation of msi.dll that I know of. Where did you hear that?

It's technically possible to copy the real msi.dll from a Windows machine into a Wine installation on Linux and use it there, but the legality of doing that is at least questionable, and in any case, that doesn't meet a lot of the underlying goals of wanting to run WiX on Linux in the first place: it doesn't constitute building using only open-source tools, and it doesn't constitute building without a Windows environment for my malware-avoidance purposes.

My current bodge for running WiX on Linux without Wine is to provide a .so file that provides the msi.dll API, or rather, the subset of it that my particular use of WiX needs, and to implement the API calls that build MSI files by calling out to the rather nifty GNOME msitools utilities.

I suppose I can carry on doing that if necessary. It works on Mono because Mono interprets the .NET DLL load operation by trying to load a .so of the same name; I think the .NET Core runtime does the same, so probably I can stick to the existing plan. It'll just look like a horrid bodge when the instructions get written up for people other than me to run!

[sgtatham]

But thanks for merging (what's left of) this PR, anyway!

[rseanhall]

Not as far as I know: Wine doesn't provide any reimplementation of msi.dll that I know of. Where did you hear that?

Anecdotally, but I found this that seems to support my statement: https://wiki.gnome.org/Projects/GTK/Win32/WiX.

I guess the plan might have changed somewhere I wasn't watching?

Maybe Rob still plans on moving the invocations from Core, but I don't think he was ever planning on touching Dtf or moving the msi.dll invocations into the executable.

It works on Mono because Mono interprets the .NET DLL load operation by trying to load a .so of the same name; I think the .NET Core runtime does the same, so probably I can stick to the existing plan.

Yes, .NET Core does the same.

It'll just look like a horrid bodge when the instructions get written up for people other than me to run!

.NET Core should have taken care of everything. All we should have to do is install WiX as a dotnet tool, and it will just work. The hard part is getting someone to contribute the Linux implementations in Core.Native.

Edit: I had forgotten that most of your implementation is to fork/exec out to other utilities. I see what you mean now. Ideally we could get it to where they just have to install the utilities you depend on.

[sgtatham]

Hmm. Apparently you're right, and Wine has had a msi.dll.so all along, in the dlls/msi subdir of the Wine git repo, including a full implementation of SQL for the database update commands.

In that case, I wonder why I even tried using the official msi.dll from a Windows machine back when I was trying to run unmodified WiX 3 under Wine. Perhaps at the time Wine's version hadn't got good enough to do everything WiX needed.

(Naturally, when I was actually trying this, I hadn't anticipated the need to take careful enough notes to tell you what went wrong 4 years later! But I wish I had, because then I'd know the answer too.)

The hard part is getting someone to contribute the Linux implementations in Core.Native.

That's the part I'm working on (in between distractions). I've already got implementations of all the CAB-file munging facilities currently performed by wixnative.exe, some by calling out to other Linux tools and some by writing my own code (for making LZX- and gzip-compressed CAB files). The current problem is that a third of it is in Python, a third in C, and a third still waiting to be migrated across from my previous DLL-shaped bodge for working with WiX 3. But I think every piece of needed functionality is working somewhere in my untidy source directory.

As I just mentioned, I was expecting the wixnative.exe API to be extended to include MSI-building facilities, in which case I'd have added implementations of those in the same tool. But now it sounds as if that's not happening, so I'm going to have to either provide a Linux libmsi.so of my own to go with my Linux wixnative.exe, or – now that I know it's there – find a way to turn Wine's msi.dll.so source code into something that will do the same job. (But I expect Wine's .so won't work unmodified, because of function call ABI mismatch and/or too-tight integration with the rest of Wine.)

[ewheelerinc]

Also of note, with respect to finding a suitable Linux replacement for msi.dll, is this project:

• https://wiki.gnome.org/msitools
  "msitools is a set of programs to inspect and build Windows Installer (.MSI) files. It is based on libmsi, a portable library to read and write .MSI files. libmsi in turn is a port of (and a subset of) Wine's implementation of the Windows Installer. "

[robmen]

If it becomes really onerous to make the PInvoke's into msi.dll work, we can look at porting more to wixnative.exe. The problem is that will likely make Windows-based WiX slower because we'd be going out of proc.

Another alternative would be to convert wixnative.exe into wixnative.dll and PInvoke everything. There are some advantages to keeping wixnative.exe in the picture (there is some tooling out there that can spread separate processes across machines and wixnative.exe is definitely the slowest part of the MSI build when invoked). Or we could go with a hybrid solution.

Basically, we've not had a lot of direct input on how to best shape things to make all the x-plat stuff work so I'm waiting to see what becomes real. :)