Use WiX Toolset on Linux

[wixbot]

It would be nice if I could compile and run WiX in Mono. I use Linux as my main OS and when I develop for Windows I usually cross-compile then test in either Wine or in VirtualBox depending on the program. Cross-compiling is usually faster than compiling in Wine or a VM.

Originally opened by pouar

[wixbot]

Originally changed by pouar
Area set to sdk

[wixbot]

The managed code should work on Mono but WiX contains native code and relies on an implementation of msi.dll. You end up needing Wine anyway.

Resolution set to declined
Status changed from Untriaged to Resolved

[sobczyk]

Why this was closed? is it really so hard to get proper path manipulation, and use msi.dll.so from wine?

[robmen]

Because there is very little interest in keeping the WiX Toolset working on alternative platforms. If you'd like to contribute the fixes then we'd certainly look at taking them in (assuming they don't complicate the code significantly).

[sgtatham]

Hello. I've just found this closed issue. I am interested in running WiX on Linux via Mono, and in the last couple of weeks I've just developed a proof-of-concept means of doing so that doesn't need the real msi.dll, and hence doesn't need Wine either.

[Rationale: I want to build Windows software – most importantly, PuTTY – with high confidence that it isn't infected with Windows malware. I think that cross-compiling in general is a good first defence against malware infecting your build machine, on the grounds that that way, any malware that only targets one OS cannot both infect your build machine and have any interest in the software you're building there. For this reason I like to build my Windows software on Linux. But using Wine is kind of cheating, in that it's enough like Windows that Windows malware may be able to survive on it too; so I'd prefer not to even use Wine, if I can get away without it.]

I've recently managed to get a successful build of a reasonably simple MSI by running the standard WiX 3.10 binary distribution of candle.exe and light.exe using Mono. Of course on my first attempt it tried to load a native-code DLL (winterop.dll) and failed. But Mono interprets 'load a DLL' as 'load the local platform's analogue', i.e. it actually tried to load a Linux .so file. So I provided one containing stubs of the functions WiX was looking for, and Mono loaded it fine, which got me far enough to find the next problem, and so on; then I found ways to implement all the needed functionality without msi.dll.

It turned out that in order to build my one test MSI, WiX depended on the following functionality from native code. (Of course I don't doubt there are other features of WiX that would need more functionality, which I don't personally happen to be using.)

• Extracting a .cab file. My replacement libwinterop.so calls the Linux 'cabextract' utility via fork and exec. (But also, the only reason WiX wanted to extract cab files in the first place was that they were part of the WiX binary distribution; so if I were modifying upstream WiX to be more Mono-friendly, an easier approach would be to tweak the distribution so that it provides the same data in a simpler container format.)

• Creating a .cab file. There is an existing Linux program called 'lcab' that will do this, but it doesn't create compressed cab files. Fortunately, the cab file format is documented, and since one of the compression options is Deflate, it only took me about 100 lines of Python using the zlib module to write a simple compressing .cab generator, which again I call via fork/exec.

• MsiGetFileVersion. This looks to me as if it digs through the resource section of a PE executable in order to extract the four-part numeric version id and language tag. There is Linux software that understands those formats (e.g. windres) but it doesn't output a format that's any easier to parse, so I found it easier to just write my own PE-untangling code to do that.

• MsiGetFileHash. Experimenting with a test program calling the real MS version of this, the hash in question seems to be MD5 repackaged into four little-endian 32-bit words. No trouble finding Linux implementations of that.

• Creating the final MSI. The GNOME program 'msibuild' will create an MSI file if you give it a collection of binary stream files and database tables in .idt format – and, conveniently, that's exactly the set of things WiX hands to msi.dll when it wants to do this job. So this was another reasonably simple fork/exec exercise.

(There's also one annoying thing where WiX-under-Mono doesn't quite seem to do the right thing with Linux-style pathnames: when it wants to access a file from the WiX binary directory, it prefixes $PWD to the absolute path of the binary directory, so that it ends up trying to access a path along the lines of /home/user/some/build/dir/home/user/opt/wix/some.file. I initially worked around this by creating a symlink to /home in the build directory :-) and am now using LD_PRELOAD instead, but finding and fixing whatever part of WiX is generating the weird path in the first place would be preferable!)

So. Having solved all of those subproblems, I have what appears to be a perfectly good MSI generated by WiX running on Mono. But several pieces of this edifice could in principle be done by the C# side of WiX itself, which might make life easier. So I might be interested in contributing some changes, but I don't really want to go to the effort unless there's also interest in accepting them; the alternative is that I polish up my existing collection of Linux supporting .so files and just carry on using those, which I think would work OK for me but perhaps not be as widely useful.

So, would there be interest in this?

[barnson]

Added to retriage at next week's online meeting. Issues noted earlier still apply so we'd discuss that at the meeting. A WIP would be required to document the design of WiX-side changes.

[iainnicol]

Hi @sgtatham, I'm interested in the cross compilation work you're doing.

Unlike you I'm less much adverse to a Wine dependency (I'd just build in a container, or wipe ~/.wine before a build).

Would you be willing to share your code, even in its unpolished form?

[sgtatham]

@iainnicol: sure. Here's a git repo containing my current set of scripts. It lacks comments in the code, and it's been tested on exactly one project so far, but I wrote a README this morning that should at least explain how to use it.

@barnson: I see the triage label has now been removed, but what if anything did you decide? Should I set up a C# development environment, go through the CLA process, and try to send some patches, or should I just polish up the repo I've got right now?

[barnson]

@sgtatham : I removed the triage label because we forgot to check for it. 🥇 So I removed the labels and they'll show up during "normal" triage at our next meeting. A few comments/ideas:

• Take a look at the WIPS to see what we're looking for on details with big changes.
• This is something that would have to happen in wix4, not wix3, which is essentially done. WiX v4 already drops the use of embedded cabs in object/library files.
• Consider a post-processor that worked off .wixpdb files, which have all the symbol info from the compiler.

[sgtatham]

Hmmm. I didn't want to get into writing a lengthy document specifying a complete proposal until I'd got at least a vague idea of what the proposal ought to be.

Off the top of my head, several high-level possibilities spring to mind for adapting WiX to be Linux-friendly:

1. Essentially what I've got already, modulo some polishing: a collection of Linux native-code .so files that substitute for winterop.dll and friends using the various strategies I described earlier, so that the C# parts of WiX hardly need to change at all (except that I'd still like to fix that pathname-construction bug I mentioned above). Pro: I already have this more or less working.
2. Implement the same strategies on the C# side, so that WiX when run under Mono would directly construct cab files, figure out file versions by decoding the PE resource section, and spawn a subprocess for msibuild, without the need for Linux native-code .so libraries at all. Pro: this would allow the binary distribution of WiX to run under Mono successfully, without any need to add a set of Linux shared library files to its build system.
3. Instead of building Linux support into WiX itself, arrange for light.exe to have a mode in which it outputs a set of .idt files and a specification of what cabs should be constructed to go with them. Then the Linux tool flow would involve running light in that mode and then running a followup script over that output to do the actual cab-building + msibuild step. (But there would still need to be some substitute for MsiGetFileVersion / MsiGetFileHash earlier in the procedure.)

I could write up any of those as a more detailed proposal, if you need me to do that before even deciding whether it's worth me going through the CLA process, but I'd rather not write up all three! :-)

[barnson]

I recommend posting those options on the wix-dev mailing list to see what the assembled masses say.

[iainnicol]

The aim of this issue is to run WiX on Linux, which is useful for Free and open source projects. Inspired by @sgtatham's work, I've however taken a different approach.

I've ported WiX to .NET Core 2.0. .NET Core is cross platform and open source. Whilst Mono is impressive, Microsoft is putting resources into .NET Core that Mono never had. Concretely, msggen or xsdgen generates broken code on Mono but works on .NET Core. Another example of .NET Core's better maturity is that I can also compile WiX on Linux, as opposed to just running WiX.

I have the following working: tools/src; src/dtf except the main SfxCA bit; src/tools/{candle,light,lit,winterop}; src/libs/{dutil,wcautil}; and the pièce de résistance, src/ext/UIExtension.

The work wasn't too bad. Of course, there were a mixture of changes required:

• Some are simple and worth including upstream. For example filename casing, backslash versus forward slashes, different defines
• Some changes are simple but probably shouldn't be moved upstream. For example changing the framework in the .csproj to .NET Core would force WiX off the traditional .NET Framework
• Some changes were downright ugly and shouldn't be moved upstream. For example, .NET Core doesn't support .vcxproj (C++ projects); I used Makefiles instead of figuring out how to adapt msbuild.

Obviously WiX requires still requires cab support and msi.dll. My .NET Core builds of WiX run well under Wine. For some semblance of isolation you can run Wine with a unique WINEPREFIX directory. It would probably also be straightforward for me to port @sgtatham native libs, if Wine is not desirable.

I'll make my code available in the next couple days. However it's definitely not at the stage of a WIP or pull request.

[sgtatham]

That sounds cool! I did try to build WiX using the Mono C# compiler, but didn't have much luck, either by using xbuild with the project files or by just trying to construct an alternative build script. Being able to compile on Linux will certainly make it easier for me to do whatever it is I end up doing, even if it's only fixing that pathname bug I keep alluding to.

(I've not looked into .NET Core, but presumably it includes a CLR runtime that runs on Linux, i.e. a replacement for the Mono runtime that I've been using? If so then I should probably see what that does about native-library loading, and whether it's compatible with the way Mono does the same thing...)

[robmen]

There is a discussion about this on wix-devs mailing list.

[iainnicol]

My rough 'n' ready code is available at https://gitlab.com/brainsick/wix-cross. See README.linux.txt. (Yes, .NET Core's CoreCLR supports PInvoking .so files on Linux). @sgtatham, please feel free to mail me any questions directly (email address in the git log).

Thanks, Rob, I'll follow up the rest of the conversation on the mailing list.

[develar]

Currently, Wine is not a solution, because LZX compression is not supported. Only mszip (if you set high, wine implementation simply ignores compression type and none is used). So, in my case it is 40 MB vs 56 MB. Why ability to build MSI on Linux/macOS is important?

Because

1. Windows is not good and popular OS for developers of Electron applications.
2. Windows CI is not cheap.
3. Ability to use the same CI. e.g. you can build NSIS on Linux (Travis CI). No need to setup AppVeyor.

electron-builder now uses WiX 4.x to produce MSI. Wine and special wine home (with installed Net Framework 4.6.2, Mono is not supported by WiX — produced MSI is corrupted) is prebundled for macOS. For Linux we provide special Docker image to build MSI on Linux.

As workaround of compression issue, I am going to build uncompressed cab on POSIX system and after that use Msidb.exe to extract cab and compress it manually using makecab.

I hope in the feature it will be possible to use WiX on macOS/Linux using only Mono.

[HumanG33k]

Hello some news about this feature ? It can be great for have easiest GNU/Ilinux base CI.

[BMurri]

At this point, WiX v4 has finally received some great momentum. It runs in .Net Core.

I don't recall if the MSFT-provided DLLs we called in v3 are still being used (in which case Linux replacements need to be identified/sourced) or if that code has been moved into managed code. The ability to use the toolset as part of your toolchain is a design goal, but runtime native code on non-msft platforms is not an expertise of the core WiX team, so where that's needed we need the commitment of one (or more) of you to help us test and maintain it.

We may want to consider changing the title of this issue, since Mono is not the same as .Net Core, yet the same platforms are supported. Please help us here. Most of the interest seems to be for Linux. What about MacOS? Is anyone building Windows software on any other platforms besides these three?

[sgtatham]

At this point, WiX v4 has finally received some great momentum. It runs in .Net Core.

Great!

Does that mean it's conveniently possible to build it on Linux right now, or just that it's possible in principle to write some build scripts that will enable it to be built? I'm interested in having a go at the next steps.

but runtime native code on non-msft platforms is not an expertise of the core WiX team, so where that's needed we need the commitment of one (or more) of you to help us test and maintain it.

It may not actually be necessary to link in runtime native libraries on Linux. That's how I do it in my current WiX 3 bodge, because WiX 3 was already set up to look for a native-code DLL, so the simplest way to get it to do what I want was to provide one. But if WiX 4 can be recompiled directly on Linux, then a lot of the tasks I list above could just as well be done by C# code as by C. There's no reason I couldn't port my simple CAB-builder and my MsiGetFileVersion emulation into C#, for example.

Actually building an output MSI file is trickier, but my current bodge library doesn't do the job itself: it calls out to GNOME msitools. And that too could be done directly from C# without having to put a native-code .so in between.

@develar: if you're interested in LZX, I happen to have some code in another of my projects that knows how to do LZX compression (my docs generator Halibut can write CHM). I've never tried making a CAB using that code, but perhaps I should give it a go and see if it works...

[robmen]

@sgtatham In WiX v4 we've designed for the possibility of natively supporting build on other platforms but all of the core contributors to the WiX Toolset use Windows. That means we will need the help of developers on the other platforms to contribute to the gaps_and point out "Window-isms" that we are blind to (for example, I expect there are a great many x-plat path handling mistakes because we only work on a case-insensitive filesystem using \ as the separator).

What needs to be done:

1. Core.Native - this project is intended to isolate all of the platform-specific interaction. Ideally, this is the only project you have to interact with to add a new supported build platform. (note: Ignore winterop it is legacy and will be deleted).

2. P/Invoke - I'm sure there are some P/Invoke still spread about that need to be migrated to Core.Native. It should be straight forward to move those to Core.Native as we find them.

3. "Windows-isms" - as mentioned above, I'm sure there are many of these that we are blind to spread about the codebase. I expect they are generally shallow and easy to address but reserve the right to be surprised. We'll need help finding those that are impactful to the build.

If you are interested, take a look at Core.Native and jump on the wix-devs mailing list to discuss next steps.

[sgtatham]

@robmen thanks. I had a quick look through Core.Native, and the first thing that leaped out at me was that the code that actually makes an MSI doesn't seem to be in there. As far as I can see, that's still done by DllImport from msi.dll in Core/src/WixToolset.Core.WindowsInstaller/Msi/MsiInterop.cs. But msi.dll is part of Windows and not available on any other platform, so that is certainly another platform-dependent piece of code that will need replacing on Linux.

(In my WiX 3 bodge, I have Linux .so libraries that take the place of WiX's winterop.dll and of Windows's msi.dll.)

But that's just a first glance. I'll look through it more carefully and also try to familiarise myself with .Net Core (on which I currently don't even know how to build things).

[robmen]

still done by DllImport from msi.dll in Core/src/WixToolset.Core.WindowsInstaller/Msi/MsiInterop.cs

Will get those PInvokes moved to Core.Native. Of course, we didn't reimplement msi.dll. It was already in Windows. 😄

take the place of WiX's winterop.dll

I think you'll find wixnative.exe very similar to winterop.dll.

[sgtatham]

@robmen Right. I've managed to get the wix binary itself to compile on Linux with a few minor build adjustments. Of course without the native code support the only thing I've so far made it do is print its help message (or rather, the TODO saying a help message would be nice :-) but getting it to compile at all is a start.

Would you like patches against the Core and Tools repos?

[robmen]

Sure. PRs there would be a good first step.

PS: completing command-line output is tracked in #6211

[sgtatham]

My current patches are in wixtoolset/Core#165 and wixtoolset/Tools#44. But the CI results suggest that I might have been overoptimistic about saying that my changes to the CompileCoreTestExtensionWixlib command execution wouldn't break on Windows. Guess I need to figure out how to make those changes conditional now...

[ewheelerinc]

We have a software agent that gets deployed by GPO and we are considering using WiX to build customized MSI's that embed the customer logins and API keys in a single convenient download to save the customer from messing with MST deployments. (We already use WiX to build static non-customized MSIs.) Our backend software and build environment is all Linux (well, except for the static WiX build) so it would be nice to build the MSI on Linux and dynamically provide the end-user with a pre-customized MSI. The work on this github is encouraging, maybe native Linux MSI building under WiX will be available soon!

Here are some additional resources/references if it helps the project along:

Here is a non-wix package for Linux that uses Wine's re-implementation of the MSI installer for low-level reading/modifying/creating of MSI packages:

• https://wiki.gnome.org/msitools

These are articles of people using WiX with Wine. I know the goal is to run native in Linux without Wine, but here are some options in case it helps:

• https://bozaro.ru/2016/04/19/wix-wine-wixsharp-en/
• https://wiki.gnome.org/Projects/GTK/Win32/WiX

Cheers,

-Eric

[robmen]

@ewheelerinc you will probably want to look into contributing to get this over the finish line. Not sure anyone is actively working on it at this time.

[cakiremre]

We have a software agent that gets deployed by GPO and we are considering using WiX to build customized MSI's that embed the customer logins and API keys in a single convenient download to save the customer from messing with MST deployments. (We already use WiX to build static non-customized MSIs.) Our backend software and build environment is all Linux (well, except for the static WiX build) so it would be nice to build the MSI on Linux and dynamically provide the end-user with a pre-customized MSI. The work on this github is encouraging, maybe native Linux MSI building under WiX will be available soon!

Here are some additional resources/references if it helps the project along:

Here is a non-wix package for Linux that uses Wine's re-implementation of the MSI installer for low-level reading/modifying/creating of MSI packages:

• https://wiki.gnome.org/msitools

These are articles of people using WiX with Wine. I know the goal is to run native in Linux without Wine, but here are some options in case it helps:

• https://bozaro.ru/2016/04/19/wix-wine-wixsharp-en/
• https://wiki.gnome.org/Projects/GTK/Win32/WiX

Cheers,

-Eric

@ewheelerinc Hello Eric; how do you generate/manage custom msis for your clients; because we have same issue, and it started getting out of our hands. If you could direct us a working solution; I will be glad. Thanks

[ewheelerinc]

@cakiremre, We are still working out the details, but here is the preliminary procedure that (we think) will work:

This uses msitools from here: https://wiki.gnome.org/msitools

First build your MSI with WiX (or really any MSI builder) on Windows. We tried for months to get WiX to build under Linux with the latest WINE from git, but there are subtle bugs that prevent GPO deployment (at least EmbedCab fails, not sure what else).

Here is a solution using msibuild and a file "Property-template.idt" to set the property in an existing MSI:

• cp app-template.msi app_.msi
• cat Property-template.idt | sed s/CUSTOMER_CHANGEME/cust3/ | sed s/PIN_CHANGEME/698758/ | msibuild app_cust3.msi -i /proc/self/fd/0

We took the original app.msi file and opened it in Orca. In Orca the properties "CUSTOMER" and "PIN" where added with the values PIN_CHANGEME and CUSTOMER_CHANGEME and saved it as app-template.msi. Be sure add these default values to the WiX template so WiX rebuilds will come with the _CHANGEME defaults so we don't need to edit with Orca. Then you can write a C# assembly (or some other mechanism?) to read the parameters at install time, maybe like this (but I'm not the C# programmer, so could be wrong here):

namespace CustomAction1
{
    public class CustomActions
    {

        [CustomAction]
        public static ActionResult GetArgumentValues(Session session)
        {
           string CUSTOMER = session["CUSTOMER"];
           string PIN = session["PIN"];
           // Now, do something useful at install time with PIN and CUSTOMER
[...]

Using msidump we dumped the .idt files to get a copy of the Parameters.idt tab-delimited table that looks like this:

Property    Value
s72    l0
Property    Property
Manufacturer    app
ProductCode    {...}
ProductLanguage    1033
ProductName    app_client
ProductVersion    1.0.0.0
UpgradeCode    {...}
CUSTOMER    CUSTOMER_CHANGEME
PIN    PIN_CHANGEME

This file was saved as Property-template.idt so sed can replace the fields. Then msibuild can replace the Property.idt content with the modified content. The example above is perhaps more complicated than it needs to be, msibuild is called as msibuild file.msi -i file.idt; we just used /proc/self/fd/0 to read it from stdin. In addition, the Property.idt file should be re-exported each time you do a WiX build in case something changes.

Here is the whole process:

    - mkdir idt-temp
    - msidump -d idt-temp/ app-template.msi
    - cp app-template.msi app_<customer>.msi
    - cat idt-temp/Property.idt | sed s/CUSTOMER_CHANGEME/cust3/ | sed s/PIN_CHANGEME/12345abcdef/ > Property_cust3.idt
    - msibuild app_cust3.msi -i Property_cust3.idt
    - rm Property_cust3.idt
    - # Now you can install app_cust3.msi (I hope!)

[socketpair]

https://git.tartarus.org/?p=simon/wix-on-linux.git;a=summary

Actually works. But seems abandoned.

[sgtatham]

That isn't abandoned – I just haven't had to fix it in a while, because it's been working reliably!

That's the lash-up I use for running unmodified WiX 3 on Linux, by running the WiX CLR images using mono and substituting my own code for the native libraries WiX tries to load. It's used to generate all the MSIs I publish, including all the installers for PuTTY (release and snapshot).

But it's entirely a set of downstream bodges, which I was hoping to retire if built-in support appeared in WiX itself.

[sgtatham]

... but now I go back and look at the state of README.txt in that repo, I understand why you thought it was abandoned! I've updated it so that it doesn't say "last tested on Ubuntu 14.04" any more :-)

[socketpair]

@sgtatham The only reason: the last commit date.

If the code is perfect and does not need to be changed – please add to the main readme something (using h1 font) like:

"The project is alive in spite of the last commit date. Please contact XXX if you have any questions.".

Another way - is to test on recent distros time-to-time and update readme. So last commit date will be bumped.

[to-marss]

Can we expect some fix on this? I would like to know if any workaround is available for this to work in Linux build pipeline. If not, a reference to docker to work with WiX V4 .NET 6.0 runtime with Mono/Wine would help better. Any help would be much appreciated..
@barnson @robmen

[opuech]

Does someone try to use these images (for Wix 3) ?

• https://hub.docker.com/r/felfert/wix (dockerfile : https://hub.docker.com/r/felfert/wix/dockerfile)
• https://github.com/dactivllc/docker-wix

Any progress to be able to run WiX 4 on Linux ?

[TedMANNERs]

With .NET 8 it should now be possible to build applications that require Win32 resources on Linux. See dotnet/core#8439. Might this be of any use for building WiX 4 on Linux as well? I suppose the Windows Installer relies on Win32 resources.

[jkroepke]

I had no success with .NET 8 on linux, however I figure out that wix partially works with WINE.

https://hub.docker.com/r/jkroepke/wixtoolset - dotNet 8 with wix5

There only Issue I figure out was that wine does only support compressionLevel=mszip and none. LZX compression is not supported.

[BMurri]

With .NET 8 it should now be possible to build applications that require Win32 resources on Linux. See dotnet/core#8439. Might this be of any use for building WiX 4 on Linux as well? I suppose the Windows Installer relies on Win32 resources.

Win32 resources are resources (like icons, strings, etc.) embedded in Win32 binaries. Those aren't the primary remaining issue. That issue is the Windows Installer (and one or more adjacent) APIs we use to build/parse Windows Installer filetypes (e.g. MSI, MSP).

The secondary remaining issues are the APIs used when harvesting (e.g. DCOM).

Implementation steps are here.