Problems when installing new version of application from MSI

[cainesi]

Describe the bug
When installing a new version of already installed application from MSI installer (so that the previously installed version is automatically uninstalled before the new installation), the executable files are omitted.

Problem was introduced by 99b3ad9.

A clear and concise description of what the bug is.

To Reproduce

1. Create MSI installer with cx_freeze.
2. Install from MSI
3. Create second MSI with the same executable and upgrade code.
4. Install from second MSI

Expected behavior
Should uninstall the first version and fully install the second. Instead, the executable from the second installer is omitted.

Additional context
The problem appears to be caused by putting the executable in its own component, with the executable file as the keypath. I think what happens on the second install is that: it first scans for the existing files and determines that the component for the executable is already there so does not need to be installed, then removes the entire existing installation, then installs the new new version except the executable component that it determined was unnecessary.

Problem goes away if the following code in windist.py is disabled.

if sep_comp is not None:
   restore_component = dir.component
   dir.start_component(
      component=sep_comp,
      flags=0,
      feature=f,
      keyfile=file,
   )
   dir.add_file(file)
   dir.component = restore_component

@Cimbali -- If you are reading, was there a need to put the executables into separate components? I am not familiar enough with the extension registration system to know if this was somehow required.

As a "quick and dirty" fix, maybe we could only put the executables into separate components if extensions are being registered?

[Cimbali]

Yes it was required. To register an executable as the one to be used for a given file extension it needs to be a component.

I think I need to do some reading and look into why this happens, it's really weird and unexpected tbh, and I didn’t notice it though I provide software updates with cx_Freeze MSIs that have upgrade codes. Can you maybe provide a minimal example ?

As a "quick and dirty" fix, maybe we could only put the executables into separate components if extensions are being registered?

I think the best quick and dirty workaround would be to generate unique component names. When you register extensions you still want to upgrade executables.

[Cimbali]

It would be interesting to see what your 2 MSIs contain, especially with regards to ProductCode and ProductVersion.

[cainesi]

In the particular example I am looking at, I have 2 installers with the same ProductVersion and UpgradeCode, but different ProductCodes, and they exhibit the problem I described above.

I found this discussion online where people seemed to be having similar problems (but a long long time ago): https://community.flexera.com/t5/InstallShield-Forum/Major-Upgrade-removing-files-Not-installing-required-Files/m-p/33801#M15138

However, I just tried to make a stripped-down example of the problem (based on samples/simple) and the issue disappeared. So there must be something else I'm doing that is also needed to trigger the issue. I'll try to isolate the exact requirements to reproduce this before I waste any more of your time.

[cainesi]

@marcelotduarte Oops. Sorry I missed that.

I am not having any success isolating this into a simple example. The problem seems quite mysterious to me.

[cainesi]

I think I have a fairly simple example that illustrates the problem. If I install "helloN" and then install "helloM" I have the missing executable problem (but not for "helloM" followed by "helloN"). "helloM" can be repaired to add the missing exeuctable. I would be interested to know if other people see the same result?

The installers were created by freezing the same python script (which just contained a print statement) with the same setup.py script (both attached). The only difference is that they were made in different virtual environments. The only relevant difference I can see is that the "helloM" venv had setuptools 50.3.2 and wheel 0.36.0, but the "helloN" venv has versions 54.2.0/0.36.2. Both have the latest cx_freeze.

When I look at what is installed by the two msi files, the hello.exe file from "helloN" has more information shown on the "Details" sheet of the "Properties" display. I suspect that extra file information is what is causing the issue. I don't suppose anyone knows where along the process that information gets added to the executable file?

helloN - simp2 - blineenv - cx later.msi.zip
helloM - simp2 - cxenv - cx later.msi.zip
code.zip

[Cimbali]

First off, I can reproduce the issue with the MSIs you provided (on Win10) with the same considerations: the bug only appears in one direction (N > M but not M > N) and repairing (i.e. N > M > M) fixes the issue.

Secondly, there’s apparently only 3 tables in the MSI that differ, all are UUIDS:

• Component which shows just different unique ids,
• Property.ProductCode
  • this seems OK as the page says « This ID must vary for different versions and languages. »
• _SummaryInformation’s Revision number
  • this also says « Nonidentical .msi files should not have the same package code. »

So I don’t think we’re generating something incorrectly, but that something goes wrong in the install sequence similarly to the link you found. I’m thinking:

• some arbitrary metric shows versions M > N (this could be comparing Revision GUIDs or the file creation date, N is 1 minute younger than M).
• installer computes & does removal of old binary
• installer only installs new binary if “newer”

Maybe we can find a flag to always force the install of the executable. At the moment I’m going through installer log files to try and confirm this theory. If this is true, then it would only happen with non-incrementing version numbers.

Anyway here’s the diff of the tables:

diff -u -r helloM/Component.csv helloN/Component.csv
--- helloM/Component.csv	2021-09-20 12:19:43.642840523 +0200
+++ helloN/Component.csv	2021-09-20 12:19:44.630846229 +0200
@@ -1,26 +1,26 @@
 Component	ComponentId	Directory_	Attributes	Condition	KeyPath
 s72	S38	s72	i2	S255	S72
 Component	Component
-TARGETDIR	{E340A509-2581-40D3-9958-77DCF91FF1B1}	TARGETDIR	256		
+TARGETDIR	{21F53009-5355-4D42-A3C4-4B368276168C}	TARGETDIR	256		
-_cx_executable0__Executable_script_hello.py_	{22AC3684-15F6-49A3-94E5-0398F5B99C14}	TARGETDIR	256		hello.exe
+_cx_executable0__Executable_script_hello.py_	{1EF58639-599D-4E20-B276-C1C05279CC31}	TARGETDIR	256		hello.exe
-lib	{27895FB0-B2AA-4135-B8CD-71934F6F36F4}	lib	256		
+lib	{4C0E0573-E65E-46B0-937D-41BC2CA639D6}	lib	256		
-asyncio	{3A9C64E8-B33E-4316-8B48-91A37E1ED8C0}	asyncio	256		
+asyncio	{27860E25-9DA5-41AD-9F3C-EAC8D091A2C5}	asyncio	256		
-collections	{373A02F1-71D7-4015-A061-E1B79721C8E3}	collections	256		
+collections	{917F4DD2-25A4-4DCF-8A79-DFC7055C7244}	collections	256		
-concurrent	{2A8C6B3D-E175-4D5A-AE2A-7BE5C501D561}	concurrent	256		
+concurrent	{AB2D2280-9765-4985-8F3C-2B483B1D6988}	concurrent	256		
-ctypes	{846DFA4B-6884-46B0-BC5B-79BF22A24AA5}	ctypes	256		
+ctypes	{5811EC95-8965-4A53-B17F-FFE1C7D5C31D}	ctypes	256		
-email	{16654BBC-9254-4D32-98EE-ADE34F8A08B1}	email	256		
+email	{5B0502A6-F027-4F0D-99A6-66A0DED7035C}	email	256		
-encodings	{14C9F7EC-E727-4E78-A9D6-9C17DF14D13F}	encodings	256		
+encodings	{CC390850-FBBC-4579-9268-20C03569F447}	encodings	256		
-html	{BB197B3E-6E65-4751-A3A7-8C23011D6B87}	html	256		
+html	{9606ADDC-257F-4B96-8594-35BCC809F04C}	html	256		
-http	{4528F495-11C5-432D-88EF-EB8E5A4BAAAC}	http	256		
+http	{F3F48B10-FFD9-4B74-8805-5CE3C6DC65E6}	http	256		
-importlib	{1B5E31DC-AF8E-4920-BE50-7EC96A9E096A}	importlib	256		
+importlib	{1E47D18E-16F9-462E-8A52-F0472CA17998}	importlib	256		
-logging	{496E227B-2862-4C37-AB0C-F3955B69E1FC}	logging	256		
+logging	{7957F735-01E1-4E49-A0A1-7B65281891F1}	logging	256		
-multiprocessing	{7058D5DF-099B-4685-9856-51ED0B940D5D}	multiprocessing	256		
+multiprocessing	{F651CC0A-4E9D-46F4-B736-81BA2783C90A}	multiprocessing	256		
-pydoc_data	{68EB0C6C-639A-4EAA-9936-9B4C9F568F62}	pydoc_data	256		
+pydoc_data	{FB418F90-1190-4401-B74B-8F743FFA1A44}	pydoc_data	256		
-unittest	{62AA1FD1-2B0A-4F81-B0B3-4C679C54BC95}	unittest	256		
+unittest	{6F868F2E-BB5E-466F-A197-B72E6FC49972}	unittest	256		
-urllib	{77B9EF02-E126-4698-B6A5-A75B412301F2}	urllib	256		
+urllib	{47DF39CA-6DEC-45F6-9EEB-C31515DC1C85}	urllib	256		
-xml	{454D8345-043D-437C-96E4-B6F889F00891}	xml	256		
+xml	{ED368B78-FDF8-43FB-8A4E-535D8D718299}	xml	256		
-xmlrpc	{A415A344-0DD6-4BE4-B4E4-D5F7FB8F5AD2}	xmlrpc	256		
+xmlrpc	{F1FBFAC6-0F6B-42B1-9E10-E38421159F22}	xmlrpc	256		
-parsers	{8FBB10C3-ACEA-4FF8-AE36-C289CA902CBF}	parsers	256		
+parsers	{C66C840B-697F-4010-ADF7-BC2CE9BA664C}	parsers	256		
-dummy	{F316A46C-8024-4CE9-94C6-3793B5B9EF24}	dummy	256		
+dummy	{3E722735-B9E7-4D49-BADD-05C4E3E75F15}	dummy	256		
-macholib	{83D1F7AC-485E-4F92-96B8-F0EFD3AF88C8}	macholib	256		
+macholib	{4F1DFBB8-7167-496D-96EA-4796D2EAAE40}	macholib	256		
-futures	{E6B3F034-F3A8-4C8D-9002-64F7059445C9}	futures	256		
+futures	{5BF7D4EF-64D3-4882-8A86-1DC3B1B9AC75}	futures	256		
diff -u -r helloM/File.csv helloN/File.csv
--- helloM/File.csv	2021-09-20 12:19:43.598840269 +0200
+++ helloN/File.csv	2021-09-20 12:19:44.578845928 +0200
@@ -1,12 +1,12 @@
 File	Component_	FileName	FileSize	Version	Language	Attributes	Sequence
 s72	s72	l255	i4	S72	S20	I2	i2
 File	File
-hello.exe	_cx_executable0__Executable_script_hello.py_	HELLO.EXE|hello.exe	117760			512	1
+hello.exe	_cx_executable0__Executable_script_hello.py_	HELLO.EXE|hello.exe	118272			512	1
 python3.dll	TARGETDIR	PYTHON3.DLL|python3.dll	58952			512	2
 python38.dll	TARGETDIR	PYTHON38.DLL|python38.dll	4209224			512	3
 libcrypto_1_1.dll	lib	LIBCRY~1.DLL|libcrypto-1_1.dll	3399200			512	4
 libffi_7.dll	lib	LIBFFI-7.DLL|libffi-7.dll	32792			512	5
-library.zip	lib	LIBRARY.ZIP|library.zip	956781			512	6
+library.zip	lib	LIBRARY.ZIP|library.zip	959439			512	6
 libssl_1_1.dll	lib	LIBSSL~1.DLL|libssl-1_1.dll	689184			512	7
 pyexpat.pyd	lib	PYEXPAT.PYD|pyexpat.pyd	189000			512	8
 select.pyd	lib	SELECT.PYD|select.pyd	26696			512	9
diff -u -r helloM/Property.csv helloN/Property.csv
--- helloM/Property.csv	2021-09-20 12:19:43.530839877 +0200
+++ helloN/Property.csv	2021-09-20 12:19:44.494845443 +0200
@@ -3,7 +3,7 @@
 Property	Property
 UpgradeCode	{CABDEAD2-8BB7-0000-1234-383BE82C91BA}
 ProductName	hello
-ProductCode	{F91FF4C5-7407-4AB2-AF9D-610B25A3B14F}
+ProductCode	{639B3441-DB90-4F86-8C84-3C1CF80CE1B7}
 ProductVersion	1.1
 Manufacturer	Ian Caines
 ProductLanguage	1033
diff -u -r helloM/_SummaryInformation.csv helloN/_SummaryInformation.csv
--- helloM/_SummaryInformation.csv	2021-09-20 12:19:43.402839137 +0200
+++ helloN/_SummaryInformation.csv	2021-09-20 12:19:44.350844612 +0200
@@ -5,7 +5,7 @@
 3	hello
 4	Ian Caines
 7	x64;1033
-9	{F71D9729-90BB-4A44-9686-F72C2991F4B4}
+9	{9505635B-3BA1-4D60-AF51-7AAEACF7E700}
 14	200
 15	2
 18	Python MSI Library

[cainesi]

@Cimbali -- I think I have found the metric that the installer uses to decided that M > N. If you examine the .exe files installed by the two installers, you will see that the M executable has version information added to the file (look on the properties sheet for the file). I cannot find the reference just now, but I think for a component with a keypath but no version specified, the msi might look at the keypath file version information to determine which version of the component is newer.

It turns out that one other difference in the two virtual environments is that the environment for the M installer also had pywin32 installed. pywin32 is needed for WinFreezer._add_resources() to add version information to the base executable.

If you could find a way to force install all the components, that should fix it.

This would also explain why the problem cropped up with 99b3ad9 — that was the first time we had a component with a keypath I think?

[Cimbali]

I think you’re right, if we put together a number of different docs pages we can see that such an interaction could rule one file “better” than the other

Replacing Existing Files

Because unnecessary file copying slows an installation, the Windows Installer determines whether the component's key file is already installed before attempting to install the files of any component. If the installer finds a file with the same name as the component's key file installed in the target location, it compares the version, date, and language of the two key files and uses file versioning rules to determine whether to install the component provided by the package. If the installer determines it needs to replace the component base upon the key file, then it uses the file versioning rules on each installed file to determine whether to replace the file.

File Versioning Rules

Highest Version Wins—All other things being equal, the file with the highest version wins, even if the file on the computer has the highest version.

Versioned Files Win—A versioned file gets installed over a nonversioned file.

So I think what happens when upgrading from a “newer” executable to an “older” executable:

1. installer determines file is newer and doesn’t need to be installed
2. Upgrade table causes removal of old file
3. uninstall is committed
4. install is performed without re-evaluating missing files

That still seems wrong − I feel like if the process decides to remove a file it should take that removal into account at later steps.

Regardless, we either need to:

• remove the old package early enough that the missing file are taken into account.
  • That seems impossible, since RemoveExistingProducts can be scheduled at its earliest after InstallValidate
• re-evaluate the file costing after RemoveExistingProducts
  • This seems impossible too, with the strict ordering FileCost < CostFinalize < InstallValidate < RemoveExistingProducts
• Find a way to always force installing the executables. This 15 year old thread seems especially relevant with a number of suggestions:
  • By modifying the REINSTALLMODE property. This defaults to omus and is the most promising option as running msiexec REINSTALLMODE=dmus /i helloM.msi fixes the issue. We need to investigate whether the stronger amus is in fact required, by looking at what happens (and what would be expected) when an executable is « upgraded » to another executable with the same version.
    However, I wasn’t able to test this as any modifications I made to the CustomAction table seemed to somehow corrupt the file. i think we could try by modifying the source around here to add REINSTALLMODE along TARGETDIR:
    

    cx_Freeze/cx_Freeze/windist.py

    Lines 77 to 111 in f4d03e1

    	msilib.add_data(
    	self.db, "Property", [("SecureCustomProperties", "TARGETDIR")]
    	)
    	msilib.add_data(
    	self.db,
    	"CustomAction",
    	[
    	(
    	"A_SET_TARGET_DIR",
    	256 + 51,
    	"TARGETDIR",
    	self.initial_target_dir,
    	)
    	],
    	)
    	msilib.add_data(
    	self.db,
    	"InstallExecuteSequence",
    	[("A_SET_TARGET_DIR", 'TARGETDIR=""', 401)],
    	)
    	msilib.add_data(
    	self.db,
    	"InstallUISequence",
    	[
    	("PrepareDlg", None, 140),
    	("A_SET_TARGET_DIR", 'TARGETDIR=""', 401),
    	("SelectDirectoryDlg", "not Installed", 1230),
    	(
    	"MaintenanceTypeDlg",
    	"Installed and not Resume and not Preselected",
    	1250,
    	),
    	("ProgressDlg", None, 1280),
    	],
    	)

  • By first removing the file through the RemoveFile table which may apparently cause the installer to be aware of the file’s absence. This 2009 post also suggests this. I’ve managed to test this, and it does not work.
  • By using Companion files
    I haven’t had the time to look into this one yet, but it seems not to be recommended by the docs:

    Note that a file that is the key path for its component must not be a companion file.

[Cimbali]

I think 7eb94a1 should fix the issue

• it sets the REINSTALLMODE=amus which fixes the issue when passed on the command line
  • a stands for overwrite-all. I don’t think the difference with d, e, etc. matters that much: we want this anyway since we uninstall the previous version.
• it installs/uninstalls/repairs correctly

However I don’t have the precise virtualenv setup to build installers that cause the issue. Could you maybe test it @cainesi ?

[cainesi]

Apology for my ignorance, but how do I get that commit from github? It doesn't seem to be on a branch.

[Cimbali]

Ah it’s the main branch in my fork so you can check that out. Referring to the commit directly did something strange but you can also get the raw patch.

[cainesi]

@Cimbali -- Just did a quick check, and your fork works for me. Thanks!

[Cimbali]

Alright ! I made a PR @marcelotduarte see #1252

[marcelotduarte]

Do you need a 6.8.1 or 6.9b1 version?

[cainesi]

No rush from my perspective, it can just go into the next version whenever it comes out.

[marcelotduarte]

cx_Freeze 6.8.1 has just been released.