Make windows installers quiet, issue #572 #4727

Closed
wants to merge 3 commits into
from

Conversation

Projects
None yet
5 participants

The possibility for a silent numpy and/or scipy installer has been requested a number of times.
I have made a number of changes that should enable this, please review this. I'm not familiar with compiling numpy, so this is untested!

The basic idea is to create MSI installers instead of exe's. Then we can use msiexec and pass it a silent flag if the NSIS installer was passed a silent flag. This can be achieved using the NSIS 'IfSilent' directive.

Owner

rgommers commented May 20, 2014

Thanks a lot for having a go at this! I'll try to do some testing next week (only under Wine though).

Any news on this?

Owner

rgommers commented Jun 2, 2014

Sorry, didn't get around to it yet. Just tried, first issue:

$ paver bdist_wininst_simple -p 2.7 -m 1
...
optparse.OptionError: invalid long option string '-msi': must start with --,followed by non-dash
pavement.py
-@cmdopts([("python-version=", "p", "python version")])
+@cmdopts([
+ ("python-version=", "p", "python version"),
+ ("make_msi=", "msi", "0 or 1. If 1, build a msi instead of an exe.")
@rgommers

rgommers Jun 2, 2014

Owner

This needs to be ("make_msi=", "-msi", .... Same for other lines like this.

@mathijs-dumon

mathijs-dumon Jun 3, 2014

I disagree ;)
Otherwise the line above should also be ("python-version", "-p", ... instead of ("python-version", "p", ...
But you are correct this line is the culprit, it should be:
("make_msi", "m", "...")
Because single-dash options need to be single-character options, double-dash options are long options. Will change this accordingly.

Owner

rgommers commented Jun 2, 2014

Next one:

creating dist
Traceback (most recent call last):
  File "setup.py", line 251, in <module>
    setup_package()
  File "setup.py", line 243, in setup_package
    setup(**metadata)
  File "Z:\Users\rgommers\Code\numpy\numpy\distutils\core.py", line 169, in setup
    return old_setup(**new_attr)
  File "C:\Python27\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "C:\Python27\lib\distutils\dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:\Python27\lib\distutils\command\bdist_msi.py", line 232, in run
    sversion = "%d.%d.%d" % StrictVersion(version).version
  File "C:\Python27\lib\distutils\version.py", line 40, in __init__
    self.parse(vstring)
  File "C:\Python27\lib\distutils\version.py", line 107, in parse
    raise ValueError, "invalid version number '%s'" % vstring
ValueError: invalid version number '1.9.0.dev-Unknown'

Will only happen when you run a non-released build in a way where git cannot be found (like building master under Wine, with git not installed in Wine).

Owner

rgommers commented Jun 2, 2014

Then

  File "numpy\core\setup_common.py", line 92, in check_api_version
    MismatchCAPIWarning)
setup_common.MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 9, with checksum 3e1c969a871ad01c622aa1432a04cdc5, but recorded checksum for C API version 9 in codegen_dir/cversions.txt is 49b27dc2dc7206a775a7376fdbc3b80c. If functions were added in the C API, you have to update C_API_VERSION  in numpy\core\setup_common.py.

I don't see so quickly why this warning results in an exception. Looks build_msi specific. Would be nice to have fixed, because we don't always update the C_API_VERSION in master (only before a release).

EDIT: also happens for a normal build. Looks like a regression in master (?!).

mathijs-dumon added a commit to mathijs-dumon/numpy that referenced this pull request Jun 3, 2014

Updated pavement.py
Update to fix the short build_msi command line option to "m" instead of the invalid "msi". See [this comment](numpy#4727 (comment))

I'm interpreting there's not much I can do about the last two errors? If so, I'll be impatiently awaiting a slot of your time for further testing after these are resolved ;-)

I'm currently testing the build process on my Linux machine using wine and MingW, and this seems to be working (albeit without the blas/lapack/atlas support, by forcing a "release" or distutils keeps complaining and by updating the C API checksums).
Can you confirm the issues are resolved?

I have been able to compile and install the superpack in and on Wine. This confirms the changes I've made are working. The issues with 'actually' compiling numpy are not related to my changes so I see no more reason not to move forward with the merge?

For anyone interested in how I got it to build, this is a script I wrote to keep track of everything I needed to do. It is untested, but should give you a rough idea of what to do and where to find everything.

#!/bin/sh
echo "CLEANING WINEPREFIX:"
rm -Rfv "`pwd`/.wine"
echo "DONE CLEANING";
export WINEPREFIX="`pwd`/.wine";
export WINEARCH="win32";

# Download & install SCONS:
wget http://downloads.sourceforge.net/project/scons/scons/2.3.1/scons-2.3.1-setup.exe
wine scons-2.3.1-setup.exe

# Download & install git:
wget https://github.com/msysgit/msysgit/releases/download/Git-1.9.4-preview20140611/Git-1.9.4-preview20140611.exe
wine Git-1.9.4-preview20140611.exe

# Download & install python and add to path:
wget https://www.python.org/ftp/python/2.7.7/python-2.7.7.msi
wine msiexec /i python-2.7.7.msi /qn

# This is a registry file exported using regedit where I added the Python, 
# MinGW and NSIS paths to the PATH environemnt variable in 
# HKEY_CURRENT_USER/Environment
wine regedit python27+mingw32+NSIS.reg

# Download & install setuptools and pip:
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
wine python ez_setup.py
wine easy_install pip

# Download & install MingW and set Python default compiler to MingW:
wget http://downloads.sourceforge.net/project/mingw/Installer/mingw-get-setup.exe
wine mingw-get-setup.exe
printf "[build]\ncompiler = mingw32" > /home/mathijs/python_wine_prefix/.wine/drive_c/Python27/Lib/distutils/distutils.cfg 

# Download & install cython
wine pip install cython

# Download & install msvc dll:
winetricks vcrun2008

# Download & install paver:
wine easy_install paver

# Download & install NSIS:
wget http://downloads.sourceforge.net/project/nsis/NSIS%202/2.46/nsis-2.46-setup.exe?r=http%3A%2F%2Fnsis.sourceforge.net%2FDownload&ts=1402918286&use_mirror=sunet
wine nsis-2.46-setup.exe
# TODO you also need to install the CpuCaps dll. I managed to compile it from µ
# the numpy source files using MingW by first installing this:
#  mingw-get.exe upgrade "w32api=3.17-2"
# Second step is to add a header file to the MingW installation,
# called "sdkddkver.h". Put it in the C:\MinGW\include\ of the wine prefix. 
# Contents are:

#ifndef _WIN32_WINNT
#  ifdef WINVER
#    define _WIN32_WINNT WINVER
#  else
#    ifdef _WARN_DEFAULTS
#      warning _WIN32_WINNT is defaulting to _WIN32_WINNT_WIN2K
#    endif
#    define _WIN32_WINNT _WIN32_WINNT_WIN2K
#  endif
#endif

#ifndef WINVER
#  define WINVER _WIN32_WINNT
#endif

# Then I could cd into the source dir:
#  cd numpy/tools/win32build
#  wine python build-cpucaps.py
# If all goes well, this should install the CpuCaps.dll file where it should be.

# Finally clone git & build numpy:
git clone https://github.com/mathijs-dumon/numpy.git
cd numpy
wine paver bdist_superpack -p 2.7 -m 1
#wine python setup.py build --compiler=mingw32 bdist_wininst
# If the installer builds the exe's but fails at making the superscript, you can
# run this last part seperately like this;
#  cd build-superpack/
#  wine makensis numpy-superinstaller.nsi

@rgommers: sorry for bugging you, but what's the status on this? If at all possible I'd like to see this land for the next release.

Contributor

juliantaylor commented Jun 28, 2014

I built msi binaries using this patch on our wine build setup and tried installing them but it immediately gives an error with no further useful information.
the regular exe installers work fine

Contributor

juliantaylor commented Jun 28, 2014

nope made a mistake the regular exe installers are broken with this patch too

That's strange, when I tested the installation went fine. I won't be able to test again until tuesday (maybe), if I can test I'll report back (and also test on a windows install).

mathijs-dumon added some commits May 20, 2014

Changes related to solving issue #572 by creating MSI installers and …
…adjusting the NSIS installer script to allow for a silent install on windows.
Updated pavement.py
Update to fix the short build_msi command line option to "m" instead of the invalid "msi". See [this comment](#4727 (comment))

Alright, time for some more work on this:

  • I can confirm that the installer didn't work in Windows with the very clarifying message: "This action is only valiud for products that are currently installed". No idea what this is causing, but I have made a small change that might have been the cause. I'm re-building as we speak,
  • I have rebased my fork onto the current upstream master, to keep this work relevant.

I'll get back in a few minutes if this build works out

Owner

charris commented Aug 23, 2014

@mathijs-dumon Is this something that might get worked out at Euroscipy?

@charris charris changed the title from Changes related to solving issue #572 to Make windows installers quiet, issue #572 Aug 23, 2014

Sadly I'm not able to go to Euroscipy, so unless somebody else has a go at this there, it'll have to go the electronic way. I just need to find the time to sit down and figure out what is causing this, but with the start of the academic year in a few weeks, I'm pretty tied up for the time being.

@charris @rgommers @juliantaylor
I have been able to successfully produce an msi installer with the current patch. However, building this installer under wine (either linux or mac) does not work and produces the error we've seen above.
Silent install also worked as expected when built under windows using MinGW.

Can an 'official' builder confirm wether this works or not?

@mathijs-dumon mathijs-dumon reopened this Nov 26, 2014

Owner

charris commented Jan 21, 2015

@mathijs-dumon @rgommers @juliantaylor I'm tempted to close this. Is the Wine failure important? Is there another way to get the same result?

Well, I mostly wanted this for my own project, but have resorted to manually unpacking the three versions and using a python script to detect the capabilities of the user's PC and select the correct version. It works, but I would have rather had a way supported by the project.

So you may close this, I have no idea why this doesn't work under wine or how to solve it anyway, and I don't have enough time to figure it out anytime soon.

Contributor

matthew-brett commented Jan 21, 2015

If we get the new OpenBLAS / mingw-w64 binaries working then we won't have this problem any more because we won't need the separate installer step to choose different versions according to the CPU, because OpenBLAS detects the CPU at run-time.

Owner

rgommers commented Jan 21, 2015

It feels like a shame to close this, because a silent installer is nice to have. However, I don't really have the time to dig into Wine issues......

Owner

charris commented Jan 21, 2015

OK, closing this. It can be found with a search after if someone wants to open it again.

@charris charris closed this Jan 21, 2015

Owner

rgommers commented Jan 22, 2015

@mathijs-dumon thanks for the effort! I hope this'll still make it in somehow in the future

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment