"click and run" Windows installation

[mfrasca]

From @mfrasca on October 24, 2015 16:12

quite rightfully, Ross Demuth writes:

The windows installation process needs to be a lot simpler or people will not even try it out

Copied from original issue: Bauble/bauble.classic#155

[mfrasca]

see #219

[mfrasca]

author @takluyver has written pynsist, specifically thought to build Windows installers for Python applications, and it even has a gtk example. I only do not manage to find the time to test it. (actually, I first need to grab a Windows machine, a powerful antiemetic, then I can learn how to do this.)
it would be so good if someone else would do this for Ghini!

[takluyver]

Hi! Coincidentally, I have a background in plant science, so I'm happy to see you're thinking of using Pynsist for this. Feel free to ping me with any questions you have.

It sounds like you have an aversion to Windows, so you may be pleased to know that Pynsist can build installers from Linux/Mac. You'll still need to use Windows a bit as you're figuring out how to make a working installer, but once it's set up, it's totally possible to build installers for new minor versions without running Windows.

The trickiest bit is getting the necessary files for pygtk in the right place. The grab_files.sh script in the gtk example deals with this, so long as the URLs it downloads from haven't changed since I wrote it.

[mfrasca]

Hi @takluyver . things sometimes meet nicefully. I will be able to use a Windows machine in a couple of weeks, possibly (or, إن شاء الله‎‎, as people here say.)
do I understand you correctly, that I would first need set it up on such a machine, then, that is once I'm sure it works, I can produce a one-click-installer executable file from my linux system?

[takluyver]

Yep, that's right. In fact, you can do even the initial setup from Linux, but it's a good idea to test the first installers on a Windows machine to make sure they're not missing anything.

[brettatoms]

If it helps there was an NSIS installer script that worked at some point: https://github.com/Ghini/ghini.desktop/blob/master/scripts/build.nsi

[RoDuth]

Let me know if you want me to test on my win 10 set up.

[mfrasca]

thank you all for the input. the NSIS installer script, I'm aware of it and it's not practical because I would need a windows machine to run it, isn't it? the pynsist would allow me set things up once and for all and then create the installer without having the windows machine I do not have.
sorry for the delay, I do not have the time to look into anything, at least for a couple more weeks.

[takluyver]

The NSIS script could also be run on a non-Windows machine. This is actually what Pynsist relies on: it generates a NSIS script using a template, and then runs it to build the installer. The NSIS builder can run on non-Windows platforms (on Debian/Ubuntu, run sudo apt-get install nsis).

[mfrasca]

oh, I was not aware of that. seems I need to review the whole thing then. 😟

[takluyver]

The difference is that your shortcut points to ghini.exe, which is built by py2exe (this is based on what I can see in build-multiuser.nsi - I haven't tried it out). I'm suggesting that an alternative is to have a shortcut that runs something like python.exe ghini.py, and not use py2exe at all. That makes it possible to change the code in ghini and rebuild the installer from Linux.

But that change would be some extra work, so it may be easier to just accept that Windows installers are built on Windows. It depends how often you're likely to need to build installers, and how painful using Windows is.

[mfrasca]

to the question "how often likely to build an installer": three years ago we were at bauble-1.0.11, now we are at ghini.desktop-1.0.83, so that's in average once every two weeks. my point of view is that if it isn't automated, I will forget using it, or how to use it. so I now have a script that does everything associated to publishing to the production branch: merge development into production, giving the merge commit a meaningful comment, bump version, commit the bump, push to github, build a debian package, verify it still matches the terribly strict quality checks by debian, push the package to mentors, push the new release to pypi, and I wish to do also the windows installer, never mind if I have to push it to github by hand, but the goal would be to automate that, too. (but the part regarding debian we still need to seriously review)

this way, if I also document the procedure, like "please use this script", and the script is consistently commented, then who knows, the next maintainer of this project will not need to reinvent the wheel.

apart from all the above consideration: I would VERY much prefer a Windows installer that does not compile the python sources!

actually, would it mean that the windows sources could stay in a git checkout? that would allow us produce only ONE initial Windows installer, and let user update by git pull, just like we do on a "normal" installation. did I understand you correctly? and do you know how to do what you're suggesting? or would be able to assist others in achieving this?

[RoDuth]

I've said it before I believe you could produce an nsis installer to do what you would describe but that wouldn't work for me.

Issue for me is that the government department in charge of the botanic garden has a very restricted environment (as most do in Australia, New Zealand and no doubt its similar in many others parts of the world). They flatly do not allow git, internet access is limited (I still haven't been able to get them to allow ghini access for the plant list name check function for instance), they prefer no full python install also. They allow 1 or 2 shared terminals to have global installs of ghini and all others to have per user installs. Also everything must be able to be remote deployed silently. For them almost everything is now deployed on servers and accessed by web browsers, thin clients, remote desktops, etc. with most of that largely run remotely by contractors who take responsibility for the security etc. Ghini in a remote desktop is a pain if you ask me. As it stands they would not be in for running a fresh installer every 2 weeks so I have to be strategic about it. This is what has had me looking again at collectionspace or @brettatoms bauble.web (neither are ideal).

if you wish to just have it automated in linux then what is wrong with a VM and VBoxManage? Its simple enough from what I can gather. Would be happy to give it a try. I spend 98% of my time on linux and as little as possible in windows, most of that in a linux hosted VM. At the end of the day your going to need to test all is working on a windows install anyway and I highly recommend a VM for that (for testing there are these free VMs, use the win7 one)

There are other benefits to the py2exe, NSIS combination that Brett chose to use. It really does compress things quite a lot for instance. I think I got rid of most of the excess but no doubt it could be improved further. And it could be possible to have smaller changes made as a patch release.

[takluyver]

Even besides the issues with locked down systems, I think an install that you update by doing git pull is not terribly practical. Your initial installer would need to install git and ensure it's on PATH, and you'd need extra steps if you ever wanted to add or update a dependency.

Another option for automation would be to use Appveyor to host Windows continuous integration. I see you already have config for Travis CI; Appveyor is the same kind of service, but with Windows servers. You can get it to build any version with a tag and make the result available for download.

[mfrasca]

we indeed need git, and we do need it in the PATH, and git pull is indeed followed by python setup.py install. I do git merge && git push (actually publish.sh), and that's all I need, users will see the software informing them they can update, and they have a script to do that.
I'll check www.appveyor.com … do you know whether they can push files from their server back to git? in that case, when it runs our tests on the production branch, it could even conclude a successful test with the windows builder, commit and push to github. like travis-ci lets you do, for example, to produce lektor web sites on git pages.

[takluyver]

Looks like it has support for uploading to Github releases. I don't think there's built-in support for pushing to git - storing large binary files in git is generally not recommended. But if you really want installers to go into a git repository, you can configure it with arbitrary commands to do that.

[RoDuth]

@mfrasca You may need git but no one I work with can use it, are you sure you wish to alienate them? Just spent a few moments looking into Appveyor and so far it looks great. Thanks @takluyver for the ref.

[RoDuth]

as for including anything in the path that is the reason for the visual basic script add_to_path.vbs and you are welcome to use it. Drawing from my aging memory.. apparently anything batch file wise is likely to stuff your PATH up because of it's limits on length of a string or something along those lines... Apart from the fact that I really wasn't comfortable with writing VB I do remember putting this in the scripts comments if you do wish to use it:

' a simple script used by the nsis installer to prepend to the PATH
'
' EXAMPLES:
' cscript.exe Add_to_PATH.vbs /path:"C:\TEST" /env:"USER"
' cscript.exe //E:vbscript Add_to_PATH.vbs /env:"SYSTEM" /path:"C:\TEST\"

[mfrasca]

I guess I can live with uploading to github releases. 😉
are you interested in having all necessary permission for doing this part?

(@RoDuth : alienate whom? I'm not aware of anybody down there, besides you.)

[takluyver]

are you interested in having all necessary permission for doing this part?

Me? Not really, sorry! I've got too many projects on the go already that I'm trying to find time to work on. I'm happy to continue answering questions and providing pointers.

I think that having end users pull updates with git, even through a wrapper script, would probably cause you more pain in the long run. It's impractical for updating any computer that's not on the internet, and it's hard to debug when something goes wrong.

[mfrasca]

Me? Not really, sorry!

never mind, one asks, just in case.

pull updates with git […] pain in the long run.

my experience with it was quite the opposite, you do need at least one user able to run a command from the command prompt, but then they can help tracing back errors, when something goes wrong. it's been more a help factor than a burden.

the whole python setup.py install is based on pip install and relies on being on the internet, and that yes has been a problem in several cases. … well, however, if we first exclude those where they had no permission to install anything anyway, it was at one station somewhere in Peruvian Amazonas.

[takluyver]

Your call, of course, but I think @RoDuth is pointing out that internet restrictions at their institution would cause issues, even if it's not deep in the Amazon. Given what Ghini does, I'd imagine that it's not unusual for potential users to be in similar institutional environments.

[mfrasca]

I think that offering both ways is a good option. I have a user who's unable to install because, well, I have not the faintest idea, they just panic and can't tell me where it goes wrong. so a 'click and run' would be the only option in such cases.
but wherever I manage to install the program myself, I prefer git-based, so the update becomes a click and run action.

[mfrasca]

so, apart from me not understanding what was going on because of mismatched labels, we have a perfectly functional tool for producing both: a windows installer; a windows relocatable runtime.
I think I would only like to find the windows installer in the dist directory (ends now in scripts), and the windows relocatable runtime in a new directory that suggests this usage, something like ghini-runtime maybe, or copy-me-to-a-usb-drive, (ends now in dist, which gets previously completely emptied, possibly not completely desirable).

[mfrasca]

2b2c47f
bb38fa0
243c32c
74919ef
a6a8f5d

[mfrasca]

does any of you know how to solve windows issues like:
WindowsError: [Error 32] The process cannot access the file because it is being used by another process
we get quite a bunch of them on AppVeyor.
https://ci.appveyor.com/project/mfrasca/ghini-desktop

[takluyver]

It looks like it's an error from a file being deleted during teardown of a test. On Windows, you have to ensure that a file is closed before deleting it. In some situations you can also only have a file open in one place at once.

Looking at the code which is failing the tests, one possible culprit is this pattern:

result = json.load(open(self.temp_path))

That's relying on garbage collection to close the file, which may not happen until after the teardown tries to remove it. Try replacing those instances with:

with open(self.temp_path) as f:
    result = json.load(f)

That closes the file deterministically as soon as you're done reading it.

[mfrasca]

It's more than the above, and there are subtle differences. see #384.