Test against all supported Python versions and target platforms

[bittner]

We should automate testing (and building installers) against all combinations of:

1. target platforms (Windows, macOS, GNU/Linux distros)
2. supported Python versions (2.7, 3.5, 3.6, 3.7, PyPy, PyPy3)
3. supported versions of wxPython (wxgtk2.8, wxgtk3.0, wxgtk4.0)

This may or may not require switching from Travis to (or using in addition) AppVeyor.

It will make sense to use prepared Docker images for all combinations.

Related Issues

• Migrate PythonTurtle to wxgtk4.0 #99
• Fix src module name (src -> pythonturtle) #106
• Build installer packages automatically #107
• Prepare Docker image with wxPython + OS/distro combinations wxWidgets/Phoenix#1003

[cool-RR]

I think that some of these could be pruned out. We've been doing a lot of talk over the last couple of weeks, but no one has done real persistent work on PythonTurtle since I developed it a decade ago. I worry that setting goals that are too difficult are just going to make it less likely that you're going to stick around and develop PythonTurtle to an actual release. If you're intent on supporting all the combinations you specified, and you're going to actually do it, that'll be great. I won't stand in your way. I think that the most important setup for users is an installer for Windows that bundles Python 3.7 and the latest wxPython, and for this case there's no need for so many combinations of versions. I know you're not very interested in Windows. Keep in mind that this Windows version is why PythonTurtle became part of the official curriculum in Saudi Arabia, and is also used in other schools throughout the world. It's helped introduce Python to millions of students. In any case, feel free to proceed. The only question to me is whether you're going to keep working past the initial step of excitement and to the point where there's an actual release, whatever platform it may be.

…

On Sun, Sep 23, 2018 at 3:10 PM Peter Bittner ***@***.***> wrote: We should automate testing (and building installers) against *all combinations* of: 1. target platforms (Windows, macOS, GNU/Linux distros) 2. supported Python versions (2.7, 3.5, 3.6, 3.7, PyPy, PyPy3) 3. supported versions of wxPython (wxgtk2.8, wxgtk3.0, wxgtk4.0) This may or may not require switching from Travis <https://travis-ci.org/cool-RR/PythonTurtle> to (or using in addition) AppVeyor <https://packaging.python.org/guides/supporting-windows-using-appveyor/>. It will make sense to use prepared Docker images <wxWidgets/Phoenix#1003> for all combinations. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#108>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADdygcd9Gu_l-xD35zZ05RxGtLQ3useks5ud3odgaJpZM4W1rRM> .

[bittner]

If PyInstaller is actually packaging Python (3.7) in a way that is completely independent of the target platform and the wxPython version used in PythonTurtle's implementation is actually available for the target platform (or - even better - if also the wxPython dependencies are bundled with the binary) then there should be no need to worry about testing against several combinations.

What I'm worrying about is: If it proves true that substantial changes done to the code base that make PythonTurtle run on Python 3.7 alone entail that it won't run on, say, Windows or macOS you will most probably refuse to merge the changes. For obvious reasons.

Given that there are no tests that allow us to verify (future) changes will work out we need to do some preparation work. Employ linting was one of them (unfortunately with less impact than hoped for -- it would require some pair-programming, a dedicated sprint -- or just so personal dedication of yours as you are the one to know the code base best in order to make critical decisions).

I think, now it's your turn. You've already given your 👍 to using PyInstaller. Now we need a list of what steps are required to make packaging a release happen. For simplicity's sake we take it for granted that the result for running PyInstaller will work on all three supported target platforms (Windows, macOS, GNU/Linux) is some way. If there is anything specific needed for GNU/Linux that you can't determine I'm ready to jump in. Thank you!

[cool-RR]

1. I think that PyInstaller indeed bundles Python and all its packages in the package. That's what py2exe does. The PyInstaller docs say "PyInstaller bundles a Python application and all its dependencies into a single package".

2. I don't understand your second paragraph. If you want to move forward with PythonTurtle, you should just make the code run under Python 3.7 and the latest wxPython. Normally, differences between operating system are small in wxPython, and it's not too difficult to fix them. And even if there are differences or minor bugs, fuck it. As long as it kinda works, that would be great.

3. Regarding tests: Testing GUIs is very difficult. I think that it's good to test GUIs, but even GUIs of programs much more complex with a bigger budget than ours are sometimes lacking in testing. If you just want to quickly see that the program doesn't fail terribly, here's an idea: Make a test that just starts PythonTurtle, waits for 5 seconds and asserts that the process is still alive. This will catch many errors with a low cost. If you want to go a bit further, you can use a GUI interaction library to assert that a window was created with the right title. Most of the actual testing must be manual. Unless you're willing to invest a whole lot of time in a cross-platform GUI testing framework.

4. I don't see any importance in adding linting right now. I don't object to it, and you can go ahead and do that, but I don't see why it's important or urgent, or what the problems with it are.

5. Regarding what steps are required: Set up a virtualenv with Python 3.7 and the latest wxPython. Attempt to run PythonTurtle. There will be bugs, fix them one-by-one. Once it works, go through the PyInstaller tutorial to create a PythonTurtle package for one of the OSes. Test that it works. Then try it for the two other OSes. Once that works, ensure that it can be done with just one command, automatically. Set up a CI job to create these installers from source. You can add the minimum-effort testing I mentioned before, if you'd like. When that works, and the program doesn't have bugs that are too severe, we could make a release. Then you could develop more features and fix more bugs and make more releases with those changes. Yes, it's a lot of work.

[bittner]

This issue now seems to be fixed for most parts:

1. PyInstaller has proven to bundle all dependencies (verified on Ubuntu by @bittner, and on Windows by @cool-RR). ✔️
2. PyInstaller completely isolates the dependencies, including the Python version and wxPython. This makes it irrelevant to continue supporting older Pythons and matrix-test with different wxPython versions. In future we may also build AppImages to provide yet another distribution format implementing full isolation (on GNU/Linux). ✔️
3. We now have a test suite. The "tests" are somewhat pointless (in a way that they test what I see that Pylint already does: import the wx module to detect whether the installed prerequisite is actually promoting through to the Python version being tested against). While Pylint spits out warnings, the test abort, though. ✔️
4. Linting is in place, and being run automatically by Travis. Pylint is now still allowed to fail. I plan to have the remaining complaints fixed, though, and then flip the switch to start enforcing code quality (by stopping to allow build jobs to fail). ✔️
5. What has to be done for packaging, this is now implemented with a bundle command of setup.py, in a cross-platform way. ✔️

We should probably also run the test suite on both Windows and macOS. Just to make sure that differences across platforms, if they affect our code base, are potentially spotted (or at least, that we can write a test reproducing a platform dependent behavior, if this is being discovered).

[bittner]

Ooops, there is something wrong with the macOS build on Travis:

• It installs the wxPython for Ubuntu Xenial! (see build details)
• Instead, it should simply install the package directly from PyPI (via python3 -m pip install wxPython)
• Not sure why the tests don't fail (neither for installing the package, nor for importing it in Python)

[bittner]

Aha, it actually installs the correct version (a wheel for macOS).

However, the --find-links option should be removed for the macOS case. To avoid confusion, and potential pitfalls.

[bittner]

I will likely replace the Travis and AppVeyor builds by GitHub Actions, which covers the whole spectrum and actually works, as proven by the setup I have for PyClean.

For building the MSI installer I found two interesting projects that may be able to help us out:

• https://github.com/microsoft/github-actions-for-desktop-apps
• https://github.com/mmckegg/msi-packager (WiX-based)

EDIT There is even a Python-powered WiX-based MSI creator: WiX.Py (unfortuantely still based on Python 2)

• https://wix.sk1project.net/
• https://github.com/sk1project/wixpy

I opened sk1project/wixpy#18. Python 3 support seems on its way.

[cool-RR]

Sounds good.

…

On Sun, Aug 30, 2020 at 4:20 PM Peter Bittner ***@***.***> wrote: I will likely replace the Travis and AppVeyor builds by GitHub Actions, which covers the whole spectrum and actually works, as proven by the setup I have for PyClean <https://github.com/bittner/pyclean/actions>. For building the MSI installer I found two interesting projects that may be able to help us out: - https://github.com/microsoft/github-actions-for-desktop-apps - https://github.com/mmckegg/msi-packager — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#108 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAN3SQTXXSLSMOFYJEHFS3SDJGZNANCNFSM4FWWWRGA> .