Proposal for an automated build system for RT windows packages

[aferrero2707]

I am currently setting up and automated cross-compilation of the Windows version of RT code via Travis CI. The final goal is to have a .zip package automatically deployed to a specific github release page each time a new commit is made into certain branches.

The mechanism is based on a custom-crafted Docker container which provides a pre-installed Windows cross-compilation environment for GTK applications. The cross-compilation is automated via a tool called crossroad which is provided by one of the GIMP developers.

The goal is to remove the need of human intervention in the tedious process of keeping up-to-date Windows packages from the development branches, and also guarantee the reproducibility of the builds by making sure that the same environment is used each time. Moreover, the deployment does not rely on the availability of specific users and/or systems.
Another clear advantage is that the whole Windows build chain is public, and all developers can see it and do modifications if needed.

The current Travis CI configuration and scripts can be found in my forked RT repo, respectively [here](https://github.com/aferrero2707/RawTherapee/blob/dev/.travis.yml] and here.

ZIP packages are automatically deployed here. A first package is ready for testing here. Packages are tagged with the branch, build date and commit hash, and will be stored in the same github release page.

Known issues:

• the rewtherapee.exe executable is big, because it is built in RelWithDebInfo mode
• the lensfun database seems to be not correctly found, and automatic lens corrections are disabled - I am looking into this
• few UI icons are missing

Any feedback is very much appreciated!

P.S: I am looking into the possibility to automate the creation of OSX .dmg packages with Travis CI as well, using homebrew to provide the required dependencies. I will open a separate issue if you are interested into this as well.

[ClaesGitHub]

Hi! Problem when trying to unzip .

I am on Win10 Pro, 64-bit, 10.0.16299 Build 16299.
Have fun!
Claes in Lund, Sweden

[aferrero2707]

The full path of the offending file in the .zip package is the following:

./share/gdb/auto-load/usr/x86_64-w64-mingw32/sys-root/mingw/lib/libglib-2.0.so.0.5200.2-gdb.py

I am surprised that this path is too long, but I have no big experience in this kind of win-related problems.

Anyhow, a new package in which the ./share/gdb folder is completely removed is currently being built, and should appear soon here.

[aferrero2707]

The new package is here.

[ClaesGitHub]

Ha! You most certainly know that the length is of importance!
Here is perhaps an explanation:
https://community.spiceworks.com/topic/174820-cannot-unzip-file-file-path-too-long
OK - I will download the new pkg in a few minutes'...

[ClaesGitHub]

a) Unzipped OK.
b) Rapid testing ... no problems found :-)
c) Perhaps a little slow, as compared to under linux?

Have fun!
Claes in Lund, Sweden

[aferrero2707]

Thanks! Is it also slower than the official windows builds? Il 10/nov/2017 19:52, "Claes" <notifications@github.com> ha scritto:

…

a) Unzipped OK. b) Rapid testing ... no problems found :-) c) Perhaps a *little* slow, as compared to under linux? Have fun! Claes in Lund, Sweden — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#4174 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF8M9YZwvVlHa69hnm5teSD0Aqj-kuz6ks5s1JtXgaJpZM4QX1Q8> .

[ClaesGitHub]

You're welcome :-)
Slower than the official Win builds? Sorry, cannot say, since I am rarely visiting my Win partitions these days. My feeling was based on a comparison with the Linux/Ubuntu build.

Hm... Il 10/nov/2017 19:52, "Claes" ha scritto???
Scusi, Dottore, io stupido, mi Italiano e solo poco, poco!
I once took a Linguaphone course in the Italian language,
so now I can say that the market square is 60 steps to
the left of the church, and other useful phrases!
Cordialemente,
Claes

[gaaned92]

@aferrero2707
I have some remarks on this work. Please take them in a positive manner.

The goal is to remove the need of human intervention in the tedious process of keeping up-to-date Windows packages from the development branches

On windows with MSYS2, you can completely automate the process with bash scripts, included uploading the builds. So it is not a concern.
My only problem is that I am not a bash wizzard, my scripts are rather clumsy

also guarantee the reproducibility of the builds by making sure that the same environment is used each time

Presently, the windows builds are reproducible are they are all done with the same Msys2 environment if we care to have an up to date Msys2 and use the same flags for building (this can be contrommed in the aboutthisbuild.txt). What can differ is the lensfun version used.
Unless you use also Msys2, you are introducing variability in what was uniform.
For instance I notice that you use an old version of GTK+. For windows builds, in order to have native windows, you must use version >=3.22.24

Moreover, the deployment does not rely on the availability of specific users and/or systems.
Another clear advantage is that the whole Windows build chain is public, and all developers can see it and do modifications if needed.

Agreed

The mechanism is based on a custom-crafted Docker container which provides a pre-installed Windows cross-compilation environment for GTK applications.

So we will have two environments : a Msys2 native environment and a Unix environment.
Th Unix environment doesn't seem up to date (see GTK+).
Until now there was a consensus in the dev team about windows building described in http://rawpedia.rawtherapee.com/Windows.
They will have to make a decision.

the rewtherapee.exe executable is big, because it is built in RelWithDebInfo mode

Yes you should build in release mode.
The windows package include now Rawtherapee.exe and rawtherapee-cli.exe in release mode, rawtherapee-debug.exe in debug mode and gdb.exe (see above link)
Moreover, it seems there are a lot of useless files, dlls and directories that bloat the zip. grab a windows build to see what is needed.

the lensfun database seems to be not correctly found, and automatic lens corrections are disabled - I am looking into this

As windows builds are bundled , we can hope that the lensfun version is the latest one from master branch built in the same way as rawtherapee. Msys2 lensfun being obsolete is not used for windows builds.

few UI icons are missing

see on existing windows build. At least adwaita icons are missing (we use only a few icons)

Slower than the official Win builds?

It seems so and I did not made exact measure. And comparison is not informative as official win builds use -O3 flag.

Other remarks

• For development buids you should use 5-dev suffix for option file. option file is thus located in .../rawtherapee5-dev.
  for stable official builds suffix is blank, thus option file is located in .../rawtherapee

• aboutthisbuild.txt: for some reason, as it is usual in unix distributions, version and branch are not what they should be and coherent with rawtherapee repository:
  branch: continuous-77-g7fb6be6 there is no such branch in rawtherapee git repository
  version : continuous-77-g7fb6be6 there is no such version in rawtherapee git repository

• zip naming: rawtherapee-w64-20171110_1742-git-dev-7fb6be6cddca75f3d11b2e3a9cfd89b794204a35.zip
  name fields should be identical to corresponding fields in aboutthisbuild.txt. I notice here that branch name is correct. No need for complete commit hash, version field of aboutthisbuild.txt is sufficientand name will be shorter.

Otherwise, it runs ok.

[Beep6581]

On windows with MSYS2, you can completely automate the process with bash scripts, included uploading the builds. So it is not a concern.

Well it is a concern as the only way to actually get the build is to depend on someone who has Windows to have that whole environment set up and to be available to create it. With this automated method, the build is created the moment a commit to a designated branch is made, without having to rely on people being available and not missing some step, having virus-free machines, timezones, etc.

Some dependency being of a wrong version is just something which needs to be fixed, it's not an argument against the whole idea.

@aferrero2707 I will be on IRC for the next few hours, if you're available we could go through the script.
http://rawpedia.rawtherapee.com/IRC

[gaaned92]

@Beep6581

You miss the point. I never said I was against that. As@aferrero2707 asked for remarks, I made remarks hoping it will help.
So I will not make any other remark on this topic.
The decision is of course, up to the developpers.
The problem seems worst on the different Linux systems with people complaining with obsolete versions available in their distributions.

[Beep6581]

@gaaned92 I did not mean to upset you, apologies if I did.

I agree/confirm that many of the latest distributions lack recent versions of key packages (the easiest way to check is through https://distrowatch.com/ ), however that is not a problem here, as the Docker image can be made to use whatever we need. Hopefully.

[gaaned92]

@Beep6581
It's ok. I was only a little disoriented by your answer as I think it is a way to go if it can fulfill all dev's needs.
What you quoted is only an answer to the burden to build on windows which is really not.

[sguyader]

@gaaned92 it's not a burden, but it still requires human action at this point, if I'm not mistaken.