New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal for an automated build system for RT windows packages #4174

Closed
aferrero2707 opened this Issue Nov 9, 2017 · 22 comments

Comments

Projects
None yet
5 participants
@aferrero2707
Collaborator

aferrero2707 commented Nov 9, 2017

I am currently setting up and automated cross-compilation of the Windows version of RT code via Travis CI. The final goal is to have a .zip package automatically deployed to a specific github release page each time a new commit is made into certain branches.

The mechanism is based on a custom-crafted Docker container which provides a pre-installed Windows cross-compilation environment for GTK applications. The cross-compilation is automated via a tool called crossroad which is provided by one of the GIMP developers.

The goal is to remove the need of human intervention in the tedious process of keeping up-to-date Windows packages from the development branches, and also guarantee the reproducibility of the builds by making sure that the same environment is used each time. Moreover, the deployment does not rely on the availability of specific users and/or systems.
Another clear advantage is that the whole Windows build chain is public, and all developers can see it and do modifications if needed.

The current Travis CI configuration and scripts can be found in my forked RT repo, respectively [here](https://github.com/aferrero2707/RawTherapee/blob/dev/.travis.yml] and here.

ZIP packages are automatically deployed here. A first package is ready for testing here. Packages are tagged with the branch, build date and commit hash, and will be stored in the same github release page.

Known issues:

  • the rewtherapee.exe executable is big, because it is built in RelWithDebInfo mode
  • the lensfun database seems to be not correctly found, and automatic lens corrections are disabled - I am looking into this
  • few UI icons are missing

Any feedback is very much appreciated!

P.S: I am looking into the possibility to automate the creation of OSX .dmg packages with Travis CI as well, using homebrew to provide the required dependencies. I will open a separate issue if you are interested into this as well.

@ClaesGitHub

This comment has been minimized.

Show comment
Hide comment
@ClaesGitHub

ClaesGitHub Nov 10, 2017

Hi! Problem when trying to unzip .
cgl_171110a
I am on Win10 Pro, 64-bit, 10.0.16299 Build 16299.
Have fun!
Claes in Lund, Sweden

ClaesGitHub commented Nov 10, 2017

Hi! Problem when trying to unzip .
cgl_171110a
I am on Win10 Pro, 64-bit, 10.0.16299 Build 16299.
Have fun!
Claes in Lund, Sweden

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Nov 10, 2017

Collaborator

The full path of the offending file in the .zip package is the following:

./share/gdb/auto-load/usr/x86_64-w64-mingw32/sys-root/mingw/lib/libglib-2.0.so.0.5200.2-gdb.py

I am surprised that this path is too long, but I have no big experience in this kind of win-related problems.

Anyhow, a new package in which the ./share/gdb folder is completely removed is currently being built, and should appear soon here.

Collaborator

aferrero2707 commented Nov 10, 2017

The full path of the offending file in the .zip package is the following:

./share/gdb/auto-load/usr/x86_64-w64-mingw32/sys-root/mingw/lib/libglib-2.0.so.0.5200.2-gdb.py

I am surprised that this path is too long, but I have no big experience in this kind of win-related problems.

Anyhow, a new package in which the ./share/gdb folder is completely removed is currently being built, and should appear soon here.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Nov 10, 2017

Collaborator

The new package is here.

Collaborator

aferrero2707 commented Nov 10, 2017

The new package is here.

@ClaesGitHub

This comment has been minimized.

Show comment
Hide comment
@ClaesGitHub

ClaesGitHub Nov 10, 2017

Ha! You most certainly know that the length is of importance!
Here is perhaps an explanation:
https://community.spiceworks.com/topic/174820-cannot-unzip-file-file-path-too-long
OK - I will download the new pkg in a few minutes'...

ClaesGitHub commented Nov 10, 2017

Ha! You most certainly know that the length is of importance!
Here is perhaps an explanation:
https://community.spiceworks.com/topic/174820-cannot-unzip-file-file-path-too-long
OK - I will download the new pkg in a few minutes'...

@ClaesGitHub

This comment has been minimized.

Show comment
Hide comment
@ClaesGitHub

ClaesGitHub Nov 10, 2017

a) Unzipped OK.
b) Rapid testing ... no problems found :-)
c) Perhaps a little slow, as compared to under linux?

Have fun!
Claes in Lund, Sweden

ClaesGitHub commented Nov 10, 2017

a) Unzipped OK.
b) Rapid testing ... no problems found :-)
c) Perhaps a little slow, as compared to under linux?

Have fun!
Claes in Lund, Sweden

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Nov 10, 2017

Collaborator
Collaborator

aferrero2707 commented Nov 10, 2017

@ClaesGitHub

This comment has been minimized.

Show comment
Hide comment
@ClaesGitHub

ClaesGitHub Nov 10, 2017

You're welcome :-)
Slower than the official Win builds? Sorry, cannot say, since I am rarely visiting my Win partitions these days. My feeling was based on a comparison with the Linux/Ubuntu build.

Hm... Il 10/nov/2017 19:52, "Claes" ha scritto???
Scusi, Dottore, io stupido, mi Italiano e solo poco, poco!
I once took a Linguaphone course in the Italian language,
so now I can say that the market square is 60 steps to
the left of the church, and other useful phrases!
Cordialemente,
Claes

ClaesGitHub commented Nov 10, 2017

You're welcome :-)
Slower than the official Win builds? Sorry, cannot say, since I am rarely visiting my Win partitions these days. My feeling was based on a comparison with the Linux/Ubuntu build.

Hm... Il 10/nov/2017 19:52, "Claes" ha scritto???
Scusi, Dottore, io stupido, mi Italiano e solo poco, poco!
I once took a Linguaphone course in the Italian language,
so now I can say that the market square is 60 steps to
the left of the church, and other useful phrases!
Cordialemente,
Claes

@gaaned92

This comment has been minimized.

Show comment
Hide comment
@gaaned92

gaaned92 Nov 12, 2017

@aferrero2707
I have some remarks on this work. Please take them in a positive manner.

The goal is to remove the need of human intervention in the tedious process of keeping up-to-date Windows packages from the development branches

On windows with MSYS2, you can completely automate the process with bash scripts, included uploading the builds. So it is not a concern.
My only problem is that I am not a bash wizzard, my scripts are rather clumsy

also guarantee the reproducibility of the builds by making sure that the same environment is used each time

Presently, the windows builds are reproducible are they are all done with the same Msys2 environment if we care to have an up to date Msys2 and use the same flags for building (this can be contrommed in the aboutthisbuild.txt). What can differ is the lensfun version used.
Unless you use also Msys2, you are introducing variability in what was uniform.
For instance I notice that you use an old version of GTK+. For windows builds, in order to have native windows, you must use version >=3.22.24

Moreover, the deployment does not rely on the availability of specific users and/or systems.
Another clear advantage is that the whole Windows build chain is public, and all developers can see it and do modifications if needed.

Agreed

The mechanism is based on a custom-crafted Docker container which provides a pre-installed Windows cross-compilation environment for GTK applications.

So we will have two environments : a Msys2 native environment and a Unix environment.
Th Unix environment doesn't seem up to date (see GTK+).
Until now there was a consensus in the dev team about windows building described in http://rawpedia.rawtherapee.com/Windows.
They will have to make a decision.

the rewtherapee.exe executable is big, because it is built in RelWithDebInfo mode

Yes you should build in release mode.
The windows package include now Rawtherapee.exe and rawtherapee-cli.exe in release mode, rawtherapee-debug.exe in debug mode and gdb.exe (see above link)
Moreover, it seems there are a lot of useless files, dlls and directories that bloat the zip. grab a windows build to see what is needed.

the lensfun database seems to be not correctly found, and automatic lens corrections are disabled - I am looking into this

As windows builds are bundled , we can hope that the lensfun version is the latest one from master branch built in the same way as rawtherapee. Msys2 lensfun being obsolete is not used for windows builds.

few UI icons are missing

see on existing windows build. At least adwaita icons are missing (we use only a few icons)

Slower than the official Win builds?

It seems so and I did not made exact measure. And comparison is not informative as official win builds use -O3 flag.

Other remarks

  • For development buids you should use 5-dev suffix for option file. option file is thus located in .../rawtherapee5-dev.
    for stable official builds suffix is blank, thus option file is located in .../rawtherapee

  • aboutthisbuild.txt: for some reason, as it is usual in unix distributions, version and branch are not what they should be and coherent with rawtherapee repository:
    branch: continuous-77-g7fb6be6 there is no such branch in rawtherapee git repository
    version : continuous-77-g7fb6be6 there is no such version in rawtherapee git repository

  • zip naming: rawtherapee-w64-20171110_1742-git-dev-7fb6be6cddca75f3d11b2e3a9cfd89b794204a35.zip
    name fields should be identical to corresponding fields in aboutthisbuild.txt. I notice here that branch name is correct. No need for complete commit hash, version field of aboutthisbuild.txt is sufficientand name will be shorter.

Otherwise, it runs ok.

gaaned92 commented Nov 12, 2017

@aferrero2707
I have some remarks on this work. Please take them in a positive manner.

The goal is to remove the need of human intervention in the tedious process of keeping up-to-date Windows packages from the development branches

On windows with MSYS2, you can completely automate the process with bash scripts, included uploading the builds. So it is not a concern.
My only problem is that I am not a bash wizzard, my scripts are rather clumsy

also guarantee the reproducibility of the builds by making sure that the same environment is used each time

Presently, the windows builds are reproducible are they are all done with the same Msys2 environment if we care to have an up to date Msys2 and use the same flags for building (this can be contrommed in the aboutthisbuild.txt). What can differ is the lensfun version used.
Unless you use also Msys2, you are introducing variability in what was uniform.
For instance I notice that you use an old version of GTK+. For windows builds, in order to have native windows, you must use version >=3.22.24

Moreover, the deployment does not rely on the availability of specific users and/or systems.
Another clear advantage is that the whole Windows build chain is public, and all developers can see it and do modifications if needed.

Agreed

The mechanism is based on a custom-crafted Docker container which provides a pre-installed Windows cross-compilation environment for GTK applications.

So we will have two environments : a Msys2 native environment and a Unix environment.
Th Unix environment doesn't seem up to date (see GTK+).
Until now there was a consensus in the dev team about windows building described in http://rawpedia.rawtherapee.com/Windows.
They will have to make a decision.

the rewtherapee.exe executable is big, because it is built in RelWithDebInfo mode

Yes you should build in release mode.
The windows package include now Rawtherapee.exe and rawtherapee-cli.exe in release mode, rawtherapee-debug.exe in debug mode and gdb.exe (see above link)
Moreover, it seems there are a lot of useless files, dlls and directories that bloat the zip. grab a windows build to see what is needed.

the lensfun database seems to be not correctly found, and automatic lens corrections are disabled - I am looking into this

As windows builds are bundled , we can hope that the lensfun version is the latest one from master branch built in the same way as rawtherapee. Msys2 lensfun being obsolete is not used for windows builds.

few UI icons are missing

see on existing windows build. At least adwaita icons are missing (we use only a few icons)

Slower than the official Win builds?

It seems so and I did not made exact measure. And comparison is not informative as official win builds use -O3 flag.

Other remarks

  • For development buids you should use 5-dev suffix for option file. option file is thus located in .../rawtherapee5-dev.
    for stable official builds suffix is blank, thus option file is located in .../rawtherapee

  • aboutthisbuild.txt: for some reason, as it is usual in unix distributions, version and branch are not what they should be and coherent with rawtherapee repository:
    branch: continuous-77-g7fb6be6 there is no such branch in rawtherapee git repository
    version : continuous-77-g7fb6be6 there is no such version in rawtherapee git repository

  • zip naming: rawtherapee-w64-20171110_1742-git-dev-7fb6be6cddca75f3d11b2e3a9cfd89b794204a35.zip
    name fields should be identical to corresponding fields in aboutthisbuild.txt. I notice here that branch name is correct. No need for complete commit hash, version field of aboutthisbuild.txt is sufficientand name will be shorter.

Otherwise, it runs ok.

@Beep6581

This comment has been minimized.

Show comment
Hide comment
@Beep6581

Beep6581 Nov 12, 2017

Owner

On windows with MSYS2, you can completely automate the process with bash scripts, included uploading the builds. So it is not a concern.

Well it is a concern as the only way to actually get the build is to depend on someone who has Windows to have that whole environment set up and to be available to create it. With this automated method, the build is created the moment a commit to a designated branch is made, without having to rely on people being available and not missing some step, having virus-free machines, timezones, etc.

Some dependency being of a wrong version is just something which needs to be fixed, it's not an argument against the whole idea.

@aferrero2707 I will be on IRC for the next few hours, if you're available we could go through the script.
http://rawpedia.rawtherapee.com/IRC

Owner

Beep6581 commented Nov 12, 2017

On windows with MSYS2, you can completely automate the process with bash scripts, included uploading the builds. So it is not a concern.

Well it is a concern as the only way to actually get the build is to depend on someone who has Windows to have that whole environment set up and to be available to create it. With this automated method, the build is created the moment a commit to a designated branch is made, without having to rely on people being available and not missing some step, having virus-free machines, timezones, etc.

Some dependency being of a wrong version is just something which needs to be fixed, it's not an argument against the whole idea.

@aferrero2707 I will be on IRC for the next few hours, if you're available we could go through the script.
http://rawpedia.rawtherapee.com/IRC

@gaaned92

This comment has been minimized.

Show comment
Hide comment
@gaaned92

gaaned92 Nov 12, 2017

@Beep6581

You miss the point. I never said I was against that. As@aferrero2707 asked for remarks, I made remarks hoping it will help.
So I will not make any other remark on this topic.
The decision is of course, up to the developpers.
The problem seems worst on the different Linux systems with people complaining with obsolete versions available in their distributions.

gaaned92 commented Nov 12, 2017

@Beep6581

You miss the point. I never said I was against that. As@aferrero2707 asked for remarks, I made remarks hoping it will help.
So I will not make any other remark on this topic.
The decision is of course, up to the developpers.
The problem seems worst on the different Linux systems with people complaining with obsolete versions available in their distributions.

@Beep6581

This comment has been minimized.

Show comment
Hide comment
@Beep6581

Beep6581 Nov 12, 2017

Owner

@gaaned92 I did not mean to upset you, apologies if I did.

I agree/confirm that many of the latest distributions lack recent versions of key packages (the easiest way to check is through https://distrowatch.com/ ), however that is not a problem here, as the Docker image can be made to use whatever we need. Hopefully.

Owner

Beep6581 commented Nov 12, 2017

@gaaned92 I did not mean to upset you, apologies if I did.

I agree/confirm that many of the latest distributions lack recent versions of key packages (the easiest way to check is through https://distrowatch.com/ ), however that is not a problem here, as the Docker image can be made to use whatever we need. Hopefully.

@gaaned92

This comment has been minimized.

Show comment
Hide comment
@gaaned92

gaaned92 Nov 12, 2017

@Beep6581
It's ok. I was only a little disoriented by your answer as I think it is a way to go if it can fulfill all dev's needs.
What you quoted is only an answer to the burden to build on windows which is really not.

gaaned92 commented Nov 12, 2017

@Beep6581
It's ok. I was only a little disoriented by your answer as I think it is a way to go if it can fulfill all dev's needs.
What you quoted is only an answer to the burden to build on windows which is really not.

@sguyader

This comment has been minimized.

Show comment
Hide comment
@sguyader

sguyader Nov 12, 2017

@gaaned92 it's not a burden, but it still requires human action at this point, if I'm not mistaken.

sguyader commented Nov 12, 2017

@gaaned92 it's not a burden, but it still requires human action at this point, if I'm not mistaken.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Nov 12, 2017

Collaborator

All remarks by @gaaned92 are perfectly legitimate, but most of them are technical aspects that are not particularly difficult to fix as part of the finalization process.

It is probably worth giving some technical details about the custom docker container I am using...
The container is based on Ubuntu 17.04 and the mingw-w64 compiler that comes with the distribution. The compilation is performed through a tool called crossroad. Apart from properly setting the compiler and toolchains for the cross-compilation, corssroad also takes care of the installation of the pre-compiled cross-packages (including an automated resolution of their dependencies) provided by OpenSUSE.
Those packages are handy for installing most of the RT dependencies, but nothing prevents from building from sources those which are not recent enough (like gtk3).

Nevertheless, I would prefer to first discuss the proof-of-principle before going into such details. However, some of the dependency issues are also valid for my own project which uses the same docker container, so I plan to work on those in any case...

Collaborator

aferrero2707 commented Nov 12, 2017

All remarks by @gaaned92 are perfectly legitimate, but most of them are technical aspects that are not particularly difficult to fix as part of the finalization process.

It is probably worth giving some technical details about the custom docker container I am using...
The container is based on Ubuntu 17.04 and the mingw-w64 compiler that comes with the distribution. The compilation is performed through a tool called crossroad. Apart from properly setting the compiler and toolchains for the cross-compilation, corssroad also takes care of the installation of the pre-compiled cross-packages (including an automated resolution of their dependencies) provided by OpenSUSE.
Those packages are handy for installing most of the RT dependencies, but nothing prevents from building from sources those which are not recent enough (like gtk3).

Nevertheless, I would prefer to first discuss the proof-of-principle before going into such details. However, some of the dependency issues are also valid for my own project which uses the same docker container, so I plan to work on those in any case...

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Nov 12, 2017

Collaborator

@gaaned92

Msys2 lensfun being obsolete is not used for windows builds.

The Lensfun version I am using is 0.3.1. I shall probably build the git master branch instead...

Collaborator

aferrero2707 commented Nov 12, 2017

@gaaned92

Msys2 lensfun being obsolete is not used for windows builds.

The Lensfun version I am using is 0.3.1. I shall probably build the git master branch instead...

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Nov 12, 2017

Collaborator

@Beep6581

I agree/confirm that many of the latest distributions lack recent versions of key packages

That's the mess I'd like to help you solve with a properly working AppImage ;-)

Collaborator

aferrero2707 commented Nov 12, 2017

@Beep6581

I agree/confirm that many of the latest distributions lack recent versions of key packages

That's the mess I'd like to help you solve with a properly working AppImage ;-)

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Nov 20, 2017

Collaborator

Some updates about this topic... I have produced a new package that incorporates most of the suggestions, namely:

  • the rawtherapee.exe executable is compiled in Release mode and with -O3 enabled
  • I have used the "5-dev" suffix for the option file
  • I have compiled and installed gtk+ 3.22.26 and gtkmm 3.22.2 from sources, as well as the latest Lensfun 0.3.2
  • I have include the Adwaita fonts
  • have cleaned a bit the bundle and shortened the .zip file name

The new package is here.

Collaborator

aferrero2707 commented Nov 20, 2017

Some updates about this topic... I have produced a new package that incorporates most of the suggestions, namely:

  • the rawtherapee.exe executable is compiled in Release mode and with -O3 enabled
  • I have used the "5-dev" suffix for the option file
  • I have compiled and installed gtk+ 3.22.26 and gtkmm 3.22.2 from sources, as well as the latest Lensfun 0.3.2
  • I have include the Adwaita fonts
  • have cleaned a bit the bundle and shortened the .zip file name

The new package is here.

@gaaned92

This comment has been minimized.

Show comment
Hide comment
@gaaned92

gaaned92 Nov 20, 2017

@aferrero2707
The exe and also RT data files seem missing.

gaaned92 commented Nov 20, 2017

@aferrero2707
The exe and also RT data files seem missing.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Nov 20, 2017

Collaborator

Oooops! My fault, due to the fact that I have no easy way to test the Win packages at the moment... a new build is under way.

Collaborator

aferrero2707 commented Nov 20, 2017

Oooops! My fault, due to the fact that I have no easy way to test the Win packages at the moment... a new build is under way.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Nov 20, 2017

Collaborator

The updated package, this time with the rawtherapee.exe executable and data files, is here.

EDIT: I did a quick test with wine, and the lensfun corrections seems to be available and applied properly. However, there might be still problems with some icons...

Collaborator

aferrero2707 commented Nov 20, 2017

The updated package, this time with the rawtherapee.exe executable and data files, is here.

EDIT: I did a quick test with wine, and the lensfun corrections seems to be available and applied properly. However, there might be still problems with some icons...

@Beep6581 Beep6581 self-assigned this Dec 27, 2017

@Beep6581

This comment has been minimized.

Show comment
Hide comment
@Beep6581

Beep6581 Feb 26, 2018

Owner

@aferrero2707 shall we close this issue and continue discussions in #4404?

Owner

Beep6581 commented Feb 26, 2018

@aferrero2707 shall we close this issue and continue discussions in #4404?

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Feb 26, 2018

Collaborator

@Beep6581 yes, sure!

Collaborator

aferrero2707 commented Feb 26, 2018

@Beep6581 yes, sure!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment