Add nightly and CI build scripts to create Windows installers #7

eht16 · 2022-02-06T23:01:31Z

The scripts, mainly start_build.sh, can be used for nightly builds
and for CI builds.

The Windows scripts use a Docker image containing a full cross
compilation environment for mingw64-x86_64. They create fully working
installer files for Geany and Geany-Plugins, optionally even signed
if a certificate is provided.

The Debian build scripts are yet to be tested and finalized.

Even though the Debian part is not yet ready, the Windows part is working fine and I plan to use it for the nightly builds soon.
The ultimative plan is to use these scripts in the CI builds for Geany and Geany-Plugins and to provide the created installer files as artifacts to download, so developers and users can easily test master or even PR builds.

kugel- · 2022-09-17T22:00:04Z

Wild!

Why is the mingw container not Arch Linux based such that pacman doesnt need to be compiled? I would also expect that other packages take less time to install.

Do I understand correctly that the Docker image is updated manually, and a PR on geany or G-P would download that pre-build image (or even use cached copies) instead of rolling the container every single time (including compiling pacman)?

Remark #1: Generally speaking I'm not a huge fan of docker but since that doesn't run on my machine it's probably OK :-)
Remark #2: I use arch linux on my machine. If you don't feel familiar enough with it I can maybe help you.

eht16 · 2022-09-18T19:46:54Z

Why is the mingw container not Arch Linux based such that pacman doesnt need to be compiled? I would also expect that other packages take less time to install.

Personal preference and the fact that most other CI resources are also Debian/Ubuntu based.
I don't know if it would make a difference in package installation speed. Why and does it matter if it is a few seconds faster or slower?
Furthermore, I'm not sure if ArchLinux's pre-compiled pacman can be used as is. The self-compiled variant uses a different path layout.

Do I understand correctly that the Docker image is updated manually, and a PR on geany or G-P would download that pre-build image (or even use cached copies) instead of rolling the container every single time (including compiling pacman)?

First, the current image available in the registry is built and uploaded by me manually.
As said in the G-P PR this is an open TODO.

A CI job will try to pull the image from the registry, due to current permissions settings this probably works only for jobs trigger from the repository itself (master builds and PRs created from the repository itself, not forks).

CI jobs which are triggered by a forked repository will not have access to the image in the registry and so fall back to build it on the fly.

In theory, we could push that on the fly built image into the registry. But I consider this too risky to open doors for malicious input.

I didn't make my mind up finally on how to deal with the proper image management:

Should we build it reguarly independent from the CI jobs?
Which repository handles updating the image, Geany or Geany-PLugins or Infrastructure?
Should we open the restrictions and allow access to the image also for forked repositories?

Remark #1: Generally speaking I'm not a huge fan of docker but since that doesn't run on my machine it's probably OK :-)

It took me a few years also to see its advantages and why it might be useful. I think in this it's a quite convenient tool to have a isolated environment to build the code.

Remark #2: I use arch linux on my machine. If you don't feel familiar enough with it I can maybe help you.

I used Arch Linux until a few months ago as well, but IMO this is no reason to use it here :). Again, I guess it won't make a relevant different except that we would have to port the existing scripts. I guess more people out there are already or can get familiar with a Debian/Ubuntu system.
Do you have any other reason for switching to ArchLinux?

kugel- · 2022-09-19T11:43:54Z

builders/Dockerfile.mingw64

+    # cleanup
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/* && \
+    yes | pacman -Scc && \


--noconfirm. yes doesn't terminate by itself which I is generally problematic IMO.

I guess you did not test what you suggest?
--noconfirm does not clean the caches as the default for the question "Do you want to remove ALL files from cache? [y/N]" is no.

In this context yes seems to work as intended. Did you experience a certain problem?

kugel- · 2022-09-19T11:48:38Z

builders/Dockerfile.mingw64

+# For more details, see build_mingw64_geany.sh where this image is used.
+#
+# Intermediate container for building pacman
+FROM debian:bullseye as build-pacman


Why is that an "intermediate container"? And what does this mean exactly, i.e. what's the advantage over building pacman in the main docker container?

The advantage is that everything necessary to build Pacman remains in the intermediate container and gets destroyed automatically once it is finished. Nothing from this container will remain in the final container except those items which you copy explicitly.
This is very common and convenient practice when building Docker images.
https://docs.docker.com/build/building/multi-stage/

good to know

kugel- · 2022-09-19T11:52:16Z

I just thought it's a better match because it comes with pacman, but also because it is generally up-to-date w.r.t. to toolchains and wine. So we could build the Windows binaries using the latest and greatest toolchain.

Debian/Ubuntu is usually several years behind and I worry a bit if that might cause trouble in the future if mingw packages require a newer toolchain (mingw is also rolling like Arch AFAIK). But it's your choice.

I think the current restrictions are OK. Perhaps we could auto-update the image based on CI but I wouldn't trigger that from Geany repos but this repository (this is where the dockerfile is after all). Do we have anything to lose when giving forks access to the image, like losing some GH actions budget to it? Anyway we can discuss that later and leave it to official repos for now.

kugel- · 2022-09-19T12:01:32Z

One more (personal) reason to use Arch: You know I'm working on a flatpak version of Geany. Assuming we ever want to create such a flatpak image for official releases via CI, then I would certainly prefer Arch Linux because that has the latest flatpak tooling.

But that doesn't necessarily mean mingw jobs must use Arch Linux containers as well, so still your choice.

kugel- · 2022-09-19T22:25:50Z

I don't really like that this repository knows how to build Geany and G-P.

A developer might make changes in that area and might become unable to proceed because he can't get CI to work as the job needs to be adapted.

A realistic example: we might want to switch to the meson build by default (or maybe only for specific platforms) and the PR to implement would be unable to fix the CI.

We should probably just provide the Docker image itself and not the scripts to build Geany or packages.

eht16 · 2022-09-20T20:19:48Z

I just thought it's a better match because it comes with pacman, but also because it is generally up-to-date w.r.t. to toolchains and wine. So we could build the Windows binaries using the latest and greatest toolchain.

Debian/Ubuntu is usually several years behind and I worry a bit if that might cause trouble in the future if mingw packages require a newer toolchain (mingw is also rolling like Arch AFAIK). But it's your choice.

One more (personal) reason to use Arch: You know I'm working on a flatpak version of Geany. Assuming we ever want to create such a flatpak image for official releases via CI, then I would certainly prefer Arch Linux because that has the latest flatpak tooling.

But that doesn't necessarily mean mingw jobs must use Arch Linux containers as well, so still your choice.

I was working on these scripts since 2021 or maybe even 2020 and had no problems with the toolchains provided by Debian, AFAIR.
Of course, I cannot guarantee that it will keep it this way. But as previous long time ArchLinux user I also know sometimes ArchLinux is so bleeding edge that its toolchain or other components might be too new for other software which is not yet compatible.
So I guess this can happen in both directions.

And please stop this too old joke of Debian being "several years" behind. It's usually rather 1-2 years and this is on purpose.

After all, I guess Debian and ArchLinux could work as base and given that the current setup works with Debian, I personally see no need to change it.

I don't really like that this repository knows how to build Geany and G-P.

A developer might make changes in that area and might become unable to proceed because he can't get CI to work as the job needs to be adapted.

A realistic example: we might want to switch to the meson build by default (or maybe only for specific platforms) and the PR to implement would be unable to fix the CI.

We should probably just provide the Docker image itself and not the scripts to build Geany or packages.

Agreed. We could move builders/scripts/build_mingw64_geany_plugins.sh and its companions to the Geany resp. G-P repos.
Will try once I find time for it.

I guess I did it this way because the initial idea of all this started with renewing the nightly build scripts and over the time it turned out to be usable for CI builds as well.

I think the current restrictions are OK. Perhaps we could auto-update the image based on CI but I wouldn't trigger that from Geany repos but this repository (this is where the dockerfile is after all).

Sounds good to me.

Do we have anything to lose when giving forks access to the image, like losing some GH actions budget to it?

Not that I know of. To me it seems that GH actions are not billed at all for us and so there is also no budget or limit.
Giving read access to forks should be OK, even if it would be read access to the world. We should note that images cannot be deleted anymore once they have been made public which is reasonable and probably also acceptable for a automatically built CI image.

eht16 · 2022-10-16T14:49:28Z

I don't really like that this repository knows how to build Geany and G-P.
A developer might make changes in that area and might become unable to proceed because he can't get CI to work as the job needs to be adapted.
A realistic example: we might want to switch to the meson build by default (or maybe only for specific platforms) and the PR to implement would be unable to fix the CI.
We should probably just provide the Docker image itself and not the scripts to build Geany or packages.

Agreed. We could move builders/scripts/build_mingw64_geany_plugins.sh and its companions to the Geany resp. G-P repos. Will try once I find time for it.

Done.
The scripts to build Geany and Geany-Plugins are now in their respective repositories.
This required to create geany/geany#3315 but it was on my TODO anyway.

kugel- · 2022-10-21T20:43:05Z

builders/scripts/build_debian_geany.sh

+	log "Create source tarball"
+	cd ${GEANY_BUILD_DIR}
+	# create a source tarball, keep .git included on purpose for Geany GIT build detection
+	tar --transform "s,^,geany-plugins-${GEANY_VERSION}/,S" \


geany-plugins-${GEANY_VERSION} ?

Fixed, thanks.

kugel- · 2022-10-21T20:45:28Z

builders/Dockerfile.mingw64

+# For more details, see build_mingw64_geany.sh where this image is used.
+#
+# Intermediate container for building pacman
+FROM debian:bullseye as build-pacman


good to know

kugel- · 2022-10-21T20:50:24Z

builders/Dockerfile.mingw64

+    tar xf pacman-${PACMAN_VERSION}.tar.xz && \
+    cd /pacman-${PACMAN_VERSION} && \
+    meson \
+        --prefix /usr/local/pacman \


Why not set prefix to /usr/local in the first place? That would avoid having to create symlinks and potential binary-relocation issues in the future.

This is a fresh system container, /usr/local should contain nothing but the pacman installation afterwards.

I prefer to install applications into their own prefix if possible. But /usr/local should work as well.
Since the current setup works this way, I'd leave as is.

What do you mean by "potential binary-relocation issues in the future"?

I don't understand. If you prefer the app-exclusive prefix, then why do you copy it to /usr/local/bin, afterwards?

It appears that pacman is run from /usr/local/bin/pacman, afterwards, yet its prefix is /usr/local/pacman. This can cause trouble if the pacman binary encodes the prefix (remember that Geany isn't relocatable by default either).

Really, I would prefer to define one prefix and then also run the program from there. Anything else is asking for trouble and is just adding confusion (I am confused). Then you don't need ln -s and LD_LIBRARY_PATH workarounds that I find in this file.

Whateveryou wish.
Pacman is now installed into /usr/local but setting LD_LIBRARY_PATH is still necessary or alternatively running ldconfig to manually update the ldcache.
This is because after installing Pacman with Meson, the ldcache is not updated automatically. So we still need to take care to update the ldcache.

Can you run ldconfig in the container?

Yes, tried it in the first place. We would need to run it in the build container as well as in the final container (as we copy only the files in /usr/local from the one to the other but no state and cache).
I don't mind whether we set LD_LIBRARY_PATH as it is or run ldconfig.

My feeling: better update the cache properly than messing with LD_LIBRARY_PATH but I don't feel strong either.

kugel- · 2022-10-21T20:51:19Z

builders/Dockerfile.mingw64

+
+# start wine to initially create config directory
+RUN /usr/local/bin/mingw-w64-i686-wine hostname.exe && \
+    /usr/local/bin/mingw-w64-x86_64-wine hostname.exe && \


Why do you need to run hostname.exe twice?

Once for wine-i686 and once for wine-x86_64. Both commands are only executed to setup the Wind environment initially for both architectures.
We need wine-x86_64 to run Geany itself and wine-i686 to execute the installer which can be built only for i686.

kugel- · 2022-10-21T20:53:07Z

builders/Dockerfile.mingw64

+    # install NSIS and exiftool to inspect binary metadata
+    nsis libimage-exiftool-perl osslsigncode \
+    # Geany build dependencies \
+    python3-lxml python3-docutils


I'm still uneasy that Geany dependencies are encoded in the infrastructure script. What happens if a PR in Geany adds new dependencies? How can that PR possibly proceed, without breaking CI?

See https://github.com/geany/geany-plugins/pull/1201/files#diff-306a1de5460f6947d8ee9383b366b8dcc6529fa5c4b18aa9523729d8456e3d0dR175 - this is the canonical place for installing necessary dependencies.

The list here is rather some sort of cache or pre-installation of what is done in each CI run to speed up the build process. If at CI job execution time newer package versions are available than at build time of the Docker image, the newer versions are used at CI job execution.

Okay. I only looked at the geany build script and didn't see how it installs additional dependencies. This was my concern, that the infrastructure script must know about all dependencies. But that's not a problem then, great!

kugel- · 2022-10-21T20:55:58Z

builders/scripts/build_debian_geany.sh

+		"gtk_version": "${GTK_VERSION}",
+		"gcc_version": "${COMPILER_VERSION}",
+	}
+EOT


Who uses versions.json?

Nothing yet but it might become useful when using the scripts for nightly builds to fill the table on https://www.geany.org/download/nightly-builds/.

But I'm quite undecided yet how to proceed with the nightly builds exactly and the whole Debian package builds scripts are yet to be finalized. For now, I would like to focus on the mingw64 parts to get them integrated into the pipeline.

I would add such artifacts only when they became needed with a clear use case.

They are gone with the whole Debian code.

kugel- · 2022-10-21T20:58:19Z

builders/scripts/build_debian_geany_plugins.sh

+}
+
+
+main


Lots of this file is identical to build_debian_geany.sh. Don't you think it would be worthwhile to share code instead of duplication?

Yeah, I was struggling with this as well.
However, consequently we should move the Debian build scripts to the Geany and G-P repositories as we did for the mingw64 scripts but then sharing the code between two repositories gets more complicated again.

Above you said you would like to focus on the mingw-parts. Might be a good idea to leave the Debian parts out of this PR for now, then, and deal with that in a separate, follow-up PR. Then we can discuss the proper place.

Ok, I removed the Debian parts for now.

kugel- · 2022-10-21T21:32:06Z

builders/scripts/update_debian_repositories.sh

+
+
+# codename -> suite mapping
+declare -A SUITES=([buster]=stable [sid]=unstable)


buster -> bullseye

Updated, thanks.

eht16 · 2022-10-23T12:10:36Z

I think the current restrictions are OK. Perhaps we could auto-update the image based on CI but I wouldn't trigger that from Geany repos but this repository (this is where the dockerfile is after all).

Sounds good to me.

I added a CI workflow on this repository to build the Docker image on push events and on a weekly schedule.
For now, the image is configured as private and so it can be used only from within the organization. In particular this means that pull requests for Geany or G-P which are created from a forked clone, cannot access the Docker image. For this, we need to make it public.
This should be ok but I would wait a bit until the new process works as expected. Once an image is made public, it cannot be deleted anymore.

kugel- · 2022-10-23T21:02:25Z

builders/Dockerfile.mingw64

+    tar xf pacman-${PACMAN_VERSION}.tar.xz && \
+    cd /pacman-${PACMAN_VERSION} && \
+    meson \
+        --prefix /usr/local/pacman \


I don't understand. If you prefer the app-exclusive prefix, then why do you copy it to /usr/local/bin, afterwards?

It appears that pacman is run from /usr/local/bin/pacman, afterwards, yet its prefix is /usr/local/pacman. This can cause trouble if the pacman binary encodes the prefix (remember that Geany isn't relocatable by default either).

Really, I would prefer to define one prefix and then also run the program from there. Anything else is asking for trouble and is just adding confusion (I am confused). Then you don't need ln -s and LD_LIBRARY_PATH workarounds that I find in this file.

kugel- · 2022-10-23T21:06:26Z

builders/Dockerfile.mingw64

+    # install NSIS and exiftool to inspect binary metadata
+    nsis libimage-exiftool-perl osslsigncode \
+    # Geany build dependencies \
+    python3-lxml python3-docutils


Okay. I only looked at the geany build script and didn't see how it installs additional dependencies. This was my concern, that the infrastructure script must know about all dependencies. But that's not a problem then, great!

kugel- · 2022-10-23T21:10:17Z

builders/scripts/build_debian_geany.sh

+		"gtk_version": "${GTK_VERSION}",
+		"gcc_version": "${COMPILER_VERSION}",
+	}
+EOT


I would add such artifacts only when they became needed with a clear use case.

kugel- · 2022-10-23T21:13:40Z

builders/scripts/build_debian_geany_plugins.sh

+}
+
+
+main


Above you said you would like to focus on the mingw-parts. Might be a good idea to leave the Debian parts out of this PR for now, then, and deal with that in a separate, follow-up PR. Then we can discuss the proper place.

kugel-

Awesome

eht16 · 2022-10-27T22:45:05Z

@kugel- are you fine when I squash the commits together or do you still need the history?

kugel- · 2022-10-28T05:01:52Z

Squashing is fine

kugel- · 2022-12-02T07:47:14Z

Go for it? Or is anything missing?

The scripts, mainly start_build.sh, can be used for nightly builds and for CI builds. The Windows scripts use a Docker image containing a full cross compilation environment for mingw64-x86_64. They create fully working installer files for Geany and Geany-Plugins, optionally even signed if a certificate is provided. The Debian build scripts are yet to be tested and finalized.

eht16 · 2022-12-04T17:40:25Z

I guess I was waiting for a final "ok" and then forgot about it.

Thanks for the review. I fixed a final typo and now it's going to be merged. Yay.

eht16 marked this pull request as draft February 23, 2022 20:28

eht16 force-pushed the add_ci_builders branch from f01890e to ef3cb32 Compare February 23, 2022 20:45

eht16 force-pushed the add_ci_builders branch from 6362b27 to 18dc699 Compare March 13, 2022 15:54

eht16 force-pushed the add_ci_builders branch 2 times, most recently from ee8ff28 to ebd3165 Compare September 17, 2022 12:10

This was referenced Sep 17, 2022

Add mingw-w64 (Windows) CI build geany/geany-plugins#1201

Merged

Use gtk_show_uri_on_window() in utils_open_browser() by default geany/geany#3178

Merged

eht16 force-pushed the add_ci_builders branch 2 times, most recently from 897982f to 0396ed3 Compare September 17, 2022 20:41

kugel- reviewed Sep 19, 2022

View reviewed changes

eht16 mentioned this pull request Oct 16, 2022

CI: Use MSYS2 for Windows build geany/geany#3315

Merged

2 tasks

eht16 marked this pull request as ready for review October 17, 2022 21:28

eht16 force-pushed the add_ci_builders branch from 6a3b282 to d434dc4 Compare October 17, 2022 22:04

kugel- suggested changes Oct 21, 2022

View reviewed changes

kugel- reviewed Oct 21, 2022

View reviewed changes

kugel- suggested changes Oct 23, 2022

View reviewed changes

kugel- approved these changes Oct 27, 2022

View reviewed changes

eht16 force-pushed the add_ci_builders branch from 2d9af0c to 3859e2c Compare October 29, 2022 14:55

eht16 force-pushed the add_ci_builders branch from 3859e2c to e25db90 Compare December 4, 2022 17:39

eht16 merged commit 19139f3 into master Dec 4, 2022

eht16 deleted the add_ci_builders branch December 4, 2022 17:41



		# codename -> suite mapping
		declare -A SUITES=([buster]=stable [sid]=unstable)

Add nightly and CI build scripts to create Windows installers #7

Add nightly and CI build scripts to create Windows installers #7

Conversation

eht16 commented Feb 6, 2022 • edited

kugel- commented Sep 17, 2022

eht16 commented Sep 18, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kugel- commented Sep 19, 2022

kugel- commented Sep 19, 2022

kugel- commented Sep 19, 2022

eht16 commented Sep 20, 2022

eht16 commented Oct 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eht16 commented Oct 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kugel- left a comment

Choose a reason for hiding this comment

eht16 commented Oct 27, 2022

kugel- commented Oct 28, 2022

kugel- commented Dec 2, 2022

eht16 commented Dec 4, 2022

eht16 commented Feb 6, 2022 •

edited

eht16 commented Sep 18, 2022 •

edited