Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS CI pipeline - automated builds #5550

Open
Krijger opened this issue Dec 2, 2019 · 26 comments
Open

macOS CI pipeline - automated builds #5550

Krijger opened this issue Dec 2, 2019 · 26 comments
Milestone

Comments

@Krijger
Copy link
Contributor

@Krijger Krijger commented Dec 2, 2019

To support macOS development, this issue is to implement a Continuous Integration pipeline. This will allow us to create macOS builds for basically each commit, and to share these with the entire comunity, making testing easier.

The following needs to be done:

  • create a pipeline using GitHub Actions (service of choice for now)
  • output a bootable artifact
  • codesign and notarize the app and DMG
  • artifact are ZIP files directly, as opposed to a ZIP within a ZIP
  • create a release flow for git tags - this will make the artifact appear in releases, and not only via github actions, which incidentally gets removed after 90 days

The following is nice to have

  • make the scripts in the build fail fast (set -euo pipefail), and working
  • include build succeeding/failing widget on the readme page

--------- ORIGINAL ISSUE POST BELOW ---------

Since there are no nightly builds for Mac OS, and I see on the website that the last build had issues, I guess a pipeline that builds for Mac would be quite useful. Have you considered this?

I can start work on this, given that both Travis CI and Azure DevOps provide Mac build agents and are completely free for open source projects. I would opt for Azure in this case, since I can use that experience. But first of all, let me ask you how the current pipeline is setup. Is there continuous integration on this project?

@Beep6581 Beep6581 added the compilation label Dec 2, 2019
@Beep6581

This comment has been minimized.

Copy link
Owner

@Beep6581 Beep6581 commented Dec 2, 2019

@Krijger that would be nice, please talk with @aferrero2707 as he is working on that as well.

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 2, 2019

Sure. Shall we keep the communication in this issue?

Any answers to the other questions though? Where do the linux nightly builds come from? And are there unit tests being run on commit anywhere?

@heckflosse

This comment has been minimized.

Copy link
Collaborator

@heckflosse heckflosse commented Dec 2, 2019

@Krijger

Thanks for you offer to help. Very much appreciated 👍

Sure. Shall we keep the communication in this issue?

Yes, please.

Where do the linux nightly builds come from?

From @aferrero2707

And are there unit tests being run on commit anywhere?

No, the only thing running automatically on any pr commit is LGTM

@Beep6581

This comment has been minimized.

Copy link
Owner

@Beep6581 Beep6581 commented Dec 2, 2019

For info on automated AppImage and Windows builds, see #5551

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 3, 2019

@heckflosse just to be sure, the builds are executed locally on machines using the scripts mentioned in #5551 ? Because I’ve also checked the repos of aferro, but the latest activity is quite some time ago on those.

If that is the case, and the binaries are uploaded manually to the /Downloads page, we have a green field here to set these things up automatically.

EDIT. O wait, I see a Travis script in the two rt- repos of @aferrero2707 . But I’m confused as to why that is not in the official RawTherapee repo. Still attempting to grasp the flow here.

@aferrero2707

This comment has been minimized.

Copy link
Collaborator

@aferrero2707 aferrero2707 commented Dec 3, 2019

@Krijger no, the scripts are used as input for Travis CI jobs that are run on a daily basis. The jobs will fetch the current git head of the target branch, and compare the hash with the one of the latest successfully built package. If they differ, a new build is started. The packages are then automatically uploaded back to the nightly GitHub release.

Concerning the reason why the scripts are not in the RT repo, this is mainly to make sure that the same scripts are used for all the branches. If the scripts were part of the repo, then maintaining the scripts across multiple branches would be quite painful. However, we could move the scripts into https://github.com/Beep6581 for consistency...

@Beep6581 Beep6581 changed the title Nightly builds for Mac OS; CI pipeline macOS CI pipeline - automated builds Dec 3, 2019
@Krijger Krijger mentioned this issue Dec 8, 2019
@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 8, 2019

On that Pull Request:

This is a first step to create CI for Mac OS. I strongly believe having CI in place and allowing all supporting operating systems feel like first citizens is important for the user base of an open source project. Now all beta mac users have to build themselves which we can also channel into improving this build for all to use.

Included for now is the build on Github Actions (free for open source). The result is that by navigating to Github Actions you are now able to download the zipped artifact. The resulting DMG for now crashes however, probably due to the patch on libiconv.2.dylib not having been implemented yet. So, maybe you may still want to accept this PR on a branch instead, however, it does not interfere with anything, so dev is safe.

Things to be done

  • fix the DMG to something that runs (probably the patch)
  • change the bundle script to not create a zip, because github actions already creates a zip, so now we have a zip in a zip
  • support releases (which will make the artifact appear in releases, and not only via github actions, which incidentally gets removed after 90 days)
  • integrate App Developer ID (in case of the official repo only)

We can also think about moving Travis CI builds into here as well later on, to have all CI and releases automated in the same way.

Please note that while I know a lot about CI/CD, my programming skills are not in the c++ area. This is also my first time building anything for Mac OS, so I do need to ask for help.

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 8, 2019

Concerning the reason why the scripts are not in the RT repo, this is mainly to make sure that the same scripts are used for all the branches. If the scripts were part of the repo, then maintaining the scripts across multiple branches would be quite painful. However, we could move the scripts into https://github.com/Beep6581 for consistency...

I'm not seeing the pain. In all my daily work projects I have scripts in the repo itself, and for branches they automatically work as well. Which in the case of RT means that forks and PRs are also automatically built, on the forkers account. In my PR you will notice that the branch is actually parameterized by using the GITHUB_REF variable. Are there more compexities that we should take into account?

Note that you can also use other triggers, like tagging, on which you can conditionally initiate the release. Or you can make a 'stable' branch or something - the possibilities are many.

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 8, 2019

I fear I need help. I performed the patch as well (for libiconv version 1.16) in the build, but the application still crashes on startup. I will attach the crash dump.

crash-on-startup.log

@stdbull

This comment has been minimized.

Copy link

@stdbull stdbull commented Dec 9, 2019

You crash looks very much like mine with the assert in the same place. What happens if you run the app from the terminal? Are you building on Catalina or older? Homebrew or MacPorts?

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 9, 2019

I haven't tried executing it from the terminal. I can't verify that quickly however.
Catalina is used in the build agent, as is homebrew (also check the logs of the build file https://github.com/Krijger/RawTherapee/blob/ci-macos/.github/workflows/main.yml and logs https://github.com/Krijger/RawTherapee/actions, they are quite insightful, and hopefully you can reuse them)

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 9, 2019

@stdbull

This comment has been minimized.

Copy link

@stdbull stdbull commented Dec 9, 2019

Hi. Thanks for that. Your latest build log on push of 8f27f4b shows exactly the same errors when making the bundle as mine regarding gtk, and not being able to find files, etc. I suspect your crash has the same root cause (and same missing files as my image). It would be nice to be able to confirm that. BTW I already have -L/usr/local/libin my flags. FWIW here is my entire configure and build script:

export CC=/usr/local/opt/llvm/bin/clang
export CXX=${CC}++
export PKG_CONFIG_PATH=/usr/local/opt/libffi/lib/pkgconfig:/usr/local/opt/expat/lib/pkgconfig &&
cmake .. -DCMAKE_C_COMPILER="$CC"
-DCMAKE_CXX_COMPILER="$CXX"
-DPROC_TARGET_NUMBER="2"
-DCACHE_NAME_SUFFIX="5.7-dev"
-DWITH_LTO="ON"
-DLENSFUNDBDIR="./share/lensfun"
-DCMAKE_BUILD_TYPE=Release
-DOpenMP_C_FLAGS=-fopenmp=libomp
-DOpenMP_CXX_FLAGS=-fopenmp=libomp
-DOpenMP_C_LIB_NAMES="libomp"
-DOpenMP_CXX_LIB_NAMES="libomp"
-DOpenMP_libomp_LIBRARY="/usr/local/lib/libomp.dylib"
-DOpenMP_CXX_FLAGS="-Xpreprocessor -fopenmp /usr/local/lib/libomp.dylib -I/usr/local/include"
-DOpenMP_CXX_LIB_NAMES="libomp"
-DOpenMP_C_FLAGS="-Xpreprocessor -fopenmp /usr/local/lib/libomp.dylib -I/usr/local/include"
-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON
-DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/opt/libffi/lib -L/usr/local/lib"
-DCMAKE_AR="/usr/local/Cellar/llvm/9.0.0_1/bin/llvm-ar"
-DCMAKE_RANLIB="/usr/local/Cellar/llvm/9.0.0_1/bin/llvm-ranlib" &&
make -j 4 install &&
sudo make macosx_bundle

@stdbull

This comment has been minimized.

Copy link

@stdbull stdbull commented Dec 10, 2019

@Krijger Did you manage to run the app in a terminal and get a trace?
I've attached two file listings taken after application install - one from the prebuilt image from the RT website, and one from my own build using the config above. The differences are interesting.

RT-mybuild-file-list.txt
RT-prebuilt-file-list.txt

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 10, 2019

I've spent a lot of time trying to solve all compiling errors and warnings, for which I needed to do a lot in the macosx_bundle.sh script. I've been able to fix a lot that mainly had to do with homebrew locations (though not all) but we're not there yet.

Fixing some more issues now, but the most important thing is that I can't seem to find the mime database on the system. Running a debug statement on trying to find that now.

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 10, 2019

@Krijger Did you manage to run the app in a terminal and get a trace?
I've attached two file listings taken after application install - one from the prebuilt image from the RT website, and one from my own build using the config above. The differences are interesting.

RT-mybuild-file-list.txt
RT-prebuilt-file-list.txt

Good idea - that's something I also can do. I've fixed a lot of the issues already though. Check out my commits :)

@stdbull

This comment has been minimized.

Copy link

@stdbull stdbull commented Dec 10, 2019

Sounds promising. Where are you commits? RT repo or elsewhere? Sorry - not familiar with how you work.
One of the problems I've seen is what GTK_PREFIX is set to (set in the env when the bundle script is invoked). I don't know where that gets set, but it is for sure wrong for my system and results in loads of things not being able to be found. I quickly tried to set it to /usr/local bit that hack didn't work (not too surprised by that)...

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 10, 2019

You can see the commits in the PR #5557. Note that if you want to see the build output (in Github Actions) you need to be within my fork of the project, since that exists within my organisation (Krijger), basically you can just change Beep6581 with Krijger in the URL.

I also do not know yet where GTK_PREFIX comes from, but I've fixed how it is used in the build at least.

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 11, 2019

Good news. The application is building and opening! See the differences in package content here:
ci-macos-app-contents.txt
original-app-contents.txt

Now, there are some differences in lib files in Frameworks/, and I don't know where these come from. An exception is the libjpeg.9.dylib instead of libjpeg.62.dylib, which is because the gtk+3 (homebrew) depends on 9 and not 62. @Beep6581 , @aferrero2707 I think a review is becoming interesting now. Also, the app can be tested: https://github.com/Krijger/RawTherapee/suites/353696780/artifacts/653809

For the review: note a lot of differences in the macos bundle script. Basically, these have to do with differences between macport and homebrew I guess. I propose however to all work using the CI so that we don't need different versions maintained.

For comparison: the output op the current released app version on my system is

➜  ~ /Applications/RawTherapee.app/Contents/MacOS/rawtherapee ; exit;

(process:79711): Gtk-WARNING **: 18:48:01.288: Locale not supported by C library.
	Using the fallback 'C' locale.

(rawtherapee-bin:79711): GLib-GObject-WARNING **: 18:48:01.780: invalid cast from 'GtkMenuBar' to 'GtkWindow'

(rawtherapee-bin:79711): Gtk-WARNING **: 18:48:05.660: Could not load a pixbuf from /org/gtk/libgtk/theme/Adwaita/assets/check-symbolic.svg.
This may indicate that pixbuf loaders or the mime database could not be found.

(rawtherapee-bin:79711): Gtk-WARNING **: 18:48:10.144: Could not find the icon 'missing-image-ltr'. The 'hicolor' theme
was not found either, perhaps you need to install it.
You can get a copy from:
	http://icon-theme.freedesktop.org/releases

[Process completed]

while the new version logs

➜  ~ /Applications/RawTherapee_OSX_10.9_64_5.7-354-g907f2c74b.app/Contents/MacOS/rawtherapee ; exit;
Warning: Failed to set locale category LC_NUMERIC to en_NL.UTF-8.
Warning: Failed to set locale category LC_TIME to en_NL.UTF-8.
Warning: Failed to set locale category LC_COLLATE to en_NL.UTF-8.
Warning: Failed to set locale category LC_MONETARY to en_NL.UTF-8.
Warning: Failed to set locale category LC_MESSAGES to en_NL.UTF-8.

(rawtherapee-bin:83580): GLib-GObject-WARNING **: 19:05:00.647: invalid cast from 'GtkMenuBar' to 'GtkWindow'

(rawtherapee-bin:83580): Gtk-CRITICAL **: 19:05:00.647: gtk_window_add_accel_group: assertion 'GTK_IS_WINDOW (window)' failed

(rawtherapee-bin:83580): Gtk-WARNING **: 19:05:04.642: Could not load a pixbuf from /org/gtk/libgtk/theme/Adwaita/assets/check-symbolic.svg.
This may indicate that pixbuf loaders or the mime database could not be found.

(rawtherapee-bin:83580): Gtk-WARNING **: 19:05:06.297: Could not find the icon 'pan-down-symbolic-ltr'. The 'hicolor' theme
was not found either, perhaps you need to install it.
You can get a copy from:
	http://icon-theme.freedesktop.org/releases
^[[1;2C
[Process completed]

Most important is the CRITICAL gtk_window_add_accel_group: assertion 'GTK_IS_WINDOW (window)' failed, of which I don't know where it comes from. Also, what is up with the locales warnings? I don't know :)

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 11, 2019

Of course, the review is about #5557 (where actually all of this discussion belongs...)

@Beep6581 Beep6581 added this to the v5.8 milestone Dec 12, 2019
@Beep6581

This comment has been minimized.

Copy link
Owner

@Beep6581 Beep6581 commented Dec 12, 2019

Would be nice to have a macOS build of 5.8...

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 12, 2019

Agreed - where do we track the minimum requirements for that? A task list in this issue?

@stdbull

This comment has been minimized.

Copy link

@stdbull stdbull commented Dec 12, 2019

@Krijger Thanks a lot for putting the work in here. The app boots, but we still have the issues discussed in #5552

@Beep6581

This comment has been minimized.

Copy link
Owner

@Beep6581 Beep6581 commented Dec 13, 2019

@Krijger here's as good a place as any I suppose.

@Krijger

This comment has been minimized.

Copy link
Contributor Author

@Krijger Krijger commented Dec 13, 2019

@Beep6581 I put some point here in the top issue post. Please see if you agree or have additions.

@Beep6581

This comment has been minimized.

Copy link
Owner

@Beep6581 Beep6581 commented Dec 17, 2019

PR #5557 merged.

@Beep6581 Beep6581 modified the milestones: v5.8, v5.9 Jan 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.