CPack: Refactor AppImage and Apple DMG Generation #7252

tresf · 2024-05-12T01:17:21Z

Features:

Simplifies Mac and Linux packaging command to simply: make package

- make install && make appimage
- make install && make dmg
+ make package
# -- AppImage create: /__w/lmms/lmms/build/lmms-1.3.0-alpha.1.638+pr7252.gaaa9b4816-linux-x86_64.AppImage

Adds new WANT_DEBUG_CPACK flag to easily show detailed console messages about packaging

cmake .. -DWANT_DEBUG_CPACK=true

  Developer options
  -----------------------------------------
  * Debug FP exceptions               : Disabled
  * Debug using AddressSanitizer      : Disabled
  * Debug using ThreadSanitizer       : Disabled
  * Debug using MemorySanitizer       : Disabled
  * Debug using UBSanitizer           : Disabled
+ * Debug packaging commands          : Enabled

Preserves old CPack TGZ behavior through a hidden flag, WANT_CPACK_TARBALL. Does anyone use this?

TODO:

Investigate possibly using the provided Mac DMG generators.
- Port the install_apple.sh script to CMake.
  It's called MacDeployQt.cmake now.
Investigate possibility using CPack for creating Linux AppImages
- Port the package_linux.sh script to Cmake
  It's called LinuxDeploy[Qt].cmake now.
Upgrade CMake for Linux runners

BUGS:

Fix background missing for DMG
Fix AppImages not building for Linux
WANT_DEBUG_CPACK is incorrectly toggled in CI. https://github.com/LMMS/lmms/actions/runs/9131345271/job/25110338288. Closed via 82a3aa6.

STRETCH GOALS:

Port project from linuxdeployqt to linuxdeploy
Examples:
- pbek/QOwnNotes (thank you!)
- wireshark/wireshark@34ce99f (thank you!)
Remove all references to linuxdeployqt (in favor of linuxdeploy)
Add ARM64 AppImage
- Can linuxdeploy support a cross-compilation environment ? linuxdeploy/linuxdeploy#258

Background:

Click to expand

The AppImage and Apple DMG installers historically (confusingly) require make install to be run prior to making packages. This PR is a proposal to remove that by switching to CPack.
- Utilizes make package now, just like Windows does
- Mac utilizes CPACK_GENERATOR of ~~DragNDrop~~ Bundle which will create a DMG without the need for node's appdmg package (still requires macdeployqt)
- Linux will utilize CPACK_GENERATOR of "External", which will use custom CMake commands to replace package_linux.sh (still requires linuxdeployqt).

tresf · 2024-05-12T04:37:58Z

Rule out possibly using the provided Mac DMG generators.

I was able to get the CPack DMG generator to work with macdeployqt however in order for us to brand our own DMG, we either need to leverage AppleScript (which is much slower and requires user interaction over appdmg), or we need to provide a premade .DS_Store file to bundle.

Edit: For simplicity, I've checked in the .DS_Store from our current DMG. Since it will be unobvious how to update this file in the event of a rebranding, I've kept basic support for appdmg for recreating it, although it's not used or needed unless we rebrand the installer.

tresf · 2024-05-14T03:35:15Z

Hmm... the AppImage is missing... investigating...

tresf · 2024-05-14T13:41:09Z

The AppImage is missing because the CPack feature for running custom scripts through variable CPACK_PRE_BUILD_SCRIPTS was not available in Ubuntu 20.04. There doesn't seem to be a replacement for this feature, which is alarming, I wonder how people used "External" packaging techniques prior to CMake 3.19. Hmm...

midi-pascal · 2024-05-14T14:03:53Z

Perhaps I have a hint for this:

I use Ubuntu 20.04 that comes with cmake version 3.16 BUT Qt (5.15.2 installed from Qt site) comes with cmake 3.27 in its tools directory. This is the one I use.

tresf · 2024-05-14T20:36:01Z

I use Ubuntu 20.04 that comes with cmake version 3.16 BUT Qt (5.15.2 installed from Qt site) comes with cmake 3.27 in its tools directory. This is the one I use.

Thanks! Any recommendation for getting this into a CI (command line?)

midi-pascal · 2024-05-14T21:06:31Z

Not sure of what I will suggest below since I build using Qt Creator(so all is set automagically) but there are environment variables related to cmake:

CMAKE_PREFIX_PATH

CMAKE_PREFIX_PATH
Semicolon-separated list of directories specifying installation
prefixes to be searched by the find_package(),
find_program(), find_library(), find_file(), and
find_path() commands. Each command will add appropriate
subdirectories (like bin, lib, or include) as specified in its own
documentation.
By default this is empty. It is intended to be set by the project.
There is also an environment variable CMAKE_PREFIX_PATH, which is used
as an additional list of search prefixes.
See also CMAKE_SYSTEM_PREFIX_PATH, CMAKE_INCLUDE_PATH,
CMAKE_LIBRARY_PATH, CMAKE_PROGRAM_PATH, and
CMAKE_IGNORE_PATH.

Hope this can help :-)

PhysSong · 2024-05-15T01:37:00Z

FYI, CMake homepage lists the official PPA and pip package as alternative installation methods.

messmerd · 2024-05-15T02:09:31Z

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

tresf · 2024-05-15T02:58:24Z

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

If parties agree on this CPack strategy, yes please (or from snap).

Tested on Ubuntu 20.04:

sudo snap install cmake --classic
alias cmake='snap run cmake'
# cmake --version
# cmake version 3.29.3
#
# CMake suite maintained and supported by Kitware (kitware.com/cmake).

... I had reservations about whether or not the snap version of cmake would play nicely... and it seems to work just fine.

tresf · 2024-05-15T06:19:33Z

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

@messmerd yeah, I'll take that offer. 🤣

Rossmaxx · 2024-05-15T06:22:26Z

can we use the PPA for mingw ci image too?

messmerd · 2024-05-15T14:09:24Z

yeah, I'll take that offer. 🤣

I'll try to do it sometime today

can we use the PPA for mingw ci image too?

Yes, I think so, though I don't think it will be needed for this PR

tresf · 2024-05-15T16:22:56Z

Yes, I think so, though I don't think it will be needed for this PR

Actually, we can justify it here since bad79a8.

tresf · 2024-05-16T04:46:39Z

I've marked this ready for review because I think it's in pretty good shape. (artifacts should start generating once cmake is updated).

ARM64.AppImage.mp4

- Refactor BashCompletion.cmake - Add initial support for Linux AppImages (untested) - Add debug flags for cpack

Add multi-arch AppImage support :D

tresf · 2024-05-17T16:24:36Z

Runners are fixed.

As it turns out, the CIs don't have FUSE installed. This wasn't obvious at first because on master, we're side-stepping this dependency by extracting linuxdeployqt before using it to expose appimagetool(blindly copied into this PR):

lmms/cmake/linux/LinuxDeployQt.cmake

Lines 47 to 51 in 3bf662b

    
           # Ensure "linuxdeployqt" binary (in extracted form) is first on the PATH 
        
           set(ENV{PATH} "${CPACK_CURRENT_BINARY_DIR}/squashfs-root/usr/bin:$ENV{PATH}") 
        
           find_program(LINUXDEPLOYQT_BIN linuxdeployqt) 
        
           find_program(APPIMAGETOOL_BIN appimagetool)

... however, this isn't needed for the new linuxdeploy (without the qt) tool; I really feel this is overkill for the average Desktop user, so I've done my best to detect if FUSE is installed in the kernel 8dfb3a9. (well, technically if one of the common FUSE tools is installed).

Some initial feedback from @ok cheez on Discord about this build (edited/formatted for readability)

Here's some good news so far:

it runs on Puppy Linux with the --allowroot ting. [...]

32-bit VST support seems stable thus far. Tested with the latest version of the GUI-less 1bitstudio plugins.

MeowSynth works somewhat.

32-bit VSTi support seems solid so far

32-bit VST effects are OK.

Pi 400 works [running the ARM64 AppImage]

I think this PR is ready for nitpicking (flow, comments, etc). Any and all feedback is greatly appreciated.

tresf · 2024-05-17T16:40:00Z

cmake/linux/CMakeLists.txt

+endif()
+
+if(CMAKE_VERSION VERSION_LESS "3.19")
+	message(WARNING "AppImage creation requires at least CMake 3.19")


Sadly, this warning is hardly noticeable. If we can find a way to halt when CPack is invoked, it would be great. Unfortunately the CPack feature that would allow us to catch this scenario -- CPACK_PRE_BUILD_SCRIPTS -- isn't available (catch-22?), so if we want to guard the users from trying to package with old CMake version, it will require some workaround that's not immediately obvious. For example, if we can detect that our install prefix contains the string _CPack*, we might be able to throw a FATAL_ERROR at install time ('install' being a subjective word here, the 'install' code is run prior to CPack, but with a custom prefix); "install" is arguably the wrong place do do it, but the alternative is a misleading and unhelpful "successful" CPack package to the user with a JSON "External" file that's neither helpful nor intuitive.

tresf changed the title ~~CPack: Proof of concept for making Apple installers~~ CPack: Refactor AppImage and Apple DMG Generation May 14, 2024

tresf marked this pull request as ready for review May 16, 2024 04:40

tresf mentioned this pull request May 16, 2024

Update MinGW CI to 20.04 #7256

Closed

tresf added 13 commits May 16, 2024 17:02

CPack: Proof of concept for making Apple installers

aa13e0b

Formatting, wording

fefa368

Remove appdmg dependency

349a13f

Shellcheck

3a8174e

Shellcheck2

08143b4

Document .DS_Store better

87590ae

Try to fix tiff file

a6f50ff

Initial refactoring

615927c

Minor changes

5c4a986

Convert apple package script to cmake

2289655

Fix appdmg

ae049e5

Order of operations: Fix broken DMG

8443819

Initial Linux support:

d95d126

- Refactor BashCompletion.cmake - Add initial support for Linux AppImages (untested) - Add debug flags for cpack

tresf added 12 commits May 16, 2024 17:04

Minor AppImage changes

b695a25

Add CMake version warning

6b87f2a

Upgrade cmake; fix typo

bca9544

Lessen [linux|mac]deployqt verbosity

4c18ac6

Remove sudo, rename CPack scripts

8d6aa20

Bundle libjack; try to fix cmake again :/

50b5903

Switch from linuxdeployqt to linuxdeploy

e2dadba

Add multi-arch AppImage support :D

Typo

33fdf4e

Document double-configure for appdmg.json.in

7747dfc

Minor formatting

09e255b

Minor formatting 2

2f962e8

Try the new runners 🎉

643f53b

tresf force-pushed the cpack branch from 2d884db to 643f53b Compare May 16, 2024 21:04

tresf added 8 commits May 16, 2024 17:38

Enable CPack debugging, fix Apple

4220d24

Try to fix verbosity for broken downloads

f25624d

Fix relative symlinks

1282515

Shim --appimage-extract into DownloadBinary.cmake for CI support

8dfb3a9

Fix comment

a1ee840

Wine guessing

c2c94ae

Finally fix linux 🤞

9ff5e44

Try to fix appimage plugin :/

3bf662b

Quiet down there, CPack

b4d347e

tresf commented May 17, 2024

View reviewed changes

tresf added 3 commits May 17, 2024 13:53

Even quieter.

82a3aa6

Qt6: Fix @rpath bug on macOS

513b0ae

Simplify include(...) statements

cbd9c1a

tresf mentioned this pull request May 21, 2024

Add Qt 6 support #6614

Open

25 tasks

Rossmaxx mentioned this pull request May 22, 2024

Request: ARM64 version for Raspberry Pi OS #6534

Open

Rossmaxx linked an issue May 22, 2024 that may be closed by this pull request

Request: ARM64 version for Raspberry Pi OS #6534

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CPack: Refactor AppImage and Apple DMG Generation #7252

CPack: Refactor AppImage and Apple DMG Generation #7252

tresf commented May 12, 2024 •

edited

tresf commented May 12, 2024 •

edited

tresf commented May 14, 2024

tresf commented May 14, 2024 •

edited

midi-pascal commented May 14, 2024

tresf commented May 14, 2024

midi-pascal commented May 14, 2024 •

edited by tresf

PhysSong commented May 15, 2024

messmerd commented May 15, 2024

tresf commented May 15, 2024 •

edited

tresf commented May 15, 2024

Rossmaxx commented May 15, 2024

messmerd commented May 15, 2024 •

edited

tresf commented May 15, 2024

tresf commented May 16, 2024 •

edited

tresf commented May 17, 2024 •

edited

tresf May 17, 2024 •

edited

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

CPack: Refactor AppImage and Apple DMG Generation #7252

Are you sure you want to change the base?

CPack: Refactor AppImage and Apple DMG Generation #7252

Conversation

tresf commented May 12, 2024 • edited

tresf commented May 12, 2024 • edited

tresf commented May 14, 2024

tresf commented May 14, 2024 • edited

midi-pascal commented May 14, 2024

tresf commented May 14, 2024

midi-pascal commented May 14, 2024 • edited by tresf

PhysSong commented May 15, 2024

messmerd commented May 15, 2024

tresf commented May 15, 2024 • edited

tresf commented May 15, 2024

Rossmaxx commented May 15, 2024

messmerd commented May 15, 2024 • edited

tresf commented May 15, 2024

tresf commented May 16, 2024 • edited

tresf commented May 17, 2024 • edited

tresf May 17, 2024 • edited

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

tresf commented May 12, 2024 •

edited

tresf commented May 12, 2024 •

edited

tresf commented May 14, 2024 •

edited

midi-pascal commented May 14, 2024 •

edited by tresf

tresf commented May 15, 2024 •

edited

messmerd commented May 15, 2024 •

edited

tresf commented May 16, 2024 •

edited

tresf commented May 17, 2024 •

edited

tresf May 17, 2024 •

edited