CPack: Refactor AppImage and Apple DMG Generation #7252

tresf · 2024-05-12T01:17:21Z

Features:

Simplifies Mac and Linux packaging command to simply: make package

- make install && make appimage
- make install && make dmg
+ make package
# -- AppImage create: /__w/lmms/lmms/build/lmms-1.3.0-alpha.1.638+pr7252.gaaa9b4816-linux-x86_64.AppImage

Adds new WANT_DEBUG_CPACK flag to easily show detailed console messages about packaging

cmake .. -DWANT_DEBUG_CPACK=true

  Developer options
  -----------------------------------------
  * Debug FP exceptions               : Disabled
  * Debug using AddressSanitizer      : Disabled
  * Debug using ThreadSanitizer       : Disabled
  * Debug using MemorySanitizer       : Disabled
  * Debug using UBSanitizer           : Disabled
+ * Debug packaging commands          : Enabled

Preserves old CPack TGZ behavior through a hidden flag, WANT_CPACK_TARBALL. Does anyone use this?

Adds new CPACK_TOOL parameter to switch between .AppImage (default) and .run (experimental)

export CPACK_TOOL=makeself
make package
# -- Installer create: /__w/lmms/lmms/build/lmms-1.3.0-alpha.1.638+pr7252.gaaa9b4816-linux-x86_64.run

TODO:

Fix Carla crashing
Investigate possibly using the provided Mac DMG generators.
- Port the install_apple.sh script to CMake.
  It's called MacDeployQt.cmake now.
Investigate possibility using CPack for creating Linux AppImages
- Port the package_linux.sh script to Cmake
  It's called LinuxDeploy.cmake now.
Upgrade CMake for Linux runners

BUGS:

Fix background missing for DMG
Fix AppImages not building for Linux
WANT_DEBUG_CPACK is incorrectly toggled in CI. https://github.com/LMMS/lmms/actions/runs/9131345271/job/25110338288. Closed via 82a3aa6.

STRETCH GOALS:

Port project from linuxdeployqt to linuxdeploy
Examples:
- pbek/QOwnNotes (thank you!)
- wireshark/wireshark@34ce99f (thank you!)
Remove all references to linuxdeployqt (in favor of linuxdeploy)
~~[ ] Add ARM64 AppImage~~
- Can linuxdeploy support a cross-compilation environment ? linuxdeploy/linuxdeploy#258
- On hold until GitHub actions Linux ARM64 runners are released to the general public. https://github.com/orgs/community/discussions/73040#discussioncomment-9661482

Background:

Click to expand

The AppImage and Apple DMG installers historically (confusingly) require make install to be run prior to making packages. This PR is a proposal to remove that by switching to CPack.
- Utilizes make package now, just like Windows does
- Mac utilizes CPACK_GENERATOR of ~~DragNDrop~~ Bundle which will create a DMG without the need for node's appdmg package (still requires macdeployqt)
- Linux will utilize CPACK_GENERATOR of "External", which will use custom CMake commands to replace package_linux.sh (still requires linuxdeployqt).

tresf · 2024-05-12T04:37:58Z

Rule out possibly using the provided Mac DMG generators.

I was able to get the CPack DMG generator to work with macdeployqt however in order for us to brand our own DMG, we either need to leverage AppleScript (which is much slower and requires user interaction over appdmg), or we need to provide a premade .DS_Store file to bundle.

Edit: For simplicity, I've checked in the .DS_Store from our current DMG. Since it will be unobvious how to update this file in the event of a rebranding, I've kept basic support for appdmg for recreating it, although it's not used or needed unless we rebrand the installer.

tresf · 2024-05-14T03:35:15Z

Hmm... the AppImage is missing... investigating...

tresf · 2024-05-14T13:41:09Z

The AppImage is missing because the CPack feature for running custom scripts through variable CPACK_PRE_BUILD_SCRIPTS was not available in Ubuntu 20.04. There doesn't seem to be a replacement for this feature, which is alarming, I wonder how people used "External" packaging techniques prior to CMake 3.19. Hmm...

midi-pascal · 2024-05-14T14:03:53Z

Perhaps I have a hint for this:

I use Ubuntu 20.04 that comes with cmake version 3.16 BUT Qt (5.15.2 installed from Qt site) comes with cmake 3.27 in its tools directory. This is the one I use.

tresf · 2024-05-14T20:36:01Z

I use Ubuntu 20.04 that comes with cmake version 3.16 BUT Qt (5.15.2 installed from Qt site) comes with cmake 3.27 in its tools directory. This is the one I use.

Thanks! Any recommendation for getting this into a CI (command line?)

midi-pascal · 2024-05-14T21:06:31Z

Not sure of what I will suggest below since I build using Qt Creator(so all is set automagically) but there are environment variables related to cmake:

CMAKE_PREFIX_PATH

CMAKE_PREFIX_PATH
Semicolon-separated list of directories specifying installation
prefixes to be searched by the find_package(),
find_program(), find_library(), find_file(), and
find_path() commands. Each command will add appropriate
subdirectories (like bin, lib, or include) as specified in its own
documentation.
By default this is empty. It is intended to be set by the project.
There is also an environment variable CMAKE_PREFIX_PATH, which is used
as an additional list of search prefixes.
See also CMAKE_SYSTEM_PREFIX_PATH, CMAKE_INCLUDE_PATH,
CMAKE_LIBRARY_PATH, CMAKE_PROGRAM_PATH, and
CMAKE_IGNORE_PATH.

Hope this can help :-)

PhysSong · 2024-05-15T01:37:00Z

FYI, CMake homepage lists the official PPA and pip package as alternative installation methods.

messmerd · 2024-05-15T02:09:31Z

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

tresf · 2024-05-15T02:58:24Z

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

If parties agree on this CPack strategy, yes please (or from snap).

Tested on Ubuntu 20.04:

sudo snap install cmake --classic
alias cmake='snap run cmake'
# cmake --version
# cmake version 3.29.3
#
# CMake suite maintained and supported by Kitware (kitware.com/cmake).

... I had reservations about whether or not the snap version of cmake would play nicely... and it seems to work just fine.

tresf · 2024-05-15T06:19:33Z

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

@messmerd yeah, I'll take that offer. 🤣

Rossmaxx · 2024-05-15T06:22:26Z

can we use the PPA for mingw ci image too?

messmerd · 2024-05-15T14:09:24Z

yeah, I'll take that offer. 🤣

I'll try to do it sometime today

can we use the PPA for mingw ci image too?

Yes, I think so, though I don't think it will be needed for this PR

tresf · 2024-05-15T16:22:56Z

Yes, I think so, though I don't think it will be needed for this PR

Actually, we can justify it here since bad79a8.

tresf · 2024-05-16T04:46:39Z

I've marked this ready for review because I think it's in pretty good shape. (artifacts should start generating once cmake is updated).

ARM64.AppImage.mp4

- Refactor BashCompletion.cmake - Add initial support for Linux AppImages (untested) - Add debug flags for cpack

tresf · 2024-06-01T14:04:51Z

Basic .run (via makeself) support has been added via 177cd21.

Testing on Discord shows the following error, investigating...

- ./lmms-1.3.0.run: /home/ubuntu/lmms2/build/cmake/linux/setup.sh: inaccessible or not found

tresf · 2024-06-02T06:40:23Z

I'll have to revert the MacDeployQt script back to using appdmg.

Done via ddca4e7.

tresf · 2024-06-05T17:47:08Z

Due to unreliability with FUSE (availability, compatibility), I'm changing this PR to always extracting AppImages before running them. Quoting #dev-only on Discord:

On the cpack branch, I'm making an executive decision to just never trust AppImages will ever work without extraction. So far:

✅ Ubuntu 20.04 Desktop

🚫 Ubuntu 22.04 Desktop

🚫 Ubuntu 20.04 Server

🚫 Ubuntu 22.04 Server

This will be transient to devs but I just don't see a good reason to ever try to run the AppImage from FUSE when the majority of systems just seem to be missing it or have problems with the version.

Note, master does this already, but for different reasons.

At a glance, this may be a bit confusing, because the build system will create a symlink with the AppImage name. e.g. linuxdeploy-x86_64.Appimage --> .linuxdeploy-x86_64.AppImage/squashfs-root/AppRun, but this allows developers to call the binary by its expected name without having to know where the AppRun was placed. Hopefully this change is percevied as intuitive and not too confuscated™️ . 🍻

Related LMMS#7201

messmerd · 2024-10-23T16:35:57Z

On Linux, Carla Rack and Carla Patchbay crash when I try to use them, and this is not the case on master

EDIT: Here's the command-line output:

Carla appears to be installed on this system at /usr/lib[64]/carla so we'll use it.
Jack appears to be installed on this system, so we'll use it.
Lv2 plugin SUMMARY: 18 of 40 loaded in 80 msecs.
For details about not loaded plugins, please set
environment variable "LMMS_LV2_DEBUG" to nonempty.
/tmp/.mount_lmms-1XSszgf/AppRun.wrapped: line 24: 800103 Segmentation fault (core dumped) QT_X11_NO_NATIVE_MENUBAR=1 "$DIR"/usr/bin/lmms "$@"

JohannesLorenz · 2024-10-24T22:05:56Z

cmake/modules/BashCompletion.cmake

+		if(BASHCOMP_PKG_PATH)
+			# TODO: CMake 3.21 Use "file(COPY_FILE ...)"
+			install(CODE "
+					execute_process(COMMAND ${CMAKE_COMMAND} -E copy \"${SCRIPT_NAME}\" \"${BASHCOMP_PKG_PATH}\" ERROR_QUIET RESULT_VARIABLE result)


Why does it install SCRIPT_NAME twice, once in line 47 and once in line 51?
Also, where is the BASHCOMP_PKG_PATH variable from?

Why does it install SCRIPT_NAME twice, once in line 47 and once in line 51?

It's due to cmake's lack of conditional INSTALL(...) logic. For some reason, I removed the comment that describes this. It's here: 00d4ddd#diff-7960a523ea4d93951917fb0445e1145899f31c5a796e123ee06c805095e37221R59-R71.

In short:

Always install to user-space

Attempt to install system-wide, but don't fail the build if it doesn't work.

We did have some talks about this "system wide" install previously because this is the first time that we've had a component to install in a prefix that's outside of our own install prefix. I suppose we could instead detect a system-wide prefix and handle this in a more standard fashion. If that's the case, we can fix it as part of this PR if needed. I know some environments (such as Apple's Homebrew) ARE user-writable, so perhaps that was my motivation for this but that does NOT make sense to try when building a DMG or AppImage, so this logic is likely flawed.

Furthermore, I think there's a typo in the status message though.... I think this should say BASHCOMP_PKG_PATH not BASHCOMP_USER_PATH.

- message(STATUS "Unable to install bash-completion support system-wide: ${BASHCOMP_USER_PATH}/${SCRIPT_NAME}) + - message(STATUS "Unable to install bash-completion support system-wide: ${BASHCOMP_PKG_PATH}/${SCRIPT_NAME})

Also, where is the BASHCOMP_PKG_PATH variable from?

It's from #4604. This is not a newly introduced variable with this PR. I believe its intended to be the completionsdir component, found by pkg-config. My memory is foggy from back then and the conversations seem to have largely occurred on Discord so decisions are not all obvious. Anyway, we also re-use this variable name for cmake's find_package(bash-completion) portion as well. It's probably a bad name, but I do not plan on changing it as part of this PR but can if needed.

I believe its intended to be the completionsdir component, found by pkg-config.

Indeed! The top of the file actually explains it. You can even set this variable using -DBASHCOMP_PKG_PATH=....

Furthermore, I think there's a typo in the status message though.... I think this should say BASHCOMP_PKG_PATH not BASHCOMP_USER_PATH.

Agreeing. Please feel free to fix.

Attempt to install system-wide, but don't fail the build if it doesn't work.

IMO, the natural behavior would be: If the user does a local install, install bashcomp locally. If they do a system install ("root"), install it system-wide. I assume that back then, we thought: "An install should always try to be system-wide, because local installed bashcomp is not sourced by your bashrc". However, you could give the same argument for the PATH variable (a local install of bin/lmms will never be in your PATH by default). In both cases, you can extend your bashrc to source a local bashcomp or to extend your PATH to a local directory. All in all, I would favor to do the install exactly one, and always in the normal install tree. However, it is just my basic opinion, and also it might be considered out of scope of this PR. Tagging @PhysSong because they were also active in that bashcomp PR in 2018.

Thanks for the feedback. I'll put some time into fixing this.

tresf · 2024-10-25T03:52:26Z

On Linux, Carla Rack and Carla Patchbay crash when I try to use them, and this is not the case on master

Thanks! Should be fixed via aab4cc5.

tresf · 2024-10-25T04:13:46Z

On Linux, Carla Rack and Carla Patchbay crash when I try to use them, and this is not the case on master

Thanks! Should be fixed via aab4cc5.

Nevermind, still crashes. Investigating...

messmerd · 2024-10-25T06:05:15Z

@tresf
There are 119 files and 2 directories under squashfs-root/usr/lib in this PR, but just 92 files and 1 directory in master.

I'm not sure why there are so many more, but of those extra files, there were libcarlabase.so and libcarla_native-plugin.so. After I removed them, Carla now longer crashed LMMS.

tresf · 2024-10-25T06:26:56Z

@tresf There are 119 files and 2 directories under squashfs-root/usr/lib in this PR, but just 92 files and 1 directory in master.

I'm not sure why there are so many more, but of those extra files, there were libcarlabase.so and libcarla_native-plugin.so. After I removed them, Carla now longer crashed LMMS.

@messmerd thank you. I'm fully aware of the cause of the crash. linuxdeploy is run twice. First using a qt plugin, which we first use to crawl dependencies and create the .AppDir structure, but then we call it a second time passing the --output appimage flag which we call only after manual cleanup of these carla libraries is performed. Instead of being a replacement for the appimagetool command, it re-runs the dependency tree, so any manually removed files are copied back in. Here's the offending line:

lmms/cmake/linux/LinuxDeploy.cmake

Lines 244 to 250 in 22f17e0

    
           execute_process(COMMAND "${LINUXDEPLOY_BIN}" 
        
           	--appdir "${APP}" 
        
           	--output appimage 
        
           	--verbosity ${VERBOSITY} 
        
           	${OUTPUT_QUIET} 
        
           	COMMAND_ECHO ${COMMAND_ECHO} 
        
           	COMMAND_ERROR_IS_FATAL ANY)

There's a new --exclude-library flag which takes a globbing pattern that I'm experimenting with to see if it can replace the manual cleanup efforts.

tresf · 2024-10-25T07:06:02Z

Apologies for the whitespace changes in the latest commit 508f430, but spaces and tabs were inconsistent.

The relevant fix is the addition of ${EXCLUDES} variable which adds --exclude-library=pattern. It could be argued that the glob should be provided to both calls to linuxdeploy and the manual cleanup effort removed (rather than having two steps 1. cmake do the glob, 2. provide any found filenames to the --exclude-library flag) but at this point I'm afraid to change too much in fear of causing other issues.

Another big miss was never providing linuxdeploy the -executable= flags. We were building it but never passing it to linuxdeploy. This was would likely have caused some linking issues if not fixed prior to merging. (ignore the double dash change, that's just for readiblity).

There are 119 files and 2 directories under squashfs-root/usr/lib in this PR, but just 92 files and 1 directory in master.

I haven't done a comparison yet, but one thing that linuxdeploy does that linuxdeployqt did not is to attempt to fetch the licensing information for all dependencies. Most of these attempts fail when verbose logging is turned on, but it's one possible culprit. The second directory is a qt wrapper script exclusive to this tool.

I haven't tested the AppImage yet but I believe the Carla loading issue is at least resolved. I'm curious what the directory tree compare looks like. I used to use a commercial tool called "Beyond Compare" for comparing file trees, perhaps there's a FOSS equivalent.

messmerd · 2024-10-25T08:21:16Z

I've confirmed that Carla no longer crashes.

However, it looks like there are copies of all the LMMS plugins and some other .so files in both usr/bin and usr/lib/lmms instead of just the latter.
And for at least the past couple commits, the LMMS icon hasn't been showing up on the AppImage file.

As a side note, I just realized xpressive is a whopping 71 MB. It's also always last to finish compiling in our CI builds, so maybe something should be done about that. (Though not in this PR of course)

tresf · 2024-10-25T14:15:29Z

Although undocumented, it appears appimagetool is available as part of the linuxdeploy tool, as can be observed when verbose logging WANT_DEBUG_CPACK is enabled. I may attempt to use this directly. I'll have to put a few more hours into it.

tresf changed the title ~~CPack: Proof of concept for making Apple installers~~ CPack: Refactor AppImage and Apple DMG Generation May 14, 2024

tresf marked this pull request as ready for review May 16, 2024 04:40

tresf mentioned this pull request May 16, 2024

Update MinGW CI to 20.04 #7256

Closed

tresf added 13 commits May 16, 2024 17:02

CPack: Proof of concept for making Apple installers

aa13e0b

Formatting, wording

fefa368

Remove appdmg dependency

349a13f

Shellcheck

3a8174e

Shellcheck2

08143b4

Document .DS_Store better

87590ae

Try to fix tiff file

a6f50ff

Initial refactoring

615927c

Minor changes

5c4a986

Convert apple package script to cmake

2289655

Fix appdmg

ae049e5

Order of operations: Fix broken DMG

8443819

Initial Linux support:

d95d126

- Refactor BashCompletion.cmake - Add initial support for Linux AppImages (untested) - Add debug flags for cpack

tresf added 2 commits June 1, 2024 11:45

Fix makeself installer script

0769ad4

Prefer back to appdmg over CMake's DMG generator

ddca4e7

Misc cleanup

fcb8e1e

tresf mentioned this pull request Jun 5, 2024

Lv2 UI - Testing #7201

Draft

19 tasks

tresf added 3 commits June 5, 2024 14:05

Don't trust FUSE, ever. :D

5e3a088

Detect and bundle LV2 UI Suil modules

fce984f

Related LMMS#7201

Pass Suil modules to macdeployqt, linuxdeploy

b099c35

tresf mentioned this pull request Jun 19, 2024

minor cleanup of MinGW-related scripts #7327

Merged

JohannesLorenz added this to the 1.3 milestone Aug 4, 2024

tresf added 6 commits September 16, 2024 13:20

Merge branch 'master' into cpack

7051ced

Remove unused CMAKE_INSTALL_PREFIX

fda485a

true -> TRUE, false -> FALSE

eb25a0d

Deobfuscate CPACK_DEBUG

cc9cde2

Don't strip Windows libs (for historic reasons)

7f217d8

Remove lingering linuxdeployqt workaround

a9fe1b3

JohannesLorenz reviewed Oct 24, 2024

View reviewed changes

Fix Carla on Linux

aab4cc5

Make DownloadBinary reproducible

22f17e0

Fix Carla, whitespace

508f430

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CPack: Refactor AppImage and Apple DMG Generation #7252

CPack: Refactor AppImage and Apple DMG Generation #7252

tresf commented May 12, 2024 •

edited

Loading

tresf commented May 12, 2024 •

edited

Loading

tresf commented May 14, 2024

tresf commented May 14, 2024 •

edited

Loading

midi-pascal commented May 14, 2024

tresf commented May 14, 2024

midi-pascal commented May 14, 2024 •

edited by tresf

Loading

PhysSong commented May 15, 2024

messmerd commented May 15, 2024

tresf commented May 15, 2024 •

edited

Loading

tresf commented May 15, 2024

Rossmaxx commented May 15, 2024

messmerd commented May 15, 2024 •

edited

Loading

tresf commented May 15, 2024

tresf commented May 16, 2024 •

edited

Loading

tresf commented Jun 1, 2024

tresf commented Jun 2, 2024

tresf commented Jun 5, 2024 •

edited

Loading

messmerd commented Oct 23, 2024 •

edited

Loading

JohannesLorenz Oct 24, 2024

tresf Oct 24, 2024 •

edited

Loading

JohannesLorenz Oct 26, 2024 •

edited

Loading

tresf Oct 26, 2024

tresf commented Oct 25, 2024

tresf commented Oct 25, 2024

messmerd commented Oct 25, 2024

tresf commented Oct 25, 2024 •

edited

Loading

tresf commented Oct 25, 2024 •

edited

Loading

messmerd commented Oct 25, 2024

tresf commented Oct 25, 2024

CPack: Refactor AppImage and Apple DMG Generation #7252

Are you sure you want to change the base?

CPack: Refactor AppImage and Apple DMG Generation #7252

Conversation

tresf commented May 12, 2024 • edited Loading

tresf commented May 12, 2024 • edited Loading

tresf commented May 14, 2024

tresf commented May 14, 2024 • edited Loading

midi-pascal commented May 14, 2024

tresf commented May 14, 2024

midi-pascal commented May 14, 2024 • edited by tresf Loading

PhysSong commented May 15, 2024

messmerd commented May 15, 2024

tresf commented May 15, 2024 • edited Loading

tresf commented May 15, 2024

Rossmaxx commented May 15, 2024

messmerd commented May 15, 2024 • edited Loading

tresf commented May 15, 2024

tresf commented May 16, 2024 • edited Loading

tresf commented Jun 1, 2024

tresf commented Jun 2, 2024

tresf commented Jun 5, 2024 • edited Loading

messmerd commented Oct 23, 2024 • edited Loading

JohannesLorenz Oct 24, 2024

Choose a reason for hiding this comment

tresf Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

JohannesLorenz Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

tresf Oct 26, 2024

Choose a reason for hiding this comment

tresf commented Oct 25, 2024

tresf commented Oct 25, 2024

messmerd commented Oct 25, 2024

tresf commented Oct 25, 2024 • edited Loading

tresf commented Oct 25, 2024 • edited Loading

messmerd commented Oct 25, 2024

tresf commented Oct 25, 2024

tresf commented May 12, 2024 •

edited

Loading

tresf commented May 12, 2024 •

edited

Loading

tresf commented May 14, 2024 •

edited

Loading

midi-pascal commented May 14, 2024 •

edited by tresf

Loading

tresf commented May 15, 2024 •

edited

Loading

messmerd commented May 15, 2024 •

edited

Loading

tresf commented May 16, 2024 •

edited

Loading

tresf commented Jun 5, 2024 •

edited

Loading

messmerd commented Oct 23, 2024 •

edited

Loading

tresf Oct 24, 2024 •

edited

Loading

JohannesLorenz Oct 26, 2024 •

edited

Loading

tresf commented Oct 25, 2024 •

edited

Loading

tresf commented Oct 25, 2024 •

edited

Loading