Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CMake support #2280

Merged
merged 138 commits into from Nov 13, 2019
Merged

Add CMake support #2280

merged 138 commits into from Nov 13, 2019

Conversation

Holzhaus
Copy link
Member

@Holzhaus Holzhaus commented Sep 12, 2019

This PR adds CMake support and aims to replace the SCons build system. This is still a work-in-progress, so maybe we can fix remaining issues collectively.
Building works on my Linux machine, but there are still issues.

  • Basic compilation support on Linux
  • Basic compilation support on Mac OS X
  • Basic compilation support on Windows
  • Make HIDAPI optional again (merge fix from Fix path in bulk feature #2279)
  • Optional Feature support
  • Basic test support
  • Fix issues with testcases
  • Fix invisible GLSL waveforms
  • Basic packaging support
  • Bring DEB packaging up to par
  • Update Travis-CI config file
  • Update AppVeyor config file
  • ...

Some of the FindXXX.cmake modules were not written by me and copied from other projects. They are under a more permissive license than Mixxx' GPL, so I think there shouldn't be a problem, but I thought I mention it in case this assumption is wrong.

@Holzhaus
Copy link
Member Author

In case you want to test this out:

# Build Mixxx
$ mkdir cmake_build
$ cd cmake_build
$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..
$ cmake --build . --parallel 4

# Install to /usr
$ cmake --build . --target install

# Make a package
$ cmake --build . --target package

# Make a Debian package
$ cpack -G DEB

# Build and run tests
$ cmake --build . --parallel 4 --target mixxx-test --target test

@Be-ing
Copy link
Contributor

Be-ing commented Sep 13, 2019

Thanks for working on this!

Why does libebur128 need to be linked statically?

@daschuer
Copy link
Member

Thank you for stating this big task.
Hopefully this will finally even lower the barrier to become a Mixxx contributor.

CMakeLists.txt Outdated Show resolved Hide resolved
CMakeLists.txt Show resolved Hide resolved
src/analyzer/analyzerkey.cpp Outdated Show resolved Hide resolved
cmake/modules/FindWavPack.cmake Outdated Show resolved Hide resolved
cmake/modules/FindWavPack.cmake Outdated Show resolved Hide resolved
cmake/modules/FindWavPack.cmake Show resolved Hide resolved
@Holzhaus
Copy link
Member Author

I updated the .travis.yml file. It builds without errors, but test 115 hangs indefinitely:

Start 115: CueControlTest.LoadUnloadTrack

@Holzhaus
Copy link
Member Author

Thanks for working on this!

Why does libebur128 need to be linked statically?

I changed the behaviour and added an option to enable static linking. By default, it now links libebur128 dynamically if it was found. If not, it's linked statically.

@Be-ing
Copy link
Contributor

Be-ing commented Sep 14, 2019

The changes to .travis.yml somehow caused Travis to stop running on this PR.

@Holzhaus
Copy link
Member Author

The changes to .travis.yml somehow caused Travis to stop running on this PR.

Fixed. Looks like the Travis YAML parser does not like commands like this:

[ "$foo" = "$bar" ] && echo "baz"

Travis can now build Mixxx using cmake, but a bunch of tests fail. Does anyone have an idea why that is?

@daschuer
Copy link
Member

Travis is still on Xenial with CMake 3.5.1-1ubuntu3 not compatible with
cmake_policy(SET CMP0071 NEW)

@daschuer
Copy link
Member

This is hard to say without a detailed log. Can you make the Travis log verbose?

@Holzhaus
Copy link
Member Author

Holzhaus commented Sep 14, 2019 via email

CMakeLists.txt Outdated Show resolved Hide resolved
@daschuer
Copy link
Member

It is libsndfile

@uklotzde
Copy link
Contributor

@daschuer @Holzhaus Please don't introduce any workarounds for supporting Xenial! Support for Xenial should be dropped as soon as it requires any extra work. I only enabled it to verify that the build is still compatible with this ancient LTS version as Daniel claimed.

Please switch the build to Bionic if needed.

@Holzhaus
Copy link
Member Author

@daschuer Apparently it's possible to compile and run Mixxx' tests without defining __SNDFILE__, so that the tests that depend on libsndfile hang forever. IMHO this is a bug. Either the tests that depend on sndfile should be disabled if __SNDFILE__ is not defined or throw an error.

@Be-ing
Copy link
Contributor

Be-ing commented Sep 15, 2019

I fully agree with @uklotzde. IMO it is not fair to ask other developers to do extra work for an outdated OS. Ubuntu 16.04 was released 3.5 years ago and has been outdated by another LTS release for 1.5 years.

@Holzhaus
Copy link
Member Author

@uklotzde @Be-ing cb52d88 is the only change I did for xenial support. And checking if the policy actually exists is the right thing to do anyway IMO:

To manage policies without increasing the minimum required CMake version, the if(POLICY)command may be used:

if(POLICY CMP0990)
  cmake_policy(SET CMP0990 NEW)
endif()

This has the effect of using the NEW behavior with newer CMake releases which users may be using and not issuing a compatibility warning.

Source: https://cmake.org/cmake/help/latest/manual/cmake-policies.7.html#introduction

CMakeLists.txt Outdated Show resolved Hide resolved
@Holzhaus
Copy link
Member Author

The windows build still fails, but I don't know why. I suppose this is the problem:

C:\Program Files (x86)\Windows Kits\10\Include\10.0.14393.0\um\winnt.h(534): error C2365: 'LP': redefinition; previous definition was 'enumerator' [C:\projects\mixxx\cmake_build\mixxx-lib.vcxproj]
c:\projects\mixxx\cmake_build\mixxx-lib_autogen\include_debug\nexqsg4pve\../../../../src/engine/filters/enginefiltermoogladder4.h(36): note: see declaration of 'LP'

Source: https://ci.appveyor.com/project/Holzhaus/mixxx/builds/27477854/job/4qwrvpcwpx9xcb7j?fullLog=true#L418

Maybe we can rename LP to LowPass, but I don't know why it worked before with SCons. Maybe the issue occurs because some other "Windows Kit" version was used? I would be nice if someone who actually has access to a windows machine and/or has productively used Windows in the last decade can help debugging this problem, because I'm lost here.

@Holzhaus
Copy link
Member Author

Is something else missing apart from fixing the windows build?

@rrrapha
Copy link
Contributor

rrrapha commented Sep 26, 2019

Does the static linking of libebur128 work for you?
On OpenBSD (without libebur128 package installed), it links against the shared library and then fails at runtime with Cannot load library libebur128.so.1.

It seems to work with the following diff, but I don't really understand it :)
I think target_link_libraries does not like the filename argument.

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ef5d94990e..da88574319 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -806,11 +806,15 @@ if(EBUR128_STATIC)
     CMAKE_ARGS ${EBUR128_CMAKE_ARGS}
   )
   set_target_properties(libebur128 PROPERTIES EXCLUDE_FROM_ALL TRUE)
-  add_dependencies(mixxx-lib libebur128)
+
+  add_library(mixxx-libebur128 STATIC IMPORTED)
+  set_target_properties(mixxx-libebur128 PROPERTIES IMPORTED_LOCATION ${CMAKE_CURRENT_BINARY_DIR}/lib/libebur128/${CMAKE_STATIC_LIBRARY_PREFIX}ebur128${CMAKE_STATIC_LIBRARY_SUFFIX})
+
+  add_dependencies(mixxx-lib mixxx-libebur128)
   target_include_directories(mixxx-lib PUBLIC
     "${CMAKE_CURRENT_SOURCE_DIR}/lib/libebur128/ebur128"
   )
-  target_link_libraries(mixxx-lib PUBLIC "${CMAKE_CURRENT_BINARY_DIR}/lib/libebur128/${CMAKE_STATIC_LIBRARY_PREFIX}ebur128${CMAKE_STATIC_LIBRARY_SUFFIX}")
+  target_link_libraries(mixxx-lib PUBLIC mixxx-libebur128)
 else()
   message(STATUS "Linking libebur128 dynamically")
   target_link_libraries(mixxx-lib PUBLIC Ebur128::Ebur128)

@Holzhaus
Copy link
Member Author

@rrrapha It works fine on my Linux machine (libebur128 is not installed). Can add this change and rerun? It prints the full path of the linked library. Maybe your CMAKE_STATIC_LIBRARY_SUFFIX is set to .so for some reason?

   set_target_properties(libebur128 PROPERTIES EXCLUDE_FROM_ALL TRUE)
   add_dependencies(mixxx-lib libebur128)
   target_include_directories(mixxx-lib PUBLIC
   "${CMAKE_CURRENT_SOURCE_DIR}/lib/libebur128/ebur128"
   )
   target_link_libraries(mixxx-lib PUBLIC "${CMAKE_CURRENT_BINARY_DIR}/lib/libebur128/${CMAKE_STATIC_LIBRARY_PREFIX}ebur128${CMAKE_STATIC_LIBRARY_SUFFIX}")
+  message(STATUS "Linking: ${CMAKE_CURRENT_BINARY_DIR}/lib/libebur128/${CMAKE_STATIC_LIBRARY_PREFIX}ebur128${CMAKE_STATIC_LIBRARY_SUFFIX}")
 else()
   message(STATUS "Linking libebur128 dynamically")
   target_link_libraries(mixxx-lib PUBLIC Ebur128::Ebur128)
 endif()

@rrrapha
Copy link
Contributor

rrrapha commented Sep 27, 2019

Maybe your CMAKE_STATIC_LIBRARY_SUFFIX is set to .so for some reason?

No, the path to the static library is correct.
My proposed diff above uses the same full path.
I still don't know why this is happening.. I use cmake-3.15.2.

Are you sure it works correctly on Linux?
Did you check the output of $ ldd ./mixxx | grep libebur?

@Holzhaus
Copy link
Member Author

Holzhaus commented Oct 1, 2019

Are you sure it works correctly on Linux?
Did you check the output of $ ldd ./mixxx | grep libebur?

Using CMake version 3.15.3:

$ ldd ./mixxx | grep -i ebur
$

Holzhaus added a commit to Holzhaus/mixxx that referenced this pull request Oct 1, 2019
@Holzhaus
Copy link
Member Author

Holzhaus commented Oct 1, 2019

I pushed a fix (hopefully), although I still can neither reproduce the issue nor see why the current implementation could cause the behaviour you described.

@uklotzde
Copy link
Contributor

uklotzde commented Nov 8, 2019

  • Qt5::X11Extras was needed for a workaround @Be-ing added to fix some Xlib event handling issues
  • -Wno-deprecated-copy was mainly intented to suppress a huge number of warnings from Qt includes, that might have been fixed in the meantime in 5.13.x/5.12.5. Any suppression of warnings is only a temporary workaround and should be disabled eventually after fixing our code.
  • I would support the decision to drop the untested asmlib feature that is not used in any of our official builds

@daschuer
Copy link
Member

daschuer commented Nov 8, 2019

I think it is quite important to have a almost warning less build because if you are constantly bothered by as bunch of unimportant message, you will miss likely the one that is a hint for a real issue.

On Xenial I have almost no warnings (need to check a full rebuild if this is still the case.) If you have new warnings from a new compiler, we should remove them.

@Holzhaus
Copy link
Member Author

Holzhaus commented Nov 8, 2019

The warnings I posted above are from gcc 9.2.0 on Arch Linux.

As soon as the code is warning-free we should enable WARNINGS_FATAL for the CI builds to prevent introducing new issues.

@Holzhaus
Copy link
Member Author

Holzhaus commented Nov 9, 2019

Are there any other issues with this PR besides the warnings?

Most of the warnings pop when building with scons as well. Of the 28 deprecated-copy warnings, only a single one is in the Mixxx code base, the other are in our copy of gmock. Can we update it? That might fix them.

Regarding the other warnings: I'm no C++ expert but the restrict warnings look like they might indicate outright bugs in Mixxx. Maybe someone can have a look a them?

I'll see if I can fix the other warnings.

@daschuer
Copy link
Member

Just building.

Can we clarify the use cases for the build types?

empty, Debug, Release, RelWithDebInfo and MinSizeRel.

I have just read "The default CMake build type is the “empty” build type, which is used in combination with the environment variables CFLAGS and CXXFLAGS." does it apply here as well?

I think we should encourage users to use RelWithDebInfo because this produces debug info, required to be able to create a GDB backlog.

What should we use for development? I like to use a fully optimized build with assertions enabled.
-o0 is only suitable for single files and hard to debug issues, because Mixxx behaves chunky and not suitable to test features under development.

What is the best way to build Mixxx with a single file at -0o?

Is MinSizeRel a reasonable build type?

@Holzhaus
Copy link
Member Author

I have just read "The default CMake build type is the “empty” build type, which is used in combination with the environment variables CFLAGS and CXXFLAGS." does it apply here as well?

It should, I didn't change anything in that regard.

I think we should encourage users to use RelWithDebInfo because this produces debug info, required to be able to create a GDB backlog.

Sounds reasonable.

What should we use for development? I like to use a fully optimized build with assertions enabled.
-o0 is only suitable for single files and hard to debug issues, because Mixxx behaves chunky and not suitable to test features under development.

Then you should probably use RelWithDebInfo. I'm unsure about the assertions. Are these disabled when QT_NO_DEBUG is set? The system-wide Qt CMake modules set QT_NO_DEBUG for all build types except DEBUG.

Another option might be to use Debug with optimizations. I copied the OPTIMIZE option from SCons: https://github.com/mixxxdj/mixxx/pull/2280/files#diff-af3b638bc2a3e6c650974192a53c7291R1969

Is MinSizeRel a reasonable build type?

Some googling revealed that MinSizeRel is a release build with -Os:

-Os Optimize for size. -Os enables all -O2 optimizations except those that often increase code size:
-falign-functions -falign-jumps -falign-labels -falign-loops -fprefetch-loop-arrays -freorder-blocks-algorithm=stc
It also enables -finline-functions, causes the compiler to tune for code size rather than execution speed, and performs further optimizations designed to reduce code size.

@daschuer
Copy link
Member

Ok, so developer should build with BUILD_TYPE=debug.
I start Mixxx with --debugAssertBreak which also breaks Mixxx in case of assertion under gdb, I can continue afterwards.
Alternatively you can enabled MIXXX_DEBUG_ASSERTIONS_FATAL to always break.

@daschuer
Copy link
Member

So from my side it LGTM. Give a hint if this is read to merge.

@uklotzde
Copy link
Contributor

@Holzhaus Please review my additions for Rekordbox. I had to mess around with the include directories to get the code (partially generated) compiled. Not particularly clean and interoperable.

The compiler warnings will be fixed directly on master.

@Holzhaus
Copy link
Member Author

I'm getting this during build:

[ 16%] Built target mixxx-qrc
AutoUic: /data/jan/Projects/mixxx/src/preferences/dialog/dlgprefautodjdlg.ui: Warning: Buddy assignment: 'ComboBoxAutoDjRequeue' is not a valid widget.
[ 16%] Built target mixxx-lib_autogen

When an include directory for a target is marked as PUBLIC, it will also
be added as to all targets that depend on it (i.e. if the target is a
library, all targets that this library is linked to via target_link_libraries
will also have this include directory). Hence, there's no need to set
this include directory twice.
@Holzhaus
Copy link
Member Author

@uklotzde Please check if this commit works for you: b17ed12

@Holzhaus
Copy link
Member Author

Feel free to merge if no other issues pop up.

@uklotzde
Copy link
Contributor

LGTM

Last chance to express any concerns. Otherwise I will merge this branch in a few hours.

@daschuer
Copy link
Member

I am looking forward to use it. It will be a real game changer for me. No more waiting for unnecessary rebuilds due to .sconsign.dblite issues. Thank you very much for this nice piece of work. :-)

@uklotzde uklotzde merged commit 883ead3 into mixxxdj:master Nov 13, 2019
@Be-ing
Copy link
Contributor

Be-ing commented Nov 13, 2019

Ubuntu builds are failing now
https://builds.renegadetech.mixxx.org/job/master-release/architecture=amd64,platform=ubuntu/894/console

WE ARE IN: /build/mixxx-2.3.0~alpha~pre/lin64_build
Building    - rev. 
Install root: /build/mixxx-2.3.0~alpha~pre/debian/tmp/usr
scons: done reading SConscript files.
scons: Cleaning targets ...
scons: done cleaning targets.
Removing autogenerated file .sconsign.dblite
Removing autogenerated directory .sconf_temp
rm -rf .sconf_temp/ cache/ linux_build/
dh_clean .sconsign.dblite cachecustom.py \
	config.log src/build.h build/*.pyc mixxx.1
dh_auto_clean
make[1]: Leaving directory '/build/mixxx-2.3.0~alpha~pre'
   dh_clean
 dpkg-source -b mixxx-2.3.0~alpha~pre
dpkg-source: warning: no source format specified in debian/source/format, see dpkg-source(1)
dpkg-source: info: using source format '1.0'
dpkg-source: info: building mixxx in mixxx_2.3.0~alpha~pre-ppa1~master~git6996~bionic.tar.gz
dpkg-source: info: building mixxx in mixxx_2.3.0~alpha~pre-ppa1~master~git6996~bionic.dsc
 debian/rules build
make: 'build' is up to date.
 fakeroot debian/rules binary
dh binary --parallel
   dh_update_autotools_config
   dh_auto_configure
	cd obj-x86_64-linux-gnu && cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=None -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_LOCALSTATEDIR=/var -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON
Can't exec "cmake": No such file or directory at /usr/share/perl5/Debian/Debhelper/Dh_Lib.pm line 356.
dh_auto_configure: cd obj-x86_64-linux-gnu && cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=None -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_LOCALSTATEDIR=/var -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON failed to execute: No child processes
dh_auto_configure: cd obj-x86_64-linux-gnu && cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=None -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_LOCALSTATEDIR=/var -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON returned exit code 2
debian/rules:34: recipe for target 'binary' failed
make: *** [binary] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2
I: copying local configuration
E: Failed autobuilding of package
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /var/cache/pbuilder/build//25720 and its subdirectories
* Build failed.

scons: *** [lin64_build/blah] Exception : Ubuntu package build failed.
Traceback (most recent call last):
  File "/usr/lib/scons/SCons/Action.py", line 1197, in execute
    result = self.execfunction(target=target, source=rsources, env=env)
  File "/home/mixxx/workspace/master-release/architecture/amd64/platform/ubuntu/SConscript", line 1129, in BuildUbuntuPackage
    raise Exception('Ubuntu package build failed.')
Exception: Ubuntu package build failed.
scons: building terminated because of errors.
OSError: [Errno 2] No such file or directory: '.sconsign.dblite':
  File "/usr/lib/scons/SCons/Script/Main.py", line 1376:
    _exec_main(parser, values)
  File "/usr/lib/scons/SCons/Script/Main.py", line 1339:
    _main(parser)
  File "/usr/lib/scons/SCons/Script/Main.py", line 1103:
    nodes = _build_targets(fs, options, targets, target_top)
  File "/usr/lib/scons/SCons/Script/Main.py", line 1313:
    jobs.run(postfunc = jobs_postfunc)
  File "/usr/lib/scons/SCons/Job.py", line 113:
    postfunc()
  File "/usr/lib/scons/SCons/Script/Main.py", line 1310:
    SCons.SConsign.write()
  File "/usr/lib/scons/SCons/SConsign.py", line 116:
    syncmethod()
  File "/usr/lib/scons/SCons/dblite.py", line 157:
    self._os_unlink(self._file_name)
Exception OSError: (2, 'No such file or directory', '.sconsign.dblite') in <bound method dblite.__del__ of <SCons.dblite.dblite object at 0x7f7966a9ed10>> ignored
Build step 'Execute shell' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
SSH: Current build result is [FAILURE], not going to run.
Finished: FAILURE

@Be-ing
Copy link
Contributor

Be-ing commented Nov 15, 2019

Ping. Builds are still failing.

@Holzhaus
Copy link
Member Author

Holzhaus commented Nov 15, 2019

This has to be unrelated. This PR didn't touch any existing files.

EDIT: Looks like dh_auto_configure sees the CMakeLists.txt and tries to use that instead of scons.

@Holzhaus
Copy link
Member Author

Someone probably needs to add this to the build configuration:

-Sbuildsystem, --buildsystem=buildsystem
Force use of the specified buildsystem, instead of trying to auto-select one which might be applicable for the package.
Source: man 7 debhelper

@Be-ing
Copy link
Contributor

Be-ing commented Nov 16, 2019

Attempt to fix: #2361

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
2.3 release
  
Done
Development

Successfully merging this pull request may close these issues.

None yet

7 participants