Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build: VST3 capable build impossible offline (without Conan) #3474

Closed
hfiguiere opened this issue Aug 30, 2022 · 24 comments
Closed

Build: VST3 capable build impossible offline (without Conan) #3474

hfiguiere opened this issue Aug 30, 2022 · 24 comments
Assignees
Labels
Build / CI Bugs and tasks regarding builds and Continuous Integration community build bugs Bug in non-official builds, such as distro-repos or self-compiled ones

Comments

@hfiguiere
Copy link

hfiguiere commented Aug 30, 2022

Bug description

I don't have Conan as the build is done offline.

Steps to reproduce

  1. Build as instructed with -Daudacity_conan_enabled=Off
  2. CMake faile to find VST3 SDK

Expected behavior

That I can pass a documented flag to CMake to tell it were to find the VST3SDK

Actual behavior

Nothing. I tries to detect things with black magic

Audacity Version

latest beta

Operating system

Linux

Additional context

QA: This is a build problem.

@hfiguiere hfiguiere added the bug An error, undesired behaviour, or missed functionality label Aug 30, 2022
@hfiguiere
Copy link
Author

hfiguiere commented Aug 30, 2022

The error message:

-- Checking for module 'expat >= 2.1.0'
--   Found expat , version 2.4.6
-- Using 'expat' system library
-- Interface expat::expat:
	include: 
	Libraries: 
CMake Error at cmake-proxies/cmake-modules/AudacityDependencies.cmake:238 (message):
  Failed to find the system package vst3sdk
Call Stack (most recent call first):
  cmake-proxies/CMakeLists.txt:26 (add_conan_lib)


-- Configuring incomplete, errors occurred!

vst3sdk doesn't have a system package. It doesn't exist as such.

@LWinterberg LWinterberg added community build bugs Bug in non-official builds, such as distro-repos or self-compiled ones Build / CI Bugs and tasks regarding builds and Continuous Integration and removed bug An error, undesired behaviour, or missed functionality labels Aug 30, 2022
@dvzrv
Copy link

dvzrv commented Sep 22, 2022

I have the same issue with the package for Arch Linux.

@dvzrv
Copy link

dvzrv commented Sep 24, 2022

After hacking around in the cmake setup to somehow make this work with:

diff -ruN a/cmake-proxies/CMakeLists.txt b/cmake-proxies/CMakeLists.txt
--- a/cmake-proxies/CMakeLists.txt      2022-09-22 18:21:04.000000000 +0200
+++ b/cmake-proxies/CMakeLists.txt      2022-09-24 14:34:41.500278961 +0200
@@ -25,6 +25,7 @@
 if(${_OPT}has_vst3)
     add_conan_lib(
         vst3sdk vst3sdk/3.7.3
+        PKG_CONFIG "vst3sdk >= 3.7.3"
         REQUIRED
         HAS_ONLY_DEBUG_RELEASE
     )
diff -ruN a/src/CMakeLists.txt b/src/CMakeLists.txt
--- a/src/CMakeLists.txt        2022-09-22 18:21:05.000000000 +0200
+++ b/src/CMakeLists.txt        2022-09-24 14:46:27.248454524 +0200
@@ -1153,6 +1153,7 @@
       ${_PRVDIR}
       ${topdir}/include
       ${TARGET_ROOT}
+      /usr/include/vst3sdk
 )

 #
@@ -1224,10 +1225,6 @@
       PortAudio::PortAudio
       sqlite
       $<$<BOOL:${${_OPT}has_crashreports}>:crashreports>
-      $<$<BOOL:${${_OPT}has_vst3}>:vst3sdk::base>
-      $<$<BOOL:${${_OPT}has_vst3}>:vst3sdk::pluginterfaces>
-      $<$<BOOL:${${_OPT}has_vst3}>:vst3sdk::sdk_hosting>
-      $<$<BOOL:${${_OPT}has_vst3}>:vst3sdk::sdk_common>
       $<$<BOOL:${USE_LIBID3TAG}>:libid3tag::libid3tag>
       $<$<BOOL:${USE_LIBFLAC}>:FLAC::FLAC>
       $<$<BOOL:${USE_LIBFLAC}>:FLAC::FLAC++>

I get a public/private issue:

[ 77%] Building CXX object src/CMakeFiles/Audacity.dir/effects/VST3/AudacityVst3HostApplication.cpp.o
In file included from /usr/include/vst3sdk/pluginterfaces/base/funknown.h:20,
                 from /usr/include/vst3sdk/pluginterfaces/vst/ivstattributes.h:19,
                 from /usr/include/vst3sdk/pluginterfaces/vst/ivstmessage.h:19,
                 from /usr/include/vst3sdk/pluginterfaces/vst/ivsthostapplication.h:19,
                 from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.h:3,
                 from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.cpp:1:
/usr/include/vst3sdk/pluginterfaces/base/ftypes.h:30: warning: "UNICODE" redefined
   30 |         #define UNICODE 1
      |
In file included from /usr/include/wx-3.2/wx/defs.h:45,
                 from /usr/include/wx-3.2/wx/wx.h:14,
                 from /build/audacity/src/audacity-sources-3.2.0/src/AudacityHeaders.h:32,
                 from /build/audacity/src/build/src/CMakeFiles/Audacity.dir/cmake_pch.hxx:6,
                 from <command-line>:
/usr/include/wx-3.2/wx/platform.h:221: note: this is the location of the previous definition
  221 | #       define UNICODE
      |
/usr/include/vst3sdk/pluginterfaces/base/ftypes.h:34: warning: "_UNICODE" redefined
   34 | #define _UNICODE 1
      |
/usr/include/wx-3.2/wx/platform.h:218: note: this is the location of the previous definition
  218 | #       define _UNICODE
      |
/build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.cpp: In member function ‘virtual Steinberg::tresult AudacityVst3HostApplication::createInstance(char*, char*, void**)’:
/build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.cpp:67:37: error: ‘Steinberg::Vst::HostAttributeList::HostAttributeList()’ is private within this context
   67 |                 *obj = safenew Vst::HostAttributeList;
      |                                     ^~~~~~~~~~~~~~~~~
In file included from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.cpp:6:
/usr/include/vst3sdk/public.sdk/source/vst/hosting/hostclasses.h:92:9: note: declared private here
   92 |         HostAttributeList ();
      |         ^~~~~~~~~~~~~~~~~
[ 77%] Building CXX object src/CMakeFiles/Audacity.dir/effects/VST3/memorystream.cpp.o
[ 77%] Building CXX object src/CMakeFiles/Audacity.dir/effects/VST3/VST3Effect.cpp.o
In file included from /usr/include/vst3sdk/pluginterfaces/base/funknown.h:20,
                 from /usr/include/vst3sdk/pluginterfaces/base/ibstream.h:19,
                 from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/memorystream.h:39,
                 from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/memorystream.cpp:37:
/usr/include/vst3sdk/pluginterfaces/base/ftypes.h:30: warning: "UNICODE" redefined
   30 |         #define UNICODE 1
      |
In file included from /usr/include/wx-3.2/wx/defs.h:45,
                 from /usr/include/wx-3.2/wx/wx.h:14,
                 from /build/audacity/src/audacity-sources-3.2.0/src/AudacityHeaders.h:32,
                 from /build/audacity/src/build/src/CMakeFiles/Audacity.dir/cmake_pch.hxx:6,
                 from <command-line>:
/usr/include/wx-3.2/wx/platform.h:221: note: this is the location of the previous definition
  221 | #       define UNICODE
      |
/usr/include/vst3sdk/pluginterfaces/base/ftypes.h:34: warning: "_UNICODE" redefined
   34 | #define _UNICODE 1
      |
/usr/include/wx-3.2/wx/platform.h:218: note: this is the location of the previous definition
  218 | #       define _UNICODE
      |
[ 77%] Building CXX object src/CMakeFiles/Audacity.dir/effects/VST3/VST3EffectsModule.cpp.o
In file included from /usr/include/vst3sdk/pluginterfaces/base/funknown.h:20,
                 from /usr/include/vst3sdk/public.sdk/source/vst/hosting/../utility/uid.h:40,
                 from /usr/include/vst3sdk/public.sdk/source/vst/hosting/module.h:39,
                 from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/VST3Effect.h:18,
                 from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/VST3Effect.cpp:13:
/usr/include/vst3sdk/pluginterfaces/base/ftypes.h:30: warning: "UNICODE" redefined
   30 |         #define UNICODE 1
      |
In file included from /usr/include/wx-3.2/wx/defs.h:45,
                 from /usr/include/wx-3.2/wx/wx.h:14,
                 from /build/audacity/src/audacity-sources-3.2.0/src/AudacityHeaders.h:32,
                 from /build/audacity/src/build/src/CMakeFiles/Audacity.dir/cmake_pch.hxx:6,
                 from <command-line>:
/usr/include/wx-3.2/wx/platform.h:221: note: this is the location of the previous definition
  221 | #       define UNICODE
      |
/usr/include/vst3sdk/pluginterfaces/base/ftypes.h:34: warning: "_UNICODE" redefined
   34 | #define _UNICODE 1
      |
/usr/include/wx-3.2/wx/platform.h:218: note: this is the location of the previous definition
  218 | #       define _UNICODE
      |

@LWinterberg @crsib it would be great if this could be addressed, as without a fix for this issue, downstreams will not be able to build audacity 3.2.0.

@dvzrv
Copy link

dvzrv commented Sep 24, 2022

FTR: Downstreams may provide vst3sdk installed system-wide and it makes sense to provide the option to either target a specific directory or find it via pkg-config (the latter is not supported by upstream because of unknown reasons though).

An example of good vst3sdk integration is offered by qtractor.

@crsib
Copy link
Member

crsib commented Sep 24, 2022

I am not aware of any Linux distribution that has binary version of vst3sdk. We do not plan to vendor it in any way either. There is a proposed way to use Conan offline. If you are strictly against it I think it is better to build Audacity without vst3 support.

@dvzrv
Copy link

dvzrv commented Sep 24, 2022

I am not aware of any Linux distribution that has binary version of vst3sdk. We do not plan to vendor it in any way either. There is a proposed way to use Conan offline. If you are strictly against it I think it is better to build Audacity without vst3 support.

I'm not asking you to vendor it, but to allow me to build this with an existing version of vst3sdk without having to use conan (that would drag in even more dependencies that I would need to package).

It is already packaged for Arch Linux and used in the build process of multiple other packages.

@crsib
Copy link
Member

crsib commented Sep 24, 2022

It packages source code, so I don't really know how it is expected to be used. We can provide some sort of FindVST3SDK.cmake, however it will not work with the package from Arch.

@dvzrv
Copy link

dvzrv commented Sep 24, 2022

It packages source code, so I don't really know how it is expected to be used.

I guess not even steinberg really knows that... ;-)

As shown in the patch above it is possible to specify additional target_include_directories which at least take care of includes. As the cmake setup is rather complex/abstract I have a hard time figuring out where things go wrong though. in regards to

[ 77%] Building CXX object src/CMakeFiles/Audacity.dir/effects/VST3/AudacityVst3HostApplication.cpp.o
In file included from /usr/include/vst3sdk/pluginterfaces/base/funknown.h:20,
                 from /usr/include/vst3sdk/pluginterfaces/vst/ivstattributes.h:19,
                 from /usr/include/vst3sdk/pluginterfaces/vst/ivstmessage.h:19,
                 from /usr/include/vst3sdk/pluginterfaces/vst/ivsthostapplication.h:19,
                 from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.h:3,
                 from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.cpp:1:
/usr/include/vst3sdk/pluginterfaces/base/ftypes.h:30: warning: "UNICODE" redefined
   30 |         #define UNICODE 1
      |
In file included from /usr/include/wx-3.2/wx/defs.h:45,
                 from /usr/include/wx-3.2/wx/wx.h:14,
                 from /build/audacity/src/audacity-sources-3.2.0/src/AudacityHeaders.h:32,
                 from /build/audacity/src/build/src/CMakeFiles/Audacity.dir/cmake_pch.hxx:6,
                 from <command-line>:
/usr/include/wx-3.2/wx/platform.h:221: note: this is the location of the previous definition
  221 | #       define UNICODE
      |
/usr/include/vst3sdk/pluginterfaces/base/ftypes.h:34: warning: "_UNICODE" redefined
   34 | #define _UNICODE 1
      |
/usr/include/wx-3.2/wx/platform.h:218: note: this is the location of the previous definition
  218 | #       define _UNICODE
      |
/build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.cpp: In member function ‘virtual Steinberg::tresult AudacityVst3HostApplication::createInstance(char*, char*, void**)’:
/build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.cpp:67:37: error: ‘Steinberg::Vst::HostAttributeList::HostAttributeList()’ is private within this context
   67 |                 *obj = safenew Vst::HostAttributeList;
      |                                     ^~~~~~~~~~~~~~~~~
In file included from /build/audacity/src/audacity-sources-3.2.0/src/effects/VST3/AudacityVst3HostApplication.cpp:6:
/usr/include/vst3sdk/public.sdk/source/vst/hosting/hostclasses.h:92:9: note: declared private here
   92 |         HostAttributeList ();
      |         ^~~~~~~~~~~~~~~~~

We can provide some sort of FindVST3SDK.cmake, however it will not work with the package from Arch.

Why is that? I believe specifying what you need to compile from the vst3sdk to integrate into your own software makes a lot of sense, as it does not rely on binary blobs uploaded to some random repository by some random entity :-)
From the outside it is unfortunately completely unclear how conan uses this under the hood and what it provides to the build so that audacity can become a vst3 host.

@crsib
Copy link
Member

crsib commented Sep 24, 2022

From the outside it is unfortunately completely unclear how conan uses this under

I disagree with this statement. Conan is as transparent as possible. It builds the SDK from source. More over, we propose offline way to build all missing 3d party libraries from source.

I believe specifying what you need to compile

I don't think that a C++ source only package is something very common. AUR usually downloads sources from upstream, not packages it directly. SRPM is also used to build a binary form of the package, not to simply install sources.

What you propose is a very strange and fragile form of vendoring - we still need to manually build the SDK within our CMake, the only difference is that the sources are packaged by a distro for some reason.

as it does not rely on binary blobs uploaded to some random repository by some random entity

You miss my point. For FindVST3SDK or pkg-config you need a compiled package to be already installed somehow on the system. VST3 SDK is not a header only library, it must be built before it is used.

I will check on Monday, but I do not expect that making a proper VST3DSK package for Arch will be easy, if possible at all. Probably you can add SDK as a separate source bundle and install it into some temp location in PKGBuild before building Audacity though.

@hfiguiere
Copy link
Author

hfiguiere commented Sep 24, 2022

(Almost) Every other app I have seen using the VST3SDK on linux either had it as a git submodule, had an option to specify the path to the SDK, or did vendor it.

How do you build offline with conan? I haven't seen anywhere how to do it. So far Conan has made the process more difficult for the Flatpak, disabling it is the simplest way.

@dvzrv
Copy link

dvzrv commented Sep 24, 2022

I disagree with this statement. Conan is as transparent as possible. It builds the SDK from source. More over, we propose offline way to build all missing 3d party libraries from source.

Then where do you define which version of vst3sdk it downloads from where and builds with which options?

AUR usually downloads sources from upstream, not packages it directly.

I'm packaging for the official repositories. This has nothing to do with the AUR.
However, as background info: If you have a dependency a and need it for package b (during build or runtime) then you'd first need to build a and use it in the context of the build process of package b. This requirement never changes (the only difference in the AUR is, that it only provides the build scripts for packages, not the actual binary packages).

You miss my point. For FindVST3SDK or pkg-config you need a compiled package to be already installed somehow on the system. VST3 SDK is not a header only library, it must be built before it is used.

I don't think FindVST3SDK would be a good solution to this problem, but rather offering an option to provide the location to the SDK and handpicking the files one needs (this is already done in includes, why not for the .cpp files?) and adding them to the sources in the cmake setup.

The cmake "integration" provided by steinberg has been lacking and it does not offer a clear separation between build and test. Additionally it does not offer any form of installation target so it is completely unclear what would even be the consumable output of using their top-level CMakeLists.txt. When building the sdk just on its own it only outputs the static libraries, if building the examples:

drwxr-xr-x 2 dave dave  220 Sep 24 16:02 .
drwxr-xr-x 3 dave dave   60 Sep 24 16:01 ..
-rw-r--r-- 1 dave dave 299K Sep 24 16:02 libbase.a
-rw-r--r-- 1 dave dave  45K Sep 24 16:02 libpluginterfaces.a
-rw-r--r-- 1 dave dave 531K Sep 24 16:02 libsdk.a
-rw-r--r-- 1 dave dave  62K Sep 24 16:02 libsdk_common.a
-rw-r--r-- 1 dave dave 190K Sep 24 16:02 libsdk_hosting.a
-rw-r--r-- 1 dave dave  25M Sep 24 16:02 libvstgui.a
-rw-r--r-- 1 dave dave 6.4M Sep 24 16:02 libvstgui_standalone.a
-rw-r--r-- 1 dave dave 376K Sep 24 16:02 libvstgui_support.a
-rw-r--r-- 1 dave dave  23M Sep 24 16:02 libvstgui_uidescription.a

Without building the examples these static libraries do not exist and they would also never be installed as they do not have an install target.

The problem with building vst3sdk using its cmake setup in the context of another project is, that it takes very long (recursive clone takes a lot of space because of docs), is brittle as a build (by default the dummy plugins are dragged in and their build failed on me several times already). Doing this by default would make builds requiring vst3sdk take much longer and would also end up being more brittle (if done using default build configuration).

This concludes: It would be much more transparent and sane to rely on the files from the SDK that are actually needed and build with them.

I will check on Monday, but I do not expect that making a proper VST3DSK package for Arch will be easy, if possible at all. Probably you can add SDK as a separate source bundle and install it into some temp location in PKGBuild before building Audacity though.

As mentioned above, this is what the vst3sdk package does: We copy the required headers and source files from the upstream source tarball and then provide a pkg-config integration (as upstream does not do that). We do not provide any of the static libs as it is unclear how they would be used as consumables for something else.

@falkTX
Copy link

falkTX commented Sep 24, 2022

is building without vst3 an option?

VST3 SDK still has licensing issues, for which Fedora/Red Hat are refusing to package it.

@hfiguiere
Copy link
Author

hfiguiere commented Sep 24, 2022

-Daudacity_has_vst3=Off

@hfiguiere
Copy link
Author

hfiguiere commented Sep 24, 2022

BTW a -Daudacity_vst3sdk=path/to/vst3sdk would go a long way to fix this.

@falkTX
Copy link

falkTX commented Sep 24, 2022

BTW a -Daudacity_vst3sdk=path/to/vst3sdk would go a long way to fix this.

afaik that would only work for vst3 sdk copies, as in full source.
not as installed package (with split headers and libraries)

@dvzrv
Copy link

dvzrv commented Sep 24, 2022

From what I understand the following libraries (i.e. libbase.a, libpluginterfaces.a, libsdk_common.a, libsdk_hosting.a) are being built by conan at some point and are then linked against:

audacity/src/CMakeLists.txt

Lines 1227 to 1230 in 482e0aa

$<$<BOOL:${${_OPT}has_vst3}>:vst3sdk::base>
$<$<BOOL:${${_OPT}has_vst3}>:vst3sdk::pluginterfaces>
$<$<BOOL:${${_OPT}has_vst3}>:vst3sdk::sdk_hosting>
$<$<BOOL:${${_OPT}has_vst3}>:vst3sdk::sdk_common>

In the current build system setup there is no alternative to using conan for that. The respective sources however could also be built in the context of the private sources (

list( APPEND SOURCES
) AFAICT.

@hfiguiere
Copy link
Author

hfiguiere commented Sep 24, 2022

BTW a -Daudacity_vst3sdk=path/to/vst3sdk would go a long way to fix this.

afaik that would only work for vst3 sdk copies, as in full source. not as installed package (with split headers and libraries)

which is what "we" want (it's IMHO the simplest solution), it then would be added as a cmake subproject (that's what other do, no idea how to actually do it, it is CMake)

@crsib
Copy link
Member

crsib commented Sep 26, 2022

The respective sources however could also be built in the context of the private sources

This is definitely not an option.

it then would be added as a cmake subproject

We will see what can be done here.

Then where do you define which version of vst3sdk it downloads from where and builds with which options?

The recipes are transparent and introspectable. For example, using conan get vst3sdk/3.7.3 (you need Audacity remotes to be added which is done by Audacity CMake script). Alternatively you can observe recipe here: https://github.com/audacity/conan-recipes/blob/main/vst3sdk/all/conanfile.py

@crsib crsib added this to the Audacity 3.2.x milestone Sep 26, 2022
@crsib
Copy link
Member

crsib commented Sep 27, 2022

@dvzrv is there a reason why you package everything but CMakeLists.txt files for the Arch package? This is quite a deal breaker, we definitely won't be adding files to our source lists manually nor copying CMakeLists.txt to our source tree.

Otherwise I am mostly ready to open a PR that allows to build without Conan, but only if full vst3sdk is present, which is not the case for the Arch Linux package.

@dvzrv
Copy link

dvzrv commented Sep 29, 2022

@crsib None of the packages so far required the CMakeLists.txt file. That's basically the only reason why it is not included. However, I can change that in an update.

Would your change also require the *.cmake files that steinberg ships alongside?

@crsib
Copy link
Member

crsib commented Sep 29, 2022

Is there a reason to strip the SDK? If there is one, we can make Arch specific hackaround, but it will break at some point surely. Probably not in the mid-term though.

@dozzzzer dozzzzer self-assigned this Sep 29, 2022
@AnitaBats AnitaBats removed this from the Audacity 3.2.2 milestone Oct 3, 2022
@AnitaBats AnitaBats added this to the Audacity 3.2.1 milestone Oct 3, 2022
@hfiguiere
Copy link
Author

hfiguiere commented Oct 4, 2022

So which commits are relevant?

@falkTX
Copy link

falkTX commented Oct 4, 2022

So which commits are relevant?

Seems to be the stuff in #3703

@hfiguiere
Copy link
Author

hfiguiere commented Oct 4, 2022

ah yeah, I missed it. thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Build / CI Bugs and tasks regarding builds and Continuous Integration community build bugs Bug in non-official builds, such as distro-repos or self-compiled ones
Projects
Status: Done
Development

No branches or pull requests

7 participants