Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collection repository #1

Open
jatinchowdhury18 opened this issue Dec 19, 2021 · 17 comments
Open

Collection repository #1

jatinchowdhury18 opened this issue Dec 19, 2021 · 17 comments

Comments

@jatinchowdhury18
Copy link
Contributor

Cloned from jatinchowdhury18/AnalogTapeModel#239:

@Bluezen wrote:

Hi, this is not specific to this plugin but didn't know where to ask.

Some plugin developers who have multiple plugins put them together into a collection. This makes it easier to package all the plugins as a bundle / one package.
So there could be one repository called something like "Chow Plugins", which contains all of your plugins as submodules.
As an example, take a look at x42-plugins:
https://github.com/x42/x42-plugins

Is there something like this planned?

@jatinchowdhury18 wrote:

Having a "collection" repo is not really something I had thought about, but I definitely see the benefits that it could have from a "packaging" standpoint.

Currently, there is a "releases" repo, which I use for making nightly builds of each plugin. The repository only contains a GitHub Actions pipeline, build scripts for each plugin, and a little bit of metadata for each plugin... I wonder if that process could be simplified by using submodules instead. Then from there maybe we could add extra scripts for building/packaging multiple plugins at a time.

Anyway, this is a cool idea... I'll have to think about it a little more, and I'm definitely open to suggestions for how this sort of thing could be structured. I'd also love suggestions for renaming the "releases" repo, since it may now be used for more than just running nightly builds. For now, I'll also move this issue into that repo.

@Bleuzen
Copy link
Contributor

Bleuzen commented Jan 17, 2022

I just started testing around if this is easy / possible at all:
https://github.com/Bleuzen/Chowdhury-Plugins
Sure I was missing something (never used cmake before)...
with the simple setup with all Plugins as submodules there is the problem that submodules of the plugins get cloned multiple times and build target conflicts.
Would be nice if all plugins could use just one shared JUCE, chowdsp_utils, ... submodule.
Do you think this is possible with your plugins or do they need all their own JUCE, chowdsp_utils, ... (submodule) versions?

I'd also love suggestions for renaming the "releases"

I think "releases" is not that bad. But another suggestion: rename it to "Chowdhury-Plugins" since this sounds like a plugin collection of you and basically it contains all your plugins so would make sense.

Also, an idea for the releases:
Every time there is a new Plugin update (or multiple ones in a short time/same day) update the submodule(s) in this repo and create a new release/tag, named just after the date ("v2022.01.17"), since plugins have their own individual versions.

@jatinchowdhury18
Copy link
Contributor Author

Ooh, thanks for taking a look at this!

Yeah, the submodule thing is kind of tricky, since some of the plugins are on older versions of JUCE, that may not be immediately compatible with the plugin code. For using the repository in a build pipeline, I bet it might be okay, since we could only clone the submodules for the plugin we want to build in that run of the pipeline. Another idea could be to use a different package manager like CPM, perhaps in tandem with git submodules.

That name suggestion sounds good as well, although I think I would prefer /ChowDSP-Plugins or /Chowdhury-DSP-Plugin (although maybe that's a bit too long).

Updating submodules in the repo should be pretty straightforward. I'm not sure how easy it is to automate creating GitHub releases, but I imagine that should be pretty do-able as well.

I should have some time on Thursday, so I can take a closer look then.

Thanks,
Jatin

@jatinchowdhury18
Copy link
Contributor Author

Okay, so I was able to do some testing in the v2 branch, with just the BYOD plugin for now. Basically, an approach that can work is to clone the repo without submodules, then use the bash scripts within each plugin's directory to build that plugin (including fetching the relevant submodules). This does mean that some submodules will be re-cloned multiple times (if you're building multiple plugins), but I think that's okay because 1) most of the submodules (except for JUCE) are pretty small in size, and 2) it makes it less likely to have version discrepancies between two instances of the same submodule.

So from here, I think the process will look like this:

  • Get build scripts working for BYOD
    • Edit build scripts inside the repo
    • Make sure the relevant environment variables are set in the GitHub Actions pipeline
    • Make sure the generated installers work correctly
  • Get build scripts working for all plugins
  • Merge v2 branch to main
  • Update nightlies.sh script to update submodules instead of metadata config

@Bleuzen
Copy link
Contributor

Bleuzen commented Jan 24, 2022

Basically, an approach that can work is to clone the repo without submodules, then use the bash scripts within each plugin's directory to build that plugin (including fetching the relevant submodules).

Some linux distributions don't allow internet access during build time. However when building all plugins anyway cloning with all submodules in the first / downloading step should be fine.

This does mean that some submodules will be re-cloned multiple times (if you're building multiple plugins), but I think that's okay because 1) most of the submodules (except for JUCE) are pretty small in size, and 2) it makes it less likely to have version discrepancies between two instances of the same submodule.

This is also okay. Just asked about it in case it could be optimized. Maybe one day, but I don't think this is a dealbreaker.


So I had a look at your v2 branch and see you currently build using bash scripts. Would recommend to stay with a standardized (and the current cmake) build system. This is what most linux build tools support ootb, have their settings for preconfigured, can automatically handle and doesn't force the dependency on bash. Would be nice to have all the plugins integrated into one cmake project and only set the build targets if not all plugins should be built.

Something like, in the meta repos base dir:

cmake -DCMAKE_BUILD_TYPE=Release
make

to build all targets/plugins.
And:

cmake -DCMAKE_BUILD_TYPE=Release
make CHOWTapeModel_VST3 CHOWTapeModel_LV2 ChowMatrix_VST3 ChowMatrix_LV2

to build only CHOWTapeModel and ChowMatrix without the standalone.

Do you think this could be possible this way, using only cmake / no bash?

I tried that in my repo linked above. Used cmakes add_subdirectory for that but this resulted in build target conflicts, since multiple plugins have the same submodules/targets with same name. Maybe renaming the plugins submodules / buildtargets for dependencies to be prefix with the plugins name could work (while ugly) or maybe there is another cmake option doing something similar as add_subdirectory but does handle the plugins separately, only exposing their top level build targets and not care about their sub-targets names.

@jatinchowdhury18
Copy link
Contributor Author

Cool, yeah I think we've got a good path forward starting from here.

I think I would like to keep the bash script interface, since I think it integrates better with GitHub Actions, and makes it easier to generate installers for Windows/Mac, but having a CMake interface would certainly be nicer for building locally, and probably for most packaging systems as well.

I'll have to do some testing (might not be able to until later this week), but I think the top-level CMakeLists.txt should look something like this:

cmake_minimum_required(VERSION 3.15)
project(ChowDSP-Plugins VERSION 2.0.0)

# This should fix the build target conflicts
set(ALLOW_DUPLICATE_CUSTOM_TARGETS TRUE)

# This plugin will be configured by default.
option(BUILD_CHOWTAPE "Build the ChowTape plugin" ON)
if(BUILD_CHOWTAPE)
    message(STATUS "Configuring build for ChowTape")
    add_subdirectory(plugins/ChowTape)
endif()

# This plugin hasn't been released yet, so it will NOT be configured by default.
option(BUILD_BYOD "Build the BYOD plugin" OFF)
if(BUILD_BYOD)
    message(STATUS "Configuring build for BYOD")
    add_subdirectory(plugins/BYOD)
endif()

Does that seem reasonable?

@Bleuzen
Copy link
Contributor

Bleuzen commented Jan 26, 2022

I pushed your suggested CMakeLists.txt changes to my repo now and tried building that. It seems to work when only building one plugin at a time but with multiple ones there are still conflicts, here for example when I try to configure build of CHOWTAPE and CHOWCENTAUR:
https://pastebin.com/raw/1tteZwCX

@jatinchowdhury18
Copy link
Contributor Author

Ah yeah, it looks like the JUCE targets are conflicting... I wonder if calling add_subdirectory() with EXCLUDE_FROM_ALL would help? Just looking at the docs at the moment.

By the way, sorry I've been pretty swamped this week, but I will definitely have some time to dig into this more fully on the weekend. Thanks!

@Bleuzen
Copy link
Contributor

Bleuzen commented Jan 27, 2022

I wonder if calling add_subdirectory() with EXCLUDE_FROM_ALL would help?

Unless I didn't use it correctly, doesn't seem to help :/
Pushed to my repo so you can review.
Output:
https://pastebin.com/raw/Gtpq0RxN

By the way, sorry I've been pretty swamped this week, but I will definitely have some time to dig into this more fully on the weekend.

No problem. Thank you :)

@jatinchowdhury18
Copy link
Contributor Author

jatinchowdhury18 commented Jan 30, 2022

Hey! So I was able to spend some time on this today. It looks like what we've been looking for in the CMake world is not add_subdirectory(), but ExternalProject_Add(). I've been able to get a couple of plugins to build using this approach on the v2 branch. Here's the CMakeLists.txt. I think this is the right approach, and the majority of work from here will be in figuring out how to pass the relevant arguments through the CMake config.

That said, with CMake external projects, it might be a bit redundant to be using Git submodules, since the external projects can just as easily download the source using the branch/commit specified in the metadata file. Anyway, just something to think about...

@Bleuzen
Copy link
Contributor

Bleuzen commented Jan 30, 2022

It looks like what we've been looking for in the CMake world is not add_subdirectory(), but ExternalProject_Add().

Hi, yeah this looks good, will test it later.

I think this is the right approach, and the majority of work from here will be in figuring out how to pass the relevant arguments through the CMake config.

Yes, building only plugins or only standalone should be possible.

That said, with CMake external projects, it might be a bit redundant to be using Git submodules, since the external projects can just as easily download the source using the branch/commit specified in the metadata file.

Remember: Some build systems don't allow internet access during build time. Not sure how they handle this ExternalProject_Add in other cases. Just a thought. Having cmake downloading the plugins at build time might break things. So if it has no downsides I think keeping git submodules could still make sense.

@Bleuzen
Copy link
Contributor

Bleuzen commented Jan 30, 2022

Hm tested this with only building chow tape

cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_CHOWMATRIX=OFF -DBUILD_CHOWKICK=OFF -DBUILD_CHOWCENTAUR=OFF -DBUILD_CHOWTAPE=ON
make

and it currently fails while trying to install JUCE.

[...]
Saving preset 21/22... done!
[ 99%] Built target CHOWTapeModel_LV2
[100%] Linking CXX shared module CHOWTapeModel_artefacts/VST3/CHOWTapeModel.vst3/Contents/x86_64-linux/CHOWTapeModel.so
[100%] Built target CHOWTapeModel_VST3
[ 87%] Performing install step for 'ChowTape_Package'
Consolidate compiler generated dependencies of target CHOWTapeModel_LV2_lv2_ttl_generator
Consolidate compiler generated dependencies of target RTNeural
Consolidate compiler generated dependencies of target BinaryData
[  1%] Built target CHOWTapeModel_LV2_lv2_ttl_generator
[ 33%] Built target BinaryData
[ 34%] Built target RTNeural
Consolidate compiler generated dependencies of target juce_plugin_modules
[ 48%] Built target juce_plugin_modules
Consolidate compiler generated dependencies of target CHOWTapeModel
[ 73%] Built target CHOWTapeModel
Consolidate compiler generated dependencies of target CHOWTapeModel_Standalone
Consolidate compiler generated dependencies of target CHOWTapeModel_VST3
Consolidate compiler generated dependencies of target CHOWTapeModel_LV2
[ 95%] Built target CHOWTapeModel_Standalone
[ 97%] Built target CHOWTapeModel_LV2
[100%] Built target CHOWTapeModel_VST3
Install the project...
-- Install configuration: ""
-- Installing: /usr/local/include/JUCE-6.1.2/modules/juce_analytics
CMake Error at modules/juce/modules/cmake_install.cmake:46 (file):
  file INSTALL cannot make directory
  "/usr/local/include/JUCE-6.1.2/modules/juce_analytics": No such file or
  directory.
Call Stack (most recent call first):
  modules/juce/cmake_install.cmake:47 (include)
  modules/cmake_install.cmake:47 (include)
  cmake_install.cmake:47 (include)


make[3]: *** [Makefile:100: install] Fehler 1
make[2]: *** [CMakeFiles/ChowTape_Package.dir/build.make:102: ChowTape_Package-prefix/src/ChowTape_Package-stamp/ChowTape_Package-install] Fehler 2
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/ChowTape_Package.dir/all] Fehler 2
make: *** [Makefile:91: all] Fehler 2

Still have to disable this install step I guess.

@Bleuzen
Copy link
Contributor

Bleuzen commented Jan 30, 2022

I think I found a way around that by using STEP_TARGETS.
See my updated CMakeLists.txt

This works now, can build all plugins or only selected ones. Nice :)
One option is still missing: build all (default), plugins only, standalone only

@jatinchowdhury18
Copy link
Contributor Author

Nice! Yeah, STEP_TARGETS looks like the right thing here.

For only building only standalone targets, I was able to start on this CMake config today. Haven't been able to test it much, plus it's a lot more repetitive that I'd like, but maybe it's a start:

option(BUILD_STANDALONE_TARGETS_ONLY "Build only standalone targets" OFF)

option(BUILD_BYOD "Build the BYOD plugin" ON)
if(BUILD_BYOD)
    message(STATUS "Configuring build for BYOD")
    set(BYOD_DIR "${CMAKE_SOURCE_DIR}/plugins/BYOD/BYOD")

    if(BUILD_STANDALONE_TARGETS_ONLY)
        ExternalProject_Add(BYOD_Standalone
            SOURCE_DIR "${BYOD_DIR}"
            CMAKE_ARGS "-DBUILD_RELEASE=ON"
            BUILD_COMMAND ${CMAKE_COMMAND} --build . --config Release --target BYOD_Standalone
            STEP_TARGETS build
        )
    else()
        ExternalProject_Add(BYOD_All
            SOURCE_DIR "${BYOD_DIR}"
            CMAKE_ARGS "-DBUILD_RELEASE=ON"
            BUILD_COMMAND ${CMAKE_COMMAND} --build . --config Release --target BYOD_All
            STEP_TARGETS build
        )
    endif()
endif()

@Bleuzen
Copy link
Contributor

Bleuzen commented Feb 1, 2022

I just made test builds to compare: building all targets (default) vs. building only LV2+VST3.

Maybe it is not even worth it, it makes a much smaller difference in build time than I thought it would.
Building all targets with the Standalone and deleting the Standalone after build to only get the plugins seems a viable option here since it only barely makes a difference compared to building only the plugins in the first place.

Can still keep this in the back of our head and think about it later for optimization, but I wouldn't see this as a dealbreaker and think it would be okay to not have this option for the initial release.

So I think it is fine now from my side. When you are ready too you can merge it with your github workflows stuff and make a first release.

@jatinchowdhury18
Copy link
Contributor Author

jatinchowdhury18 commented Feb 2, 2022

Yeah, that's a good point: most of the plugin code is built as a static library, which is then shared between the different plugin format wrappers, so adding/subtracting a plugin format won't affect the built time too much.

I think the next steps from here are:

  • add all the submodules to this repo
  • add the top-level CMakeLists.txt to the repo
  • edit the nightlies bash script to update the submodules automatically
  • do a couple test runs just to make sure
  • add some extra packaging scripts if that's helpful

I should have some time this weekend to hopefully get through the first three steps.

@jatinchowdhury18
Copy link
Contributor Author

Alright! It looks like the nightly build script is now updating the submodules in this repo automatically! I'll try to get a few more tests in over the next couple of days, but it looks like the basic workflow is in place.

@Bleuzen
Copy link
Contributor

Bleuzen commented Feb 18, 2022

I tried building that with flatpak and it fails at cloning this repo.
End of the log:

Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/BYOD/modules/chowdsp_juce_dsp'...
Submodule path 'modules/chowdsp_juce_dsp': checked out '0e8f18b4be33bdf38e07dd537a22e85d764ec4a4'
Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/BYOD/modules/sst-cpputils'...
Submodule path 'modules/sst-cpputils': checked out '916c8cdf516ec13a11145b8efa6b7ee98146bdca'
Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/ChowPhaser'...
Submodule path 'ChowPhaser': checked out 'f738a7f400891233f4772c26f14403c940e17744'
Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/ChowPhaser/modules/JUCE'...
Submodule path 'modules/JUCE': checked out 'f00a420bc348d79c4688b33b2b905b8ca0f25a3a'
Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/ChowPhaser/modules/foleys_gui_magic'...
fatal: git upload-pack: not our ref 9cab1a439d86cbf7942981b24ab59c60a55b7dcd
fatal: remote error: upload-pack: not our ref 9cab1a439d86cbf7942981b24ab59c60a55b7dcd
fatal: Fetched in submodule path 'modules/foleys_gui_magic', but it did not contain 9cab1a439d86cbf7942981b24ab59c60a55b7dcd. Direct fetching of that commit failed.
fatal: 
Error: module ChowDSP-Plugins: Der Kindprozess wurde mit Status 128 beendet

Edit:
Created a new issue for that here:
jatinchowdhury18/ChowPhaser#18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants