[Discussion] VCVRack Package Manager

[Miserlou]

First of all - I love this project, this is the greatest open source audio project since Audacity. I think this is going to seriously change electronic music.

One thing that I'd lovvvvve to have for Rack is the ability to browse and install new modules from within Rack. Currently it requires a fair bit of digging all over the internet to find good user-made modules that are built for the right architecture, etc.

I think it'd be awesome if there were something like Atom Package Manager or NPM, etc, but for VCVRack - vpm maybe? Then I could publish my own modules and play with other users's modules, etc.

Anyway, this is a pretty big ask, but I think it could be a huge boost for the project. Would love to know your thoughts!

[AndrewBelt]

A snarky note: I can really tell who's coming from the HN crowd from yesterday. 😛

You're welcome to proceed, but as the name VCV is in the process of being trademarked, please inform me of the methods you choose to use the name.
However, I should note that your efforts will be overwritten in 2-3 months as I begin to accept third-party plugins to the "VCV store/website". The first batch will be in a week or two.

[Miserlou]

I'll have you know that I've been using Rack since 0.3! HN just let me know about the new 0.5, which is coming along very nicely!

I'm certainly not trying do to anything to "side-step" your copyright, I really just want a way to publish my own modules and install other user-created modules. Since it seems like you've already had that on the roadmap, I think we can close this, although I would like to link it to #445 since I think designing a useful manifest file for modules could make some of these easier for everybody.

[AndrewBelt]

Reopening this because I've changed my mind. Instead of myself maintaining only a few select third party plugins on the VCV store with a high standard, it would be a better idea to collect all plugins into a single repository with the only standard that it is not malicious software or an IP violation. Plugins with a high enough standard will be featured on the VCV store in a rotation.

I will create it under the VCV Github organization, and you and a few others can be a maintainer if you'd like.

In the other thread, you mentioned that plugin packages should contain a manifest file with its own metadata. I don't think plugins themselves should include this information, but the package manager software (built into Rack itself?) should "own" them in the same way that Linux package managers own their packages on the filesystem. In Linux packages, the software (upstream) doesn't include metadata about itself, but the package manager software keeps a copy of all the available plugins and all installed plugins.

So let's decide on a JSON manifest format in this thread, and the package manager software itself.

[AndrewBelt]

Plugin manifest repository proposal

(Will update this post as changes are suggested.)

File structure of repo (can't decide on a name, "community"?):

• README.md
• AudibleInstruments.json
• Befaco.json
• <plugin 3 slug>.json
• <plugin 4 slug>.json
• etc.

Each JSON package file should look like the following.

{
	"slug": "AudibleInstruments",
	// optional, if not present, name is set to slug
	"name": "Audible Instruments",
	// optional, let's not parse this against a dictionary of licenses or anything, just display the verbatim text as needed
	"license": "BSD 3-clause",
	// Latest version, without the "v" prefix
	"version": "0.3.0",
	// optional
	"homepage": "https://vcvrack.com/",
	// optional
	"manual": "https://vcvrack.com/manual/AudibleInstruments.html",
	// optional
	"source": "https://github.com/VCVRack/AudibleInstruments",
	// ZIP file should contain a folder equal to the slug or your entry will be removed
	"download": "https://example.com/AudibleInstruments-0.5.0.zip",
	// used only if your plugin is sold through the VCV store
	"productId": "1234567890",
	// optional, if not present, all are assumed
	"arch": ["win", "lin", "mac"],
	// optional
	"sha256": "5993dbe15af246092678e00897dc7a4a1e2dabc494b83412fe31200d5bb58305",
}

Note that there is redundant information in plugin.hpp. I'd like to get rid of as much as I can and perhaps rely on the package manager to provide this information as needed.

Should it handle multiple release urls if you want to split builds for each OS? No, not supporting this gives an incentive for developers to properly package their builds. Someone should write a zipmerge.sh script for the community that actually works.

Should it support old versions for the hell of it? No package manager I'm aware of does this, would be too messy and too time consuming for the plugin developer.

[AndrewBelt]

Plugin manager proposal

Rack itself will handle the downloading of plugins through api.vcvrack.com, as it currently does with official VCV store plugins.
This VCV server will keep a synchronized copy of the community repo and convert the JSON data to an API.

Installed packages should be associated with your VCV account. (Yes, not the favorite for maybe 0.2% of users, but they always have the option of manually downloading the plugins from the download URLs and placing them in the plugins/ directory.)
This allows the package manager selection to be web-based on https://vcvrack.com/, which is nice because HTML is a much more familiar method for shopping for plugins, both free and nonfree, and clicking URLs is better UX when you're already using a browser. (Side note: why doesn't my operating system also have a web-based package manager already?!)

If you click "Add" or "Purchase" on the web package manager, Rack will highlight the "Refresh Plugins" button to alert the user that updates are available.

The logic for determining whether the "Refresh Plugins" button is highlighted:

• If the VCV account has a plugin slug that is not loaded into Rack (and it is available for the current OS, assuming we add the "arch" property in the manifest)
• Or, if the VCV account says that a loaded plugin has a version which is != to the loaded plugin

We need a way to handle download links to premium plugins.
The download link for premium plugins obviously cannot go in the manifest, so I suppose I could handle each of these as a special case internally through my checkout system.

[djpeterso23662]

Possible implementation notes:

Two-levels of feed. First is the subscription, second is the package. Support multiple subscription feeds. Subscription feeds hosted on GitHub.

Packages downloaded direct from GitHub releases.

[AndrewBelt]

I'm not sure what you mean by subscription feed.

The download URL should be whatever the developer wants, but GitHub release hosting is usually the best choice if you host the source on GitHub.

[jeremywen]

One of the reasons it is hard to "properly package their builds" is that we don't all have builds systems setup for all operating systems. The various operating system builds are basically crowd sourced. So when somebody sends a dev a zip of the build from another OS it gets added to the release. Please keep this in mind. It is not because we don't know how to merge zip files.

[AndrewBelt]

@jeremywen I was referring to using .rar files or zip bombs instead of running make dist. Or, using zipmerge from the ziputils package, which merges zip files incorrectly, resulting in files with broken permissions which require sudo rm to remove or messing with chattr. Since zipmerge is broken, we need a script that simply unzips a list of files into a temporary directory, overwriting as necessary, and then zipping them from scratch into a filename given by the last argument.

[jeremywen]

Ok, I misunderstood then. However, I think my comment is still something to consider.

[sanderbaan]

We could also think of something like Travis CI for automatic building on all platforms since the software is open source? this would not work for commercial closed source plugins though

[AndrewBelt]

@jeremywen Sure, I suppose having a single download URL will also encourage developers to collect contributed builds into a merged ZIP file as well.

@sanderbaan Feel free to make a travisci script that does this. Woulld be interested in the results. I looked into it months ago when Rack was a bit harder to build and the task looked too daunting, but now the build system is mostly self-contained on all platforms.
And yes, you're right. It's very expensive at $60/month.

[sanderbaan]

Enzo has some experience with this maybe he has a easy way ;)

[jeremywen]

@AndrewBelt This will also encourage me to setup a linux environment to build/test in 😄

[IohannRabeson]

Travis-ci is free for open source projects. I use it to test builds for my plugins on linux and osx, the only things missing is a script to upload the build output somewhere.
Unfortunately, Windows is not supported by Travis-ci. I tried to use Appveyor without success (I did not insist too much). You can see the .travis.yml here (or use it also if someone want, just change the Simple repository url by your own): https://github.com/IohannRabeson/VCVRack-Simple/blob/master/.travis.yml.

[lnikj]

Slight (devil's advocate) tangent - what mechanism (if any) would be in place to try to weed out malicious or otherwise non functioning code or would it just be a 'use at your own risk' free for all?

I recall the recent showstopper Andrew just prior to 0.5.0. If we had not alerted the other dev then his package could/would have crashed Rack.

If a user batch installed a bunch of binaries, rather than one at a time, they would have no idea which plugin had caused a problem (if there was one).

You might say that there is little difference to the situation as it stands in this regard .... until you start getting showered with complaints and support requests that is ....

Anyway, I am sure all this has crossed your mind already. Please move this comment if this would be
better discussed elsewhere.

[AndrewBelt]

weed out malicious

I imagine with 3-4 maintainers of the plugin manifest repo, it would take no more than 30 minutes to discover and remove malicious plugins that were accidentally added or their download files changed.

otherwise non functioning code ...

It would be unlikely for an accident to cause Rack to crash upon loading the plugin. Creation of plugins would be handled with #467

I recall the recent showstopper Andrew just prior to 0.5.0. If we had not alerted the other dev then his package could/would have crashed Rack.

Link?

[lnikj]

I knew you would have thought it through!

#413

[thetasine]

Unfortunately, Windows is not supported by Travis-ci. I tried to use Appveyor without success (I did not insist too much). You can see the .travis.yml here (or use it also if someone want, just change the Simple repository url by your own): https://github.com/IohannRabeson/VCVRack-Simple/blob/master/.travis.yml.

It's my understanding that you can cross-compile for Windows/MinGW toolchain using some scripts for Travis under Linux. Basically install the mingw packages and runtime that includes the Windows header files.

I've never done this myself however. There are a number of .yml scripts and discussion that do so in and about the net.

[jeremywen]

@AndrewBelt

• How does a third party developer publish an updated release?

• And what is "synchronized copy of the community repo"?

• Also how would a developer register/apply?

I am just looking at all this from my (3rd party developer) perspective.

Edit: Removed questions.

[AndrewBelt]

Good questions.

• Developers would send a PR with the updated JSON file to the community repository when they bump the version, much like what they do with https://github.com/VCVRack/Rack/wiki/List-of-plugins. One of the repo maintainers would then merge it. Using Github's "edit this file" feature, you don't even need to clone, commit, and push the repo.
• The "community" repo is the repo of JSON manifests. It doesn't contain anything else, like source or binaries. Closed source plugins are fine too---just leave out the "source" property.
• Just send a PR to the repo with a file addition. Or if you're asking to be a repo maintainer, that's fine too. I need 2-3.

[jeremywen]

Ok this is all making more sense now. More like a way better version of that dumb script I wrote when there was only a few third party modules.

[jeremywen]

I am willing to help maintain that repo. Or wherever you need me to help ...maybe 😄

[AndrewBelt]

A similar project is Homebrew, where all package manifests are dumped into one folder. This will do the same. https://github.com/Homebrew/homebrew-core/tree/master/Formula

[jeremywen]

Yes, I have used homebrew. I have just never implemented something like it before. This is so much more clear now. Thank You.

[AndrewBelt]

You hinted at another question: What I meant by "synchronized" repo is that upon every commit (either with Github webhooks or with 5-minute interval polling), the VCV server will pull the latest commit, so that the VCV API handling the plugin management is always up to date.

[jeremywen]

Ok so I would:

1. make a zip including all os builds.
2. make a pull request to update this plugin manifest repo with my new version number and download link
3. get pull request merged by a plugin repo maintainer

VCV server would:

1. sync with the plugin manifest repo at an interval or by webhooks

A few minutes later the user would:

1. Refresh plugins
2. VCV server would say JW-Modules v2 on server > v1 on user's machine so download

Edit: Added pull request.

[AndrewBelt]

@gratrix Start a new "Plugin Build Team" thread? I'm fine with it as long as it's the developer's choice to use the team's builds in their plugin's "download" URL.

[lnikj]

@gratrix [Posting here as I am out for the day now] As I am already building everything from source you can sign me up as a Mac builder if you like.

[AndrewBelt]

After implementing this in full, there is no longer a reason to make Rack's "local" folders visible to the user, so I will switch to hidden directories in #265

[AndrewBelt]

Last call for JSON suggestions, creating community repo in 15 minutes.

[AndrewBelt]

https://github.com/VCVRack/community

[AndrewBelt]

@Miserlou @jeremywen I've invited you to the repository. Maintaining is simple, just read the README and accept PRs which follow them. Suggest fixes to PR submitters otherwise.
As a bonus, it would be extremely helpful if you could edit some of the JSON files that are already present. I just scraped data off of the wiki list, but most of the download URLs are not direct links to the ZIP files, and the slugs may be incorrect.

[jeremywen]

ok thanks I will go through them tonight after work and see what I can correct. pfft work...

[jeremywen]

@AndrewBelt I am sure you have already thought this through but here is my question. How will users select or opt-in opt-out of certain modules? What if they don't want some of the subpar modules or they think JW-Modules suck? Will there be some popup or dialog or something? Or is this a non-issue, and they get whatever is available?

[AndrewBelt]

They will be able to add and remove them from the future vcvrack.com/manager page. All modules are not added to your account by default, except Fundamental, which comes with Rack.

[jeremywen]

Perfect! Will there be screenshots on that page like the current ones or something?

[AndrewBelt]

It'll be a pretty dry design, similar to the old wiki list except with Add buttons and a login form. It is not meant for marketing.

[jeremywen]

Ok maybe in the future there could be a json entry for a certain size image or something. That would be helpful to users.

[sanderbaan]

btw guys, i might have missed this but when is this going to be working in the vcv software? is that going to be in a next update?

[AndrewBelt]

My plan is that it'll be added to Rack v0.5.1.

[Dewb]

Just wanted to throw in my .02usd and say that in my limited experience, creating your own package manager backend is generally more trouble than it's worth.

Creating a package manager from an initial set of minimal requirements seems like a straightforward and tractable problem, but it has a lot of subtleties and can, surprisingly quickly, become a time sink and source of tech debt. What's worse, the requirements from one package manager use case to another aren't that different, so there's not a huuuge payoff to having rolled your own.

It might be a good idea to evaluate whether an existing stack could serve VCVRack's needs, maybe not now, but in the future if the backend requirements start to grow. Off the top of my head, both Nuget and Conan support C++ packages and have open-source, self-deployable server implementations. (Nuget is generally pretty good, I don't have any direct experience with Conan.)

[AndrewBelt]

I don't understand your point. It'll be 60 lines of Node/Express and take an hour and 30 minutes to write, since much of the code already exists in both Rack (the client) and api.vcvrack.com (the server).
I'm just waiting on some free time, and the VCV-community repo to be seeded with initial manifests.

[jeremywen]

I completely agree with @AndrewBelt Also, in my experience integrating some else's overly complex system into yours cause cause even more trouble than it's worth. I don't know how many libraries I have used and ended up regretting it because I depended on it so heavily.

[Dewb]

I'm just saying that I have been down this road before, and at the time, we said the same things you're saying now, and we later regretted it. :)

I'm probably being overly paranoid. If you never do anything more complicated than "install the latest version of this self-contained plugin package" then you should be fine.

But if at some point you want to support things like a patch repository, where you can pull down a sample patch and all of the plugins that the patch uses, or keep multiple versions of plugins used by different packages on the local hard drive and load the versions on-demand, then using someone else's already built and tested dependency resolution infrastructure could start to look attractive.

[AndrewBelt]

I'm probably being overly paranoid. If you never do anything more complicated than "install the latest version of this self-contained plugin package" then you should be fine.

Then I'm fine.

But if at some point you want to support things like a patch repository

God, no

multiple versions of plugins

Ahhhhh, stop

[MichaelPilyavskiy]

Hi. Very interesting thread for me since I raised idea of collecting user created script for REAPER 2 years ago. There were previously scattered through different web places. Today there is ReaPack extension. It is a package manager for all (supported) kind of user-created stuff. A lot of its good features can be usefull for VCVRack also.

• there is dedicated common repository for relatively simple user created stuff (we did it like this ), maintained by some users, for VCV it could be a pack for simple modules/utilities easy to learn (well commented),
• some big single user repositories bundled into manager as trusted (potentilly Autodafe, Hetrick)
• in ReaPack scripts syncronized online by dedicated action (implemented already via 'Refresh plugins')
• there is list of 3-rd party small (non-bundled repos ) on website so users know where to find other repos (currently only GitHub page for small VCV compatible repos, there is no any mention/promotion on website or inside VCV)
• in ReaPack adding repos performs manually via copy index file raw link (index used for performing/reverting updates of single script versions), in VCV can be implemented same way like a button "Import new plugin pack" and, say, link to repository at least
• there is a possibility to revert back to previous version of plugin (like if newer version has bug)
• there is an about page for every repo where user can see where to put bug reports, feature requests and donation to plugin/script author

I donn know what will be with backward compatibility, seems youll need to mark installed plugins as grey if their version lower than current VCVRack version (so they willn`t be uninstalled i.e. awaiting update from author and unavailable)

Sorry if i sounds unclear (not English speaker at all).
Michael.

[AndrewBelt]

there is dedicated common repository for relatively simple user created stuff (we did it like this ), maintained by some users, for VCV it could be a pack for simple modules/utilities easy to learn (well commented),

I think the Fundamental plugin serves this purpose well. If you start with 0 modular knowledge, it would take weeks to actually reach the full patching potential of those modules (and even I'm learning new things to do with them). After that, you have a pretty good idea of what you want and can branch out into other directions of your choice (drums, polyphony, physical modeling, Eurorack ports, etc).

some big single user repositories bundled into manager as trusted (potentilly Autodafe, Hetrick)

I believe everything should go into one repository and listed as a flat list (sortable alphabetically, last updated, popularity***, etc), and every module that remotely works at all should be listed under it.

The package manager will be integrated with the users' accounts, so Rack only needs to ask "what plugins are available, and what are their versions". Then it compares the versions of the currently loaded plugins. Very simple.

there is an about page for every repo where user can see where to put bug reports, feature requests and donation to plugin/script author

That would just be plugin developers' github repos.

there is a possibility to revert back to previous version of plugin (like if newer version has bug)

Precisely. There is no version less-than comparison anywhere. Simply string comparison, so if MyPlugin v0.5.3 has a bug, the community repo will be "updated" to MyPlugin v0.5.2 and Rack will download that version again.

I donn know what will be with backward compatibility, seems youll need to mark installed plugins as grey if their version lower than current VCVRack version (so they willn`t be uninstalled i.e. awaiting update from author and unavailable)

Incompatible plugins can't loaded by Rack, so Rack will consider them as nonexistent. So when it compares existing plugins with the list, it will think it doesn't have them and download the new versions.

***I haven't thought about this, but sorting by popularity will be a great idea, and since it's integrated with VCV accounts, it's easy for it to know how many have added it to their account.

[jeremywen]

👍 on the popularity idea. I have always wanted to be popular. Maybe, just maybe this will be my chance. LOL 😄

[s-ol]

(Yes, not the favorite for maybe 0.2% of users, but they always have the option of manually downloading the plugins from the download URLs and placing them in the plugins/ directory.)

It's fine to have the local version of Rack use the web server for managing installs, but the manifest format should be versatile enough that if we create a vpm CLI tool, it will be able to understand that vpm install RJModules will use the "official" API/community repo and vpm install Miserlou/RJModules (or similar) will use the one from my personal repo. This will make dev collaboration much easier.

Maybe the main rack binary could provide simple flags/commands to synchronize an external manifest URL (fetch, download version, add to local database) and remove a package?

$ ./Rack --install https://example.com/Module.json
$ ./Rack --remove https://example.com/Module.json
$ ./Rack --remove ManufacturerSlug/ModuleName # or perhaps this?

those 'manual install' packages could be tracked similarily to the ones synchronized from the online user API and live in a 'Local/Other' category, so that changing your subscription status online will either move re-classify that plugin as 'Local' or uninstall it. Local plugins probably shouldn't autoupdate which has the benefit of potentially making development simpler.

Maybe local plugins like this wouldn't even need tracking in VCVRack and would just be discovered as the plugins found in the directory that are not tracked by the API.

[AndrewBelt]

Since it's 0.2% (arbitrary number), it's a very low priority.
But it seems possible at first glance. It would still use the api.vcvrack.com server to query packages.

Removing individual modules is impossible. The manager will operating only on Plugin Slugs (there is no such thing as a manufacturer slug, just a manufacturer name).
Something like

./Rack add MyPlugin
./Rack remove MyPlugin

[s-ol]

About the travis-ci things, I just successfully cross-compiled a plugin to windows on Arch Linux, you can see my changes and tiny log in #509. I think this should be reproduceable in travis, especially if the Makefile can have an extra target that doesn't build the core modules.

@AndrewBelt right, I lost my train of thought there a bit, Module.json should be a plugin by one manufacturer and removing single module doesn't make sense.

[MovGP0]

As a personal preference, I'd recommend to use something existing instead of reinventing the wheel.

I guess NuGet might be a good option, since it is well established on all plaforms (Win/Mac/Linux) and much more stable than NPM and most other package managers.

Not sure if you can use if for not-.NET workloads, but would recommend to port to .NET Standard/Native anyway, because it is independent of the execution platform and also independent of the programming language.

[AndrewBelt]

@MovGP0 As mentioned before, this isn't something you can just drop a premade solution. All the Rack Package Manager does is 1) queries the vcvrack.com API with the user's token, 2) checks the installed plugin and compares them with available plugins in the user's account, and 3) downloads, checks SHA256, and unzips them.

This is not a personal preference thing, it is a problem that has a clear solution, and it is not NuGet, npm, etc.