Added FST Bank and Preset handling.

[ZonderP]

The 'Synth' class got one additional 'pseudo'-parameter called 'name'.
It defaults to 'Init' and gets serialized/imported alongside the regular synth parameters in the 'serialize' and 'import' functions of the 'Serializer' class.
The FST plugin code has most of the relevant changes:
'set_chunk' and 'get_chunk' got an additional parameter indicating if only a preset's or the whole bank's state should get restored/stored.
There also functions for handling opcodes 'effSetProgram', 'effGetProgram', 'effSetProgramName', 'effGetProgramName' and 'effGetProgramNameIndexed' got introduced.
I made js80p support banks of 128 presets - as this is what most FST 2 plugins have.
This means that the FST plugin code needs to keep an array of 128 serialized strings representing those 128 presets.
It also keeps an array of 128 preset names (each an empty string initially, but if accessed and still empty then get parsed from the respective preset's serialized string).
The code shouldn't affect the VST3 version at all (except that also the preset name 'Init' would be serialized) and also all other code not related to bank and preset handling is untouched.
I guess code in 'serialize' and 'import' of the 'Serialize' class could be adjusted to make it work more like the other code there, but because I couldn't really debug, I didn't do that.
I tested 'only' the 64 bit Windows version with Herman Seib's 'savihost_64.exe' and with 'real' DAW 'EXT64 beta 1' (https://www.xtsware.com/?page_id=297).

I didn't even try to give the synth the ability to name a preset from its GUI, as this is not a requirement for the functionality of the FST bank and preset handling - but this is something which would be possible now.

[attilammagyar]

Thank you, this is awesome!

I will take a deeper look at it once I'm done with the features that I'm currently working on, and I'll take a look at how this feature could be added to the VST 3 plugin as well, and then I'm going to merge it. As I said, it might take some time, but I'm on it. Until then, you don't need to bother with keeping the branch in sync with main, I'll resolve all conflicts when I get to merge your changes.

One small question though: do you mind, if I change the copyright header in the affected files like the following?

/*
 * This file is part of JS80P, a synthesizer plugin.
 * Copyright (C) 2023  Attila M. Magyar
 * Copyright (C) 2023  ZonderP
 *

[ZonderP]

Thank you - for this wonderful synth!

Take your time, no hurry at all - I've already got a version with this feature in ;-)

Regarding VST3:
I'm quite uneducated about this format, but from my observations and readings, I've the feeling that something like host supported bank and preset handling isn't there - or, if so, then in a completely different way which I didn't yet figure out.
But well, I think there is the vstpreset (format?). At least some VST3 plugins make use of it.
I guess the new preset name in the synth might be of help for storing vstpreset files - where I guess you would only need to store the serialized preset string as chunk - probably quite similar to how it is now done for storing it in the FST fxp file.

Regarding the copyright header:
First I thought that this would be too much of a honor, since all the cool DSP code comes from you.
But then I saw that you mentioned 'affected files'.
Yes, this is okay for me, but I would prefer to have my real name in it.
So 'Patrik Ehringer' instead of 'ZonderP'.

Looking forward to the new features you're working on!

And in the meantime I'm curious about how parameter automation in the FST could be implemented/work ;-)

[attilammagyar]

And in the meantime I'm curious about how parameter automation in the FST could be implemented/work

I see you have started experimenting with that already. :-) I think that directly exporting the parameters might raise a lot of problems, e.g. what should happen if the DAW is automating a parameter which also has a controller assigned within the synth? What if the in-synth one is an envelope (which is polyphonic)?

One way to avoid this problem would be to define a couple of host-automation controllers explicitly, let's say 20, and export 20 dummy automatable parameters. Then the synth would use those dummy parameters for driving those internal parameters which have these special controllers assigned.

But wait - this is almost the same thing as what VST 3 does. VST 3 plugins receive all MIDI CC messages as parameter automation rather than as actual MIDI messages. In fact, the FST could implement automation with a similar trick: export one parameter for each supported MIDI CC, and let the DAW automate those.

Though some ambiguity would still remain with this approach (e.g. if we get an automated and a MIDI value for a CC at the same time - in VST 3, the winner is the one which came later), at least there would be no conflicts with poliphony, and the two plugin types would work the same way.

At least, this is how I plan to implement it after I'm done with tempo sync and log scale filter frequencies (almost there), and with merging your PR.

[ZonderP]

Hmmm, I currently do not really understand, how those MIDI CC controllers are meant to work....
And the more I toy around with js80p, I wonder why they, resp. their connection to parameters, are even stored in the patches/presets.
I'm coming more or less only from the VST2 side and there - as I see it - MIDI CC is used as a convenient way for 'live' manipulation of a patch/preset during playing/recording.
And quite some VST2's allow dynamic assignment of controllers to parameters (e.g. free Full Bucket instruments) via something which is mostly called '(MIDI) LEARN'.
I have the feeling that this is of relevance, since people use different MIDI controllers. Real MIDI controllers - again as I understand it - allow one to freely assign MIDI CC to their knobs and sliders. But others - as me - just use their hardware keyboard controls as MIDI CC sources. On my keyboard the knobs and sliders are not freely assignable to MIDI CC but instead are fixed.
So - let's say, I load a preset for js80p where e.g. Modulator Filter 1 Frequency is assigned to MIDI CC 102 and my keyboard either even does support this MIDI CC or it was fixed to a slider in organ bars section which I wouldn't want to use, I would need to reassign it to another MIDI CC which fits more to the knobs on my keyboard, before being really of use to me....

And then, when it comes to recording of automation:
For VST2's which support MIDI LEARN my way of dealing with it that I find out the parameters I want to tweak 'live' during playback/recording and assign a MIDI CC to them to my liking (what controls on my keyboard fits best).
Then - when I record - I play the notes I want and turn the knobs which send the according MIDI CC to tweak the parameters I want to tweak 'live'. At least in energyXT this automation then also gets recorded automatically.

E.g. when I use ''blooo' VSTi from 'Full Bucket Music' and I have a nice sound where I want to live tweak the cutoff of Filter 1 with a knob on my keyboard which sends on MIDI CC 86, then I first 'LEARN' blooo to used MIDI CC 86 for Filter 1 cutoff.
Then I record some notes in energyXT 3 and whilst doing so, I turn that special knob on my keyboard which sends MIDI CC 86 messages. At least in energyXT the result is, that there will also automatically be a lane for blooo's automatable parameter 'CutFrq 1' with the parameter's value adjustments as I've done them live whilst recording the notes.

This is how I somewhat learned how live tweaking/automation of parameters is supposed to work (with VST2 at least).

I hope that what I wrote just above is somewhat understandable.
Might well be, that I completely misunderstand the concepts behind the whole stuff, but this is how I use to work with it (if I ever would 'seriously' make music...).

For js80p the current state is, that as soon as a parameter has a MIDI CC controller assigned, that parameter cannot be automated in any way - from what I have experienced with my draft...

I will create a PR now for you with for this draft - maybe it can be of help for you to try out / play around. This draft is not at all meant to be really pulled into your main repo.
The draft supports automation of all 'regular' parameters.

I will now be away from my computer for some days.

Cheers,
Patrik

[attilammagyar]

Hmmm, I currently do not really understand, how those MIDI CC controllers are meant to work....
And the more I toy around with js80p, I wonder why they, resp. their connection to parameters, are even stored in the patches/presets.

They are stored so that it is possible to create presets that use the well-defined CC controllers more or less the way one would expect them to work (e.g. modwheel CC1, volume CC7, etc.). Also, I wanted to keep things simple, and use the same serailization for patch export/import and effSetChunk/effGetChunk.

I'm coming more or less only from the VST2 side and there - as I see it - MIDI CC is used as a convenient way for 'live' manipulation of a patch/preset during playing/recording.
And quite some VST2's allow dynamic assignment of controllers to parameters (e.g. free Full Bucket instruments) via something which is mostly called '(MIDI) LEARN'.

You are right, this feature was missing from JS80P. I should have thought about users who have many reassignable knobs and faders on their keyboard.

I have the feeling that this is of relevance, since people use different MIDI controllers. Real MIDI controllers - again as I understand it - allow one to freely assign MIDI CC to their knobs and sliders. But others - as me - just use their hardware keyboard controls as MIDI CC sources. On my keyboard the knobs and sliders are not freely assignable to MIDI CC but instead are fixed.

I think there should be some limitations on what CC can be assigned freely by the user to knobs and faders, because there are CC numbers which have special meanings. E.g. assigning CC 33 to a user-assignable fader would interfere with 14 bit CC messages of the modwheel. (This is why JS80P omits CC 32-63 and many others: they have special meanings that I might want to implement some day.)

So - let's say, I load a preset for js80p where e.g. Modulator Filter 1 Frequency is assigned to MIDI CC 102 and my keyboard either even does support this MIDI CC or it was fixed to a slider in organ bars section which I wouldn't want to use, I would need to reassign it to another MIDI CC which fits more to the knobs on my keyboard, before being really of use to me....

I agree, that's why I try to restrict the presets that I include in JS80P releases to only use the modwheel from the CC numbers, because that's what's usually available on all keyboards with a standard CC number. (The only exception is demo-2.js80p which also uses CC 7 besides CC 1.)

And then, when it comes to recording of automation:
For VST2's which support MIDI LEARN my way of dealing with it that I find out the parameters I want to tweak 'live' during playback/recording and assign a MIDI CC to them to my liking (what controls on my keyboard fits best).
Then - when I record - I play the notes I want and turn the knobs which send the according MIDI CC to tweak the parameters I want to tweak 'live'. At least in energyXT this automation then also gets recorded automatically.

This is the workflow that I had in mind originally, and why I didn't export the parameters in the first few releases.

For js80p the current state is, that as soon as a parameter has a MIDI CC controller assigned, that parameter cannot be automated in any way - from what I have experienced with my draft...

I haven't checked out your draft yet, but I'm not sure I understand this part. The workflow that you described above should work in JS80P even without exporting any parameters to the DAW. If I understand correctly, the problem is that you would need trial and error to figure out which knobs on your keyboard correspond to which CC in the plugin, and that's what MIDI Learn could do instantly. Do I understand it correctly?

I will now be away from my computer for some days.

I hope I can get to show some progress with your PR by the time you get back.

[ZonderP]

I haven't checked out your draft yet, but I'm not sure I understand this part. The workflow that you described above should work in JS80P even without exporting any parameters to the DAW. If I understand correctly, the problem is that you would need trial and error to figure out which knobs on your keyboard correspond to which CC in the plugin, and that's what MIDI Learn could do instantly. Do I understand it correctly?

Yep, this works - automation gets recorded - in XT64 as 'Control Change' for the assigned MIDI CC.
What I meant is, that - with parameters exposed as in my draft - then I can - at least in XT64, select such a (named) parameter and then - manually - draw automation. But that automation then doesn't have any effect, if that parameter has a MIDI CC assigned in js80p.

I see, you already implemented MIDI learn! Great!
Will try out v1.4 as soon as I find some time - probably not the next few days though.

[attilammagyar]

But that automation then doesn't have any effect, if that parameter has a MIDI CC assigned in js80p.

This is the conflict which I plan to avoid by exporting the supported MIDI CCs as parameters instead of exporting the synth's parameters directly. Then you can automate those, and FstPlugin will handle the automation as if ordinary MIDI CC events were received.

It will work the same way as in VST3, where the plugin has no choice but to define these virtual parameters if it wants to handle CC events. (The only difference will be that in VST3, these exposed parameters are automatically assigned to the respective MIDI CC by the host, which cannot be done in FST - but fortunately it's not needed, since unlike VST3, the FST plugin receives the raw MIDI with CC and everything.)

So for example, on the GUI, you would assign the Modwheel to the wavefolder, and then it's up to you if you want to physically turn the wheel while recording, or if you want to draw automation for the exported parameter named "Modwheel".

<rant> By the way, in the last couple of weeks, I've been fighting with VST3 to achieve a similarly working program management to what you implemented, and I seem to be on the losing side of the battle. I guess they mean it when they say:

If the plug-in defines a program list to be used as pool of factory presets, it must not allow the user to change these presets by the means of parameter editing. Instead, it should load the corresponding data into a kind of working memory and store possible modifications as component state.

Though they also say that you can let the users overwrite the built-in presets (similarly to the isPreset = false case in FST) if you really-really want to, by implementing IProgramListData, but that doesn't seem to be widely supported. So I decided to keep the FST plugin working the same way as you implemented, with 128 editable, persistable programs (except that the first few will be initialized with the built-in presets), but the VST3 version will do it the VST3 way, by only persisting the current program's state, and letting the host do the program and bank management.</rant>

As for the program names: those won't be editable by the user in the FST version on the GUI, since user supplied texts can bring a lot of complications that I'd rather avoid. (Where to put it on the GUI, encodings, right-to-left writing systems, IMEI, etc. Also, since I don't use any high-level GUI libraries in the Linux version in order to have maximum portability, it would need some extra e as well. Of course, I could rely on zenity and kidalog for this as well, just like for the file open/save dialogs, but I'd rather decrease the plugins reliance on these, not increase it.)

[ZonderP]

So I decided to keep the FST plugin working the same way as you implemented, with 128 editable, persistable programs (except that the first few will be initialized with the built-in presets), but the VST3 version will do it the VST3 way, by only persisting the current program's state, and letting the host do the program and bank management.

Yes, this is nice that you'll use the first bank slots and initialize them with the built-in presets.
This is how it is usually done and I'm glad to hear that you plan to do so too.
And when it comes to VST3 then I think it looks like my feeling was right, that that format doesn't anymore support bank handling in some similar way to FST - I would say they completely dropped bank handling.

As for the program names: those won't be editable by the user in the FST version on the GUI

Absolutely fine - as this is something which is now possible from within a host (if the host supports it - and I think most hosts do so nowadays).
Only for VST3 I'm not sure. Does it mean that VST3 presets are always somewhat name-less and the only indication of a preset name is the file name of the vstpreset file? Or would there now also be the possibility to name a preset in VST3 and store it in the name parameter, or would vstpresets files always just have 'Init' in the name parameter? Not a big deal I guess, if it is so, I'm just curious here, because I do not have any experience with VST3.

This is the conflict which I plan to avoid by exporting the supported MIDI CCs as parameters instead of exporting the synth's parameters directly. Then you can automate those, and FstPlugin will handle the automation as if ordinary MIDI CC events were received.

Yes, I guess this might be somewhat conflicting - but it seems other FSTs I know either are able to deal with this kind of conflict or it is somehow possible to just ignore the conflicting situation and let the 'stronger' or first or whatever win and think:
If the user really wants to 'override' recorded automation, let him do so - he might like the result or not...

MIDI Learn works great!

[attilammagyar]

Finally, banks and presets are on main! Thank you for your work!

I made a few changes here and there, and though in theory, it should be compatible with your format, make sure to have a backup of your existing presets and banks before you try it.

Only for VST3 I'm not sure. Does it mean that VST3 presets are always somewhat name-less and the only indication of a preset name is the file name of the vstpreset file? Or would there now also be the possibility to name a preset in VST3 and store it in the name parameter, or would vstpresets files always just have 'Init' in the name parameter?

It's up to the host to manage the names of the VST3 presets, just like for VST2/FST. The only difference is that you can't overwrite the presets that come with the plugin, and you can't rename them either. But with the presets that you create using the host's preset management, you can of course do whatever you want (and whatever the host allows doing).

Now I'll start working on #7.

[ZonderP]

Wow, thanks a lot for your work!

I see that this got somewhat more complicated then what I've implemented, especially when it comes to clearing events and whatnot and ensuring that parameter values are really the ones that get loaded for a program/preset!

Looking forward to your implementation of parameter automation.
(Although I wonder, if I will understand it, because for me, it is easier to understand if I have e.g. a parameter named 'Osc1FltRes' that I can edit the automation for than a parameter named 'CC23' or 'Snd 8'...)

[attilammagyar]

And it got even more complicated: I forgot to restart the LFOs after resetting the synth's state for the patch import. 😅

[ZonderP]

Hi Attila!
Great progress with the synth - a new version every day the last days!
It sounds amazing!

But - when it comes to preset handling, something is not quite right.
It doesn't behave anymore as it did with my initial changes to support bank and preset handling.
Probably related to the changes you did, where - if I understand the code correctly - related 'work' somehow gets deferred to when audio processing takes place. Since I still can't debug, I'm not sure what's going on.
But with my changes js80p behaved well - the same way as does the majority of FST plugins which support banks and thus more than just one program.
I'm not sure if you know SaviHost? It's Windows only, but it is used by a lot of FST developers to easily check, if their bank and program handling works as it should. Basically it is not a DAW, but 'just' a host by which you can make any FST a standalone.
And it supports the standard FST stuff when it comes to loading and saving of banks and programs and stepping thru the programs either backwards or forwards or just jumping to (loading a program at) a specific position. Also Savihost remembers the last program index at exit (probably somewhere in the Windows registry) and on restarting with the same FST, it immediately tells the FST to load the program at that position.
Here some of my observations with Savihost together with js80p:
1.) If - whilst js80p is hosted by Savihost - I e.g. load the program at index 40 and then exit Savihost and then open js80p again with Savihost, then it always starts at index 0 (wrong!). But when I then click the 'next program arrow', it jumps to the index used at last exit (40 in this example) + 1. Very strange and unfortunately quite unexpected and wrong.
2.) Again with js80p loaded in Savihost. When I go to an empty slot - eg. at index 31 this time and then load a previously saved preset file (fxp) - it is just plain wrong - the GUI doesn't get updated and the sound is still the 'blank' sound there.
And then pressing 'Next program' makes Savihost somehow look like it was faulty, because then js80p goes to the previous index!
It's really a bit hard to describe - I just found out that if you use Savihost's PlugIn menu 'Programs n - m' entries then js80p just behaves super weired. It somehow always jumps to the position which was previously chosen.
Makes working with it impossible.
As said - I (re)tried with my old version - and everything works as it should. The same as e.g 'blooo' by 'fullbucket music' and most of all FSTs I know.
I couldn't really figure out what would be the reason for this very strange behavior.

I hope you can find some time to investigate.

Edit:
Here the link to Savihost in case you want to try out: https://www.hermannseib.com/english/savihost.htm
And here the link where you could DL blooo FST to see how a well-behaving FST works with it: https://www.fullbucket.de/music/blooo.html

Also: I for sure also tried in energyXT, where there are the same resp. similar issues.

Edit 2:
And here my original version:
https://github.com/ZonderP/js80p-zp/commits/MyFSTBankAndPresetHandling

[attilammagyar]

Thanks, I'll take a look at it, it looks like it's not enough to test in just Reaper and FL Studio. (I haven't seen any strange behavior in these two with regards to preset handling.)

Next week is going to be a bit more busy for me than usual, so it might take some time.

[attilammagyar]

FYI: 1.7.6 is out with the fix for this.

[ZonderP]

Wow, thanks a lot!
Will try out in the evening.

[ZonderP]

Hi Attila!

Me again...

I was busy with releasing my VeeSTeeEx VST2 wrapper plugin (which I was writing for years and tried to test with nearly every interesting VST2 plugin that was released before and during development - that's how I found your jsp80 and that it initially didn't support bank and preset handling which is a precondition to be wrapped by VeeSTeeEx in the way it is supposed to work - see here: https://www.kvraudio.com/product/veesteeex-by-zonderp-productions)

Now that I finally have some time, I want to report, that unfortunately js80p still doesn't work quite right when it comes to loading of preset (.fxp) files. Sometimes the loaded preset just doesn't sound as it should, but instead just more like the 'init' or default sound. Then clicking 'next' and 'prev' again (e.g. in SAVIHost or via my VeeSTeeEx wrapper) to be back at the index where to the preset was loaded into, the preset sounds as it should.
One has to try some times, but for me at some point it happens. Might be the next loaded preset sounds okay again - or not. Couldn't detect a pattern. But I'm quite sure, that it is also related to this deferring mechanism.

In my VeeSTeeEx I save more or less the same data as gets stored in a fxp files (just with a little bit smaller header) into a database, and it is easy to just select such a preset and then also the same procedure takes place as if a preset would get loaded from a preset file. Here the same applies. Sometimes the selected preset sounds right and sometimes not.

If you want to see js80p wrapped via 'js80pEx.ddl' (my VeeSTeeEx wrapper renamed so that it loads js80p.dll) here is a link to a 7z files which contains everything needed in a folder: https://drive.google.com/file/d/17jsisPe_CnazgQHZsnI-Qio_FvoKIeaw/view?usp=sharing

Essentially it contains SAVIHost.exe renamed to 'js80pEx.exe', then 'js80pEx.dll' and 'js80p.dll' (version 2.0.0) as well as a presets folder with all 2.0.0 presets as fxp files and VeeSTeeEx's database, local settings and GUI file.

But no matter if it just doesn't work good with my wrapper - loading of presets also doesn't (always) work with SAVIHost or any other host which offers the functionality of loading a fxp preset file into a VST2 plugin.

I hope you have an idea of what could be the reason for it and how it could be fixed.

Regards,
Patrik

[attilammagyar]

Hi! Thanks, I will take a look at it some time next week.