Discussion 2: Electric Boogaloo

[spessasus]

Hi,

I'm glad that SpessaSharp is finally going public! I'm considering mentioning this project in the upstream README, would that be okay?

Though I gave it a try and there are some things worth noting:

1. SpessaSynthDiff seems to be removed, and Rider attempts to load the project but fails: SpessaSharp/SpessaSynthDiff/SpessaSynthDiff.csproj' not found

2. SSTool uses the upstream picture instead of the SpessaSharp version:

https://github.com/Wh1teDuke/SpessaSharp/blob/master/SSTool/assets/com.spessasharp.ss.png

Is this intentional?

3. Mismatch:

SpessaSharp/SpessaSharp/Synthesizer/Engine/Parameters/GlobalSystemParameter.cs

Lines 375 to 379 in 117eda2

	/// <summary>
	/// If the synthesizer should prevent changing any parameters via NRPN.
	/// This includes the custom vibrato parameters.
	/// </summary>
	NprnParamLock,//bool

The mention about custom vibrato was removed in upstream.

4. Custom vibrato parameter shows for NrpnParamLock in SSTool:

SpessaSharp/SpessaSharp/Synthesizer/Engine/Parameters/GlobalSystemParameter.cs

Line 438 in 117eda2

"If the synthesizer should prevent applying the custom vibrato. This effect is modified using NRPN, so the recommended use case would be setting the custom vibrato then locking it to prevent changes by MIDI files.",

5. The rendered audio is quite different from upstream.

Archive.zip

• MIDI: https://github.com/AyHa1810/touhou-midi-collection/blob/main/6%20-%20Arranges/Bad%20Apple%20feat.%20nomico%20(s__msgs).mid
• SoundFont: https://musical-artifacts.com/artifacts/1176 NOTE: Version 2.5 used here.

The exact command used:

./ss c -m GENERAL_MIDI/midis/touhou/bad_apple_feat_nomico_s__msgs.mid --soundbank soundfonts/Live\ HQ\ Natural\ SoundFont\ GM.sf2 -o out.wav  

At the start you can hear that the bass sine waves are way quieter in sharp, and the "Slap" sample (key 28) is way louder.
And at 12 seconds, the reverse cymbal patch swipes from left to right in upstream, while in sharp it only plays on the right side. The swipe is done without any panning (just a stereo sample pair that has this effect, so this suggest that the left sample is not playing at all?

[Wh1teDuke]

I'm considering mentioning this project in the upstream README, would that be okay?

Sure, first let me fix this issue

At the start you can hear that the bass sine waves are way quieter in sharp, and the "Slap" sample (key 28) is way louder.
And at 12 seconds, the reverse cymbal patch swipes from left to right in upstream, while in sharp it only plays on the right side. The swipe is done without any panning (just a stereo sample pair that has this effect, so this suggest that the left sample is not playing at all?

[spessasus]

Should I open it as a bug report then?

[Wh1teDuke]

I've got my man working on it now.

[Wh1teDuke]

The issue was introduced during the api-changes saga, when I added a new cache system for BasicPreset.GetVoiceParameters method (commit 97c57a33a871c2f4a19381ce85616a0170cee3e9 Cache inception). Don't know yet what the exact problem is.

[Wh1teDuke]

I believe the bug is fixed now 13ca303

[spessasus]

I can confirm the fix! Was it hard to figure out?

BTW: I've reworked the tests for core, so now running make_tests.ts outputs all files into generated directory. Have you considered making an automated diff test that runs all of these files and compares the output with upstream?

[spessasus]

Also one more question:

Currently upstream has a main synthesizer (SynthesizerCore) and a public wrapper (SpessaSynthProcessor). I recently discovered that the @internal tag allows to hide public library functions from the consumers (you can see this pattern in MIDIChannel). And I've been wondering about doing the same to SynthesizerCore (deleting processor, renaming core to SpessaSynthProcessor and @internal-ing all methods not meant for end users.

What do you think? Did you copy the wrapper approach in Sharp or did it with internal?

[Wh1teDuke]

It's a sensible approach, but the wrapper has its own benefits. It's future-proof against refactorings, also better protection against unintended internal api leaks. In any case, it doesn't hurt either, I wouldn't bother removing it.

[spessasus]

Another issue:

I tried realtimeMidiSynth and after selecting the port it listened on with Falcosoft MIDI Player, it crashes:

dotnet run -c Release RealtimeMidiSynth.cs ~/htdocs/SpessaSynth/soundfonts/gm.dls
Initializing synthesizer ...
Listening on 'hw:0,3'
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.front
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.side
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.surround21
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.surround21
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.surround40
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.surround41
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.surround50
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.surround51
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.surround71
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.iec958
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.iec958
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.iec958
ALSA lib confmisc.c:1377:(snd_func_refer) [error.core] Unable to find definition 'cards.0.pcm.hdmi.0:CARD=0,AES0=4,AES1=130,AES2=0,AES3=2'
ALSA lib conf.c:5207:(_snd_config_evaluate) [error.core] function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5730:(snd_config_expand) [error.core] Evaluate error: No such file or directory
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM hdmi
ALSA lib confmisc.c:1377:(snd_func_refer) [error.core] Unable to find definition 'cards.0.pcm.hdmi.0:CARD=0,AES0=4,AES1=130,AES2=0,AES3=2'
ALSA lib conf.c:5207:(_snd_config_evaluate) [error.core] function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5730:(snd_config_expand) [error.core] Evaluate error: No such file or directory
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM hdmi
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM cards.pcm.phoneline
ALSA lib pcm_oss.c:404:(_snd_pcm_oss_open) [error.] Cannot open device /dev/dsp
ALSA lib pcm_oss.c:404:(_snd_pcm_oss_open) [error.] Cannot open device /dev/dsp
ALSA lib pcm_a52.c:1036:(_snd_pcm_a52_open) [error.] a52 is only for playback
ALSA lib conf.c:5697:(snd_config_expand) [error.core] Unknown parameters {AES0 0x6 AES1 0x82 AES2 0x0 AES3 0x2 CARD 0}
ALSA lib pcm.c:2722:(snd_pcm_open_noupdate) [error.pcm] Unknown PCM iec958:{AES0 0x6 AES1 0x82 AES2 0x0 AES3 0x2 CARD 0}
ALSA lib confmisc.c:160:(snd_config_get_card) [error.core] Invalid field card
ALSA lib pcm_usb_stream.c:481:(_snd_pcm_usb_stream_open) [error.] Invalid card 'card'
ALSA lib confmisc.c:160:(snd_config_get_card) [error.core] Invalid field card
ALSA lib pcm_usb_stream.c:481:(_snd_pcm_usb_stream_open) [error.] Invalid card 'card'
ALSA lib pcm_dmix.c:1000:(snd_pcm_dmix_open) [error.pcm] unable to open slave
SysEx               (0xF0 0x41 0x10)
Unhandled exception. System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at SpessaSharp.Synthesizer.Engine.Sysex.RolandGS.SystemExclusive(Synthesizer synth, ReadOnlySpan`1 syx, Int32 channelOffset) in /home/spessasus/rzeczy/c#/SpessaSharp/SpessaSharp/Synthesizer/Engine/Sysex/RolandGS.cs:line 27
   at SpessaSharp.Synthesizer.Engine.SystemExclusive.Execute(Synthesizer synth, ReadOnlySpan`1 syx, Int32 channelOffset) in /home/spessasus/rzeczy/c#/SpessaSharp/SpessaSharp/Synthesizer/Engine/SystemExclusive.cs:line 51
   at SpessaSharp.Synthesizer.Synthesizer.SystemExclusive(ReadOnlySpan`1 syx, Int32 channelOffset) in /home/spessasus/rzeczy/c#/SpessaSharp/SpessaSharp/Synthesizer/Synthesizer.cs:line 203
   at SpessaSharp.Synthesizer.Synthesizer.ProcessMessageInternal(ReadOnlySpan`1 message, Int32 channelOffset) in /home/spessasus/rzeczy/c#/SpessaSharp/SpessaSharp/Synthesizer/Synthesizer.cs:line 1017
   at SpessaSharp.Synthesizer.Synthesizer.ProcessMessage(ReadOnlySpan`1 message, Int32 channelOffset) in /home/spessasus/rzeczy/c#/SpessaSharp/SpessaSharp/Synthesizer/Synthesizer.cs:line 520
   at SpessaSharp.Synthesizer.SpessaSharpProcessor.ProcessMessage(ReadOnlySpan`1 message, Nullable`1 channelOffset) in /home/spessasus/rzeczy/c#/SpessaSharp/SpessaSharp/Synthesizer/SpessaSharpProcessor.cs:line 170
   at Program.<>c__DisplayClass0_0.<<Main>$>b__0(MidiMessage m) in /home/spessasus/rzeczy/c#/SpessaSharp/Examples/RealtimeMidiSynth.cs:line 46
   at OwnAudio.Midi.IO.Platform.LinuxMidiInputPort.ReadLoop()
   at System.Threading.Thread.StartCallback()

EDIT: After further testing, it seems that realtime synth trims the system exclusive to only 3 bytes. And also System Exclusive doesn't seem to handle malformed sysExes well.

[Wh1teDuke]

I need a midi with sysex events to test this, any suggestions? Also, isn't Falcosoft a windows application? are you using wine?

[spessasus]

For example the bad apple MIDI uses system exclusives. And in the same repo, if you go to shooter games > embodiment of scarlet devil, pick any MIDI that has (ZUN) in its name, they have A LOT of system exclusives.

Also, isn't Falcosoft a windows application? are you using wine?

Yes, there aren't really any good GM applications for Linux. And FSMP allows me to test with VST versions of GM synths like syxg50, which only have a windows version.

But you don't need to download it, using spessasynth web app and selecting your MIDI output port in the settings also makes it function like an external MIDI player that can drive your realtime synth.

[Wh1teDuke]

Once this gets merged ModernMube/OwnAudioSharp#27 rotten apple should work.

[Wh1teDuke]

@spessasus On your locked params branch, you added the function/array to SynthesizerCore, but you are not reading from it. Is this intentional?

[spessasus]

Which one?

[Wh1teDuke]

https://github.com/spessasus/spessasynth_core/blob/381333e12e4fb45bc31ab51cc36dfabb55dd3bcb/src/synthesizer/audio_engine/synthesizer_core.ts#L129

[spessasus]

It is used:

https://github.com/spessasus/spessasynth_core/blob/lock-midi-parameters/src%2Fsynthesizer%2Faudio_engine%2Fparameters%2Fmidi.ts#L77

And with setting MIDI parameter, we check it:

https://github.com/spessasus/spessasynth_core/blob/lock-midi-parameters/src%2Fsynthesizer%2Faudio_engine%2Fparameters%2Fmidi.ts#L55

[Wh1teDuke]

I'm working on a soundfont focused on effect sounds. I want to make it as small as possible while retaining some quality. If you are interested I can let you in my secret club. Will be licensed under CC0 1.0.

[Wh1teDuke]

@spessasus I'd like to be able to alter some of the instrument's generators/modulators on demand. For the effects for example, sometimes I want by default to set the loop mode to "UntilRelease", but there may be times when disabling or setting the loop to 1 would be preferable. Duplicating the instrument just to change one generator for a noteon would be like cracking nuts with a sledgehammer.

I think this is illegal, so this could be implemented through some of the undefined cc's. This would be a good time to implement a custom cc system. For example, CC 89 would mean "SpessaSynth Controller", and the first data value would indicate the actual type of the controller (like SamplesMode or ReleaseVolEnv).

[spessasus]

The custom CC system you are describing is called SF2 modulators.

In theory I can allow modulating loop type, and you could set your source to switch and amount to -3, so the final result is 0 ( no loop). Or a different CC with -2 so it results in regular loop. But I suggest that you open a discussion in fluid synth and ask if loop mode can be modulated. If you give a good use case for it, I think that they'll agree as it wouldn't hurt. And then I can confidently implement that into spessasynth, and other sf2synths will allow modulating sample loop as well.

[Wh1teDuke]

So what about a dedicated api? Using channel midi params perhaps. In fact, it is its own category (voice params). Don't actually need to be a midi message or cc, just some way to directly manipulate it.

[spessasus]

SF2 has a dedicated NRPN system that allows you to alter generators in real-time (section 9.6 of sfspec24). It is supported by spessasynth. Though I still think that what you described is a perfect fit for SF2 modulators. I can make looping mode "modulable", so it's not cached and instead evaluated on note-on, much like envelope times or sample offset. would you like that?

[Wh1teDuke]

I'm looking now at handleAWE32NRPN, I think that would do the trick.

[spessasus]

I've been thinking about replacing the "magic numbers" in the sysEx detection (in both synth and MIDI utils) with named constants. What are your thoughts on this? Would that be a good idea for performance and readability?

[spessasus]

Maybe you could add some api friendly class to make parameter tweaking simpler.

Isn't this exactly that? I made this, and channel version abstractions for adding the magic MIDI messages for every parameter. And analysis!

[Wh1teDuke]

Not quite, consider this https://github.com/spessasus/spessasynth_core/blob/3754074a835ae8f910098ac8b04c9b11f8f22493/src/synthesizer/audio_engine/system_exclusive/universal.ts#L135 I don't think there is a single way to setup these without either accessing the internal effect buffer directly or hand crafting the sysex message.

[spessasus]

You're right, I missed this in the sysEx analysis, I'll fix that.

Though the effects aren't a part of the MIDI Parameter System.
I consider the MIDI Parameter System complete (there will be new MIDI Parameters, but the system is done).

But as you can see from the simple "Reverb Param" and similar, the effect system is not done. It'll have its own setEffectParam and proper analysis once XG effects are added.
I now get how insertion works (because I asked a person knowledgeable in that topic in the related issue), but I'm still not sure how the reverb and chorus could be implemented. Do they need new interfaces or can existing GS interfaces be adapted? What do you think? (the XG spec is here)

And btw, what do you think about the MIDI parameter system, with the global, per-channel and the analyze/set abstractions? I'm quite proud of it :-]

[Wh1teDuke]

but I'm still not sure how the reverb and chorus could be implemented.

Have you read this thread? https://www.vogons.org/viewtopic.php?p=1426766#p1426766

I'm quite proud of it :-]

It's the cherry on the cake for sure.

[spessasus]

I checked back and GM reverb parameters are being detected properly in analyzeGM.

[spessasus]

Have you considered adding an automated diff test with upstream?
What I mean by that, is making a workflow that does the following for every push:

• checkout upstream
• generate test:midi files
• make upstream render them (I think gm.dls is the best sound bank suited for the tests)
• make spessasharp render them
• compare the rendered outputs
• If the difference is too large, the workflow fails and tells you about mismatched audio

What do you think?

[spessasus]

Speaking of which, are you implementing new effects? Looks like you are going to support yamaha insertions.

Yes. I read the vogons thread you linked but it doesn't answer my question. From looking at the spec, I think I can adjust the effects a bit, but I want to do this properly so XG is getting its own effects. The algorithms are probably gonna be the same but the interfaces will be a bit different.

Though first i'd like to sort out:

• XG PART MODE support properly (like drumMap for GS)
• Enhanced BLOCK B and others support in the MIDI detector.
• Keyboard range support
• dynamic modulator system for XG and GM2.

It'll have to wait though, as I have 4 exams this week.

[Wh1teDuke]

Can't you contact that user or ask on the thread? There were several exchanges on the subject recently.

It'll have to wait though, as I have 4 exams this week.

Good luck with that, though I don't think you need it. You are the smartest spessasus in spessaland.

[spessasus]

I found this page, which seems to be describing exactly what I'm looking for:
https://battleofthebits.com/lyceum/View/xg%20(format)

[Wh1teDuke]

Looks comprehensive to me. I wonder how many great resources are kept hostage behind the japanese language barrier.

[Wh1teDuke]

@spessasus anything of value here https://drive.google.com/drive/folders/18gRxGCkGYzmqh4HeW5-IakmuUyHCjwGN ? Got this from vogons.

[Wh1teDuke]

@spessasus I don't you think you included in your soundfont proposal including both msb/lsb preset data right?

https://github.com/spessasus/spessasynth_core/blob/6c7d386c15b84af6ac52af156322bbba5c6908f4/src/soundbank/basic_soundbank/basic_preset.ts#L530-L540

[spessasus]

The idea of full lsb and msb is implemented in read mode, but not in sf2 write mode. Writing both in sf2 will result in most players breaking (including fluidsynth and polyphone), making it unacceptable. Once writeSFe is added, it'll write full bank LSB and MSB. It'll also support RIFF64.

[spessasus]

About writing sound banks...

https://github.com/spessasus/spessasynth_core/tree/rf64-support

[Wh1teDuke]

@spessasus any future plans on lifting the limitation of only one insertion effect active?

[spessasus]

GS insertion only supports one at a time so the answer is no.

[spessasus]

Although now that I'm looking at the manual, it seems that in sc-88pro block B has its own EFX data set in double module mode. I supposed that could mean 2 insertion effects? Though I can't try it with SCVA as it only has 1 block. I'd need a real sound canvas for this.