Implement PCM voices support through YM2612's DAC

[Wohlstand]

As YM2612 supports DAC channel that can be used to output PCM, it was widely used for realistic percussion, SFX, and other things. DAC also can be used. No need to dedicate multiple DAC channels on multiple chip emulators, needed only one on any chip to don't steal so expensive FM channels from every simulated chip.

[freq-mod]

I have been thinking about it, just one question - do you plan staying close to limitations of Mega Drive hardware (Z80 CPU speed, RAM size, DMA, etc.) ?
If not, then, well... YM2612's DAC enable feature is entirely software controlled, so via software mixing you may have even like 8-bit 64 PCM channels @44.1KhZ with advanced volume and pitch control (all of this just from one hardware DAC channel) as long as you have enough CPU power.

[Wohlstand]

For the DAC is planned to provide some sort of simple WaveTable synth where it's output will be played through DAC. I have plan to keep 8-bit PCM output, but the WT-Synth backend possibly will don't rely on Z80, but will give only some basic percussion support with minimalistic functionality. I have to give it two modes:

• FAKE-DAC, which means the output of WT-Synth will NOT be sent into emulator's DAC interface, but will be directly mixed with output of emulated chips. This will save lot of CPU power. This way is only possible to use also with OPL3 chip which doesn't have any DAC. So, with this will be possible to simulate Junglevision's Sound Drive demo where OPL3 FM is mixed with PCM drums played through audio output interface of the sound card.
• Real-DAC, which means the output of WT-Synth will be directly sent to emulator's DAC interface. As this going to be sample-by-sample, then it's potential increasing of CPU usage due increasing of the data transfer complexity. This way would allow to even use some sort of Arduino rig to output the generated stuff into a real chip (or multi-chip in dependence from implementation you will construct).

[freq-mod]

I see, but

will give only some basic percussion support with minimalistic functionality.

Will the be possibility of handling SFX instruments, orchestra hits or timpani in melodic banks? I know OPN2 DAC requires a lot of software mixing in order to be used in a musical way as pitch control isn't natively supported, but then again...

[Wohlstand]

Some sort of melodics are will be too, but at first there are will have no envelope support. Everything will be done software-side. The DAC itself is a silly receiver of PCM samples sequence. To play the sound through DAC with specific sample rate, you will must to keep same delay between of each sample until write it into the DAC port.

[freq-mod]

Envelope support doesn't seem that important, I think. Drum-like sounds like orchestra impact hits, melodic toms, laughter SFX will do without it, for Rain, Wind, or Horse Gallop SFXs and others I guess there have to be some kind of a simple loop support.

technically there can be also mode for Ricoh RF5C164 support (8ch sega Mega-CD PCM sound chip that's something more than sample data receiver), but I don't see a reason for it 😉

[Wohlstand]

Just for a simple test, I have made the super-simple raw PCM player through Nuked OPN2's DAC: 77d0d50, and it works! 😄

P.S. The demo song to feed into my demo tool: YM2612_DAC_demo_stream.zip

Anyway, because of the reason I doing the sequence "write sample, read output sample", therefore I playing the song 1:1 and the output must be OPN2 native, but I playing it as 44100 🤣

[freq-mod]

sigh yet another problem with building

[Wohlstand]

Damn, I forgot to declare "SDL_MAIN_HANDLED", it's SDL's header has declared it's SDL_main...

[Wohlstand]

but... Yeah! it's really need to use it...

[Wohlstand]

@Papiezak , Just now I have sent a fix!

[freq-mod]

Yeah, it compiles, but it crashes the instant I load your song

[Wohlstand]

Just now I have sent another fix, please retry...

[freq-mod]

Sorry, still crashing

[Wohlstand]

Try to compile it in debug build (-DCMAKE_BUILD_TYPE=Debug), and run it through GDB:

gdb --args dac_test "Path\To\RAW\file"

then:

(gdb) run

and when crash will happen, type:

(gdb) bt

and give me the screenshot

[freq-mod]

i should have known it, segmentation fault

[Wohlstand]

I think I know why this happens... in the code I forgot to initialize Nuked's instance after allocation 🤦‍♂️

[Wohlstand]

@Papiezak , just now I have pushed my fix, try it out please 😉

[freq-mod]

I will try tomorrow

[Wohlstand]

Okay, anyway, I hope, it's now fixed, or I'll try that on Windows by myself now...

EDIT: Myself I have tested that out and my fix is working! However, I have found another issue caused by crap of Windows's console that prints anything very slow and clunky, and I have fixed that by moving printf() away from too granulated loop. For now it must play the stuff fine! 😉

[freq-mod]

yeah it's alright! playback doesn't jitter, great work 🥇

don't worry about 44.1 khz, real ym2612 can do it as well, you just need either Fujitsu FM Towns (16 MHz 386 of most basic models should do) or overclocked mega drive: http://www.sega-16.com/forum/showthread.php?20795-YM2612-playing-8-Bit-PCM-at-44-1-khz!&s=6159e5ef7714d5d1889a5b44e6c5e333

[Wohlstand]

I think, the sample rate of played stream must be chosen optimal, especially if 44100 is looks hacky for the real chip. And I'll need to make this test also play something from FM too. As in this test I have played PCM only and all other FM channels are muted.

[Wohlstand]

@jpcima , you can join the discussion here. Anyway, the first experiment with WT-Synth I want to make it on OPN2 Bank Editor side to provide the ability to upload WAV files, convert them into U8 and necessary sample rate, and then give the UI to configure the sample and test it's playing. I'll try to create the super-simple WT-Synth which we can extend with resampler (to scale the pitch), and you can try out your existing things you have suggested me when we talked via XMPP.

[freq-mod]

I think, the sample rate of played stream must be chosen optimal, especially if 44100 is looks hacky for the real chip.

26 KHz is optimal for real hardware, Deflemask and VGM music maker has 32 Khz option though. Most MD games used from 8 to circa 19 Khz

[jpcima]

What makes the Fake-DAC fundamentally different from playing OPNMIDI with a Fluidsynth instance layered on top?

[Wohlstand]

The difference is that will be used same simplified WT-Synth and the simulation of DAC sound processing (the u8 -> s16 conversion formula can be directly stolen from Nuked OPN2 emulator's code to prepare the buffer to mix with the generated FM outputs). With the same success result we are able to stream FluidSynth into DAC directly with requested sample rate and u8 mono format.

[jpcima]

What is the added value of this? Under real-dac, I understood a signal can be mixed as carrier into the FM algorithm arrangement.

But fake-dac? It's nothing special to add two ouputs of synths together. Beside, why is the insistence to tie it to the wavetable concept? It could imagine it to be any kind of generator, including from audio port input.

I use as example an analog Moog phatty synth, where there is a kinda similar concept: an input jack port is for an externally generated oscillator 3, which goes to be processed in the mixing and filter stages. I can plug a this jack to any synth of my liking.
(plugging the Moog synth into YM2612 would be a fun experiment to try definitely!)

[Wohlstand]

The purpose of Fake-DAC is only reducing the complexity of the PCM playback with emulators (in comparison with sample-by-sample posting into DAC interface of the emulated chip, which will take extra CPU usage because of unnecessary logic that can be made by simpler way) and giving ability to optionally use it with OPL3 chip for some tweaks (as OPL3 has no DAC, it's the only option to stream PCM independently but synchronously).

But fake-dac? It's nothing special to add two ouputs of synths together. Beside, why is the insistence to tie it to the wavetable concept? It could imagine it to be any kind of generator, including from audio port input.

Through ANY way (Fake-DAC that is playing separately and mixing with post-generated output of all chips, or Real-DAC which is streaming directly into DAC interface on one of simulating chips) is possible to play anything with no matter how that will be synthesized or decoded.

I use as example an analog Moog phatty synth, where there is a kinda similar concept: an input jack port is for an externally generated oscillator 3, which goes to be processed in the mixing and filter stages. I can plug a this jack to any synth of my liking.
(plugging the Moog synth into YM2612 would be a fun experiment to try definitely!)

Just for some experiments it's would be nice to try and listen the result 😄

[jpcima]

This is my personal idea about realizing this process.

• The "Fake-DAC": allow a DAC-enabled channel to transfer its events to MIDI hook functions. So, a synthesizer instance can receive and synthesize a channel in parallel (which can be Fluidsynth, simple WT, hardware, etc). Mixing can be fully external so libOPNMIDI doesn't have to care.

• Real-DAC: MIDI hook like above + frame generator hook set up by caller. The hook asks for the next frame to feed in the emulator, and there is not a significant increase of complexity. It leaves all the responsability of managing synth complexity on the user side.

[Wohlstand]

Just now I have updated DAC demo and I have allowed it to play files of different sample rates (every sample is passed to the chip with the delay declared by sample rate ratio between of native and input stream).

raw_pcm_u8_different_rates.zip

As I hearing by myself, I think the 16000 rate would be fine for OPNMIDI, tomorrow I'll try out some things again. But I'll begin the real work after we will release both libADLMIDI and libOPNMIDI stables.

You can try to generate any RAW PCM stream by yourself using ffmpeg as example. The stream is allowed unsigned 8-bit mono only.

[jpcima]

About the real DAC idea, some questions.

• when the midi channel has a DAC-enabled instrument, it has channels 1-5 as FM and 6 as special.
  as the normal notes are dispatched to the channels 1-5, do at the same time all of them are sent to the DAC master of channel 6? (being a synth of some kind)

• how is decided the ownership of the unique DAC channel, when banks have several instruments which demand the use of DAC?
  at what point is decided a channel has stopped using DAC, and ownership transfers to another?

[Wohlstand]

• As here is multichip case, yeah, enabling one DAC on a first chip and stream into it. Then, the channel categories thing can limit FM writing into that channel
• We are will use one DAC channel of one chip to stream mixed output. It's not need to have something extra: when we are using DAC-enabled instruments, we are turning DAC channel, and keeping it be always on.

[Wohlstand]

As option, I want to make the instrument have both FM and DAC implementations to be able have the "FM-Only" option that will disable usage of DAC and tell using FM implementation of instrument [which will be kept as fallback for this case].

[Wohlstand]

Move discussion from here: #58 (comment)

Hmm.. do you not think it's good to use an existing sample specification?
After all, sampling specification can be more complex than it seems, because of all looping consideration, and speaking of such things as envelopes, filters.

Existing are gig, sf2, sfz.. The later is text-based as format with sound files collections.
They have support libraries out of open source code (linux sampler), they have ready editors for use (polyphone), synthesizers

Me, I would not find a good idea to reinvent such models in weaker ways.
I'd express such a thing as: layer this instrument with program N out of such sample file.
Even if the synth is custom, it can be polished to infinity after the base is made.

Using of existing instrument format would be an optional extension, but it's not for mainstream. Why? The whole purpose is to don't depend on any external things as it's much easier to able to build the bank and use it without of any external tools. Another idea: external instrument formats can be used to import existing samples and convert them into internal format on the fly. WOPN store is designing for easiest using inside of libOPNMIDI with minimal effort to prepare data for the workflow. The stuff that will be made on OPN2-BE will be designed to manipulate internal format. The first implementation would to lack envelope support to provide basic for PCM percussions, and then extend the thing with melodic stuff.

[Wohlstand]

The draft for new WOPN format version where are PCM-bases instruments (and some other things) will be supported:
https://github.com/Wohlstand/OPN2BankEditor/blob/d8a3e0aa1c90d7c144b722fda6564129b612ce7f/Specifications/WOPN-and-OPNI-Specification-draft.txt

[jpcima]

I have insisted on such a format as sfz for such reason: I want you take a look and evaluate.
You don't need an external library to make this format work, nor editor.

A sfz is next to trivial for saving and loading process. A text editor can write it.
You can easily make an editor into OPN2-BE which saves into this format, even it just 1% of specification.
An example will show you how simple is a hello world sample: https://en.wikipedia.org/wiki/SFZ_(file_format)

Here will show you all ways it can be specified, and how all perspective of extending WT will be already existing. Imo, WOPN cannot possibly offer such an expressivity as a good sampler can have, in any short term work.

In such way, and as was my point I defended, one can implement a minimum WT synth, and have all room for adding support a few at a time.

The only thing, this format is not self-contained, but as I'm concerned, I don't find it an inconvenience to carry multiple files, rather that one large multi-megabyte file wopn.

[Wohlstand]

Checked out the WOPNv3 draft and yeah, about of PCM side:

• instrument supports only one PCM chunk (it's would be nice to support many of them with range allocation like done in most of WT synths. That needed for melodic voices, not for drums)
• instrument supports only one layer (can't have multiple PCM chunks are playing in parallel)

Have to improve the format with adding of layers and ranges per chunk

[Wohlstand]

Done!
https://github.com/Wohlstand/OPN2BankEditor/blob/master/Specifications/WOPN-and-OPNI-Specification-draft.txt
Anything that will be needed, feel free to modify the spec file 😉

[jpcima]

@Wohlstand yes, but I offer you the ultimate proposal. Reuse sfz, the existing standard.

It's simple how: you reuse exact same opcodes as it has them specified. In the event of not being enough, it's permitted to add more by extension.

Then you can serialize sfz into binary, and same with samples.
Simple: it's a dumb key-value store with strings, ints and floats.

As such, you have already a full specification, you don't reinvent sampling.
Bonus: you can make your own editor, you can also pick sample banks edited by people by their own external software.

[Wohlstand]

About of sfz: I'll adapt the internal store for sfz thing. But yeah, as WOPNv3 gives the support of difference instrument data in dependence on the header, then yeah, it's possible to put absolutely anything on the instrument part.

Then yeah, it's would be next key=value store in next format:

uint16be | count of entries
{
  1         | type
  NT-String | key
  <sz>      | raw data in dependence on a given type
}

Where are speaking on PCM chunks, there are will be stored in a bottom of WOPN file in given format, and will be re-used by indexes in the instruments.

[jpcima]

This seems nice 👍 If it's deemed useful, it can have a solution for compacting the keys.

I've thought of this: what do you think of releasing future WOPN in 2 versions?
A first version could be only for flags, velocity offset, volume model, and the ps8op.
The little things get immediate use in the OPN2 VST which will be finished in very short term.
In second step, a draft wavetable spec and synth can be developed together without a rush.

[Wohlstand]

The V2 format is still fine as mainstream even lacks those flags are unused yet, the V3 can be added as "experimental" while the development process will go. Anyway, as I said, the format allows to don't include parts are not specified. I.e. the same V3 is able to be saved with no any PCM-related stuff while it is WIP. I did the variadic size support per instrument where data blocks are not writing into file when there are disabled to reduce size of the file (I am even able to save blank instruments as 1-byte entries!!!) and also, the WIP PCM part can be changed on the side after main release.

[ghost]

this still being looked into?

[Wohlstand]

Yes, it's is! For now the work on the side of OPN2-BE to implement support into bank files to make PCM-based instruments be stored into them.

[freq-mod]

As PCM sample playback issue keeps appearing, I would like to propose something:

No need to dedicate multiple DAC channels on multiple chip emulators, needed only one on any chip to don't steal so expensive FM channels from every simulated chip.

Idk if it's good idea. Most will use PCM DAC for drumkits and one channel won't be enough, especially with GS/XG midis. So:

• Ch6 PCM mode can be enabled for any chip, most devices can handle at last 6 or so MAME 2612 cores, so FM channels count isn't bothersome.
• There can be a fallback option - an instrument by default would play PCM sample, but in case there are no PCM channels available, it would fallback to FM synthesis alternative.

also YM2608 rhythm rompler can be supported for a start, it can't be hard 👀