feat: add advanced stem loading COs

[acolombier]

Currently, this goes with a mix of extending the existing load signal/slot and creating a second one, called subsequently. Is there any preference? Should I change the implementation to use only one? (only extending existing signal or only subsequently emit)

[acolombier]

Please note that UI changes (1st and 3rd commits) will be moved to #13515 once this PR is ready to be merged!

[daschuer]

I have some doubts if this is consistent with the other PRs and the overall user experience.

The users sees only the track in the library which has four stems. The use can now load a single stem form a file or load the whole stem file and mute all 3 of them.
In the first case they cannot add a second stem in the second case they suffer the penalty of stem processing. They need a brief understanding of the engine interna to make the right decisions during mixing which could be limiting.

Can we join both? How can we allow to seamlessly switch between these cases.

A typical use case is probably:

• Play/Cue a full track
• Notice unusable part
• Switch to stem mode an silence that part
• Instant double the track in a second deck
• Bring the silenced part in using an effect

How can we allow this using the new single stem mode introduced here behind the scenes?

[daschuer]

That looks not right. How about use two lines? We may also consider to reduce the use of __STEM__ where it has no performance impact like here. We already have the default = 0.

[acolombier]

Make sense. So would you prefer I update the signature globally?

[daschuer]

Yes in general. I made this comment before raising my general concerns regarding this single stem mode. Maybe we need a slightly different signature .. unsure..

[acolombier]

I will wait for the full review before starting refactoring then :)

[acolombier]

How can we allow to seamlessly switch between these cases.

The first scenario is the only way on non-primary decks (sampler and preview), while second is the default for primary deck. realistically, I only added the second option for primary deck for feature parity and enabled that advanced capability, or workaround for light platform which would struggle with more than one stem deck + rubberband (eg. raspberrypi)

How can we allow this using the new single stem mode introduced here behind the scenes?

By default, the load process will still load the full stem file. Loading an individual stem should be reserved for "aware users". A typical use case is to load vocals or drum only in a sampler, or a deck if you want more transport control.

By the sounds of it, the use case you are describing (clone switching between individually loaded stem vs full deck with muted tracks) are falling under the the split use-case we've discussed in the epic, or am I reading this wrong?

[daschuer]

Ah, OK I forget about the sampler use case.

Than let's think about a way to gerearize this topic. Since it is handled by the same engine class anyway.

How is the transition from single stereo to stem now done? Is there way to keep the transport going during the switch? A switch from the full track to a stem mix is probably the same as a switch from a stem mix to one of the other stems. Do we need to consider something like "load as karaoke" where we process three of the stems?

I am not demanding to implement it here. I just like to brainstorm possible future solution to have the Co interface proved for that.
My feeling is that current one is too static.

Idea: can we use the mute controls for selecting the loaded stems?

[acolombier]

How is the transition from single stereo to stem now done?

Currently there is no way transition "live" for a stem deck to a stereo deck with a track. With that current feature, can chose the load a track in an empty deck. If we think of a case where one would want to switch from a stem deck to a stereo deck, they would need to use an empty deck and handle the transport synch.
In the future, we could consider the ability to do this live on a deck, but I would emit some reserves as of whether or not this is possible. Particularly, I'm not sure how we would handle the CachingReader sync, and the arming of the RubberBand. I think it safer to expect that such a transition should always be done on another deck.

Is there way to keep the transport going during the switch?

It would be fairly trivial to implement to some kind of clone_as_stereo that makes the transport cloning as well, as the existing cloning does. I'm not planing to implement cloning feature as part of this PR or the GSoC as I have recently less time (sadly back to a full-time job...), but I'm hoping to still get something together before the end of Q3.

Do we need to consider something like "load as karaoke" where we process three of the stems?

I think it would be great to even go further an allow loading a set of stem pre-mixed to stereo (eg. 2+3, 1+3+4, ...), but same as above, I think it is big enough to split it as a future feature.

Note that in case not all the above feature wouldn't be ready for 2.6 and we would like to wait for completeness before releasing, the STEM build option should still be working and allow us to keep that feature behind curtain till 2.7. Personally, I hope we can release what we have in 2.6 but it's purely subjective and selfish opinion! :)

Idea: can we use the mute controls for selecting the loaded stems?

This is already how we were planning to make the split_* CO work. Say you are playing 1+2+3+4 on A, if you split 3, you would end up with 1+2+4 on A with 3 muted, and B playing 3 but 1+2+4 muted, allowing you to play with the transport between stem track, while still allowing you eventually to "merge" back to a unify transport (by unmuting 3 on A, or 1+2+4 on B)
Now that would require a fairly powerful processing power if we consider RubberBand-ing 8 stereo channels, so I guess we could add a variant (or a user preference?) so splitting would lead to having 1+2+4 on A with 3 muted, and B playing just 3 as a stereo deck (meaning you can only merge back to A and cannot add new stem tracks to B's transport)

[JoergAtGithub]

Let's look again on the use cases for stem loading, and deck clone/splitting out of this PR.

Let me list all uses cases I know:

1. Loading mixed track of a stem file into sampler

2. Loading a single stem into a sampler (e.g. a vocal or a beat) -> Re-Using beats from know tracks is something, where I see a real benefit for the DJ

3. Playing many samplers at the same time without performance issues (may use time streching with rubberband)

4. Loaded a selected list of stems of a stem file into a sampler deck - Karaoke case (all stems can be mixed at loading time, because no stem controls are available in sampler deck

5. Support all the cases for sampler decks for the preview deck as well - A DJ might want to preview before loading them into samplers

6. Loading the stem track ino a main deck in stem mode -> Core stem functionality

7. Loading the mixed track of a stem file into a main deck -> This is what Traktor Pro 4 introduced as new feature (there Shift + Drag+Drop) to give the DJ the normal deck behavior for playing the unmodified track

8. Loading a single stem into a main deck without stems controls -> e.g. to load a beat from a known track and use it for scratching with vinyl control (which samplers don't have)

Cases which are technical possible to realize, where I don't see a use case yet:

10. Loading multiple, but not all stems of a stem file into a main deck - Why not load all and use the existing mute CO to silence the other (only difference is that the UI controls for the silenced track appear)
11. Loading stems into the sampler with stems functionality - how to use them without stem GUI controls? Also normal hardware controllers will not have knobs for more than 4 stem decks (2x4x4=32 knobs needed for 4 stem decks)

We should really focus in this PR, on what's provides a real benefit. But as we talk about COs, which we must maintain backward compatible in future, we should check if we can extend them for future use cases.

[daschuer]

I don't have the demand for on the fly switching the STEM channels in this PR. I can confirm that seamless switching will be hard to implement especially when we consider effects and pitch shifting. What we may achieve in future is to introduce a portion of silence, but keep the crowd dancing in beat/the track cued in headphones or such.

I like to have the CO interface ready for this, but implement only the engine features already in this PR. The engine can be updated at any time, the CO interface not.

[daschuer]

Thoughts:
We may expect that the user has already mapped the various loading and cloning COs to a certain extend.
We also have integrated it into the GUI. They want to continue to use them for stems.

We just(TM) need to add the information which stem configuration they like to use.

So we may add global flags CO to decide which Stems should be loaded. However I am afraid the user will always forget to adjust this field before loading. The (My) natural way would be to decide that after loading.

Is it an issue to repeat the action with the right flags set? Yes, if the user had lost library contact in between.
So we need a kind of "clone to the same Deck" control that respect the loading flags.

Using the mute buttons would be also natural, but the user has no access to all the mute buttons for all samplers. In addition they are also used for quick effect actions like the EQ kill buttons.

Idea:

• Add a global stem load flags CO
• Allow cloning to the same deck

This means:

• Load always the stereo down mix by default.
• Allow to adjust the loaded stem before loading via a stem flag CO.
• Allow to correct the choices by clone the track to the same deck with stem flags adjusted.

Does that make sense?

[Eve00000]

I just played intensively with stems (it's so mega cool)
I agree with Jörg on point 10 & 11 : useless
I do load the vocals and melody synths in samplers (at the moment parts of the stems, selected leftovers of the stemconsturction),
stem-parts in samplers are only usefull when the can have saved hotcues and loops.
-> Is this already planned?

Cloning a part of the stem to another decjk is not useful, the complete track should be cloned with the stemsetting (wich stem-tracks are muted, effects, volume). It is usefull because I played with loops with the slipmode. And confusing the audience is fun.
They start singing but the vocals come later while the beat continues + in this way I could use 2 effects on the vocals (and other
tracks), 1 in deck1, 1 in deck 2 fun!

I do have the impression that playing intensively with stems takes quiet some extra cpu, I am affraid the minimal configuration will have to be strong enough, we'll have to communicate this because else people will be disappointed.
-> Remark: I play with lossless stems.

In Zulip you can find my first version of MixxxInTouch, though I learned to play around well with my F1's I used my Ipad with it as extra to control the sampler hotcues (2nd page: soundboard = 32 hotcue buttons) + to reset the stem-tracks volume to maxxx /set mute off.

One of the coolest parts is the mixing of the drums/bass, it has become a 'piece of cake' but I am not so sure that BPM's are still correct when only playing a bass pr drum stem-track (not together), I could not rely on the computed bpm.

Question: why would you load the 5th track (original mixxx) in a deck? When you load this track you could regret while it's playing because you just thought of a nice effect.... I think it can be a user-preference, but I think I'll never use it. You jyst have to get the habbit to reset volume and mute when preparing a track (waveform is great help).
So maybe an extra reset-button (stem-volumes to max / unmute tracks / disable effects) can be useful.

PS: My Ipad is a perrfect visual aid to see thet different volumes of the tracks, I updated it to see the fx settings on the same page as well, On a 15inch it becomes all to small.

[JoergAtGithub]

So we may add global flags CO to decide which Stems should be loaded. However I am afraid the user will always forget to adjust this field before loading. The (My) natural way would be to decide that after loading.

It should be possible to map the complete fully qualified action to buttons of a controller. And we should consider race conditions if we need to set multiple COs for executing a single operation, e.g. load buttons are pressed on two CDJs at the same time.
Furthermore, the default action will likely differ between different decks or deck types.

• Load always the stereo down mix by default.

If a user spend the effort to create a stem file, I doubt that playing the stereo mix is what he expects as default. Or do you mean this only for sampler decks?

[acolombier]

Loading multiple, but not all stems of a stem file into a main deck - Why not load all and use the existing mute CO to

Because of performance reason - if you are playing an "instrumental" version of a track, (that is keeping stem track 4 muted all along):

• With the stem deck + muted 4th stem track, you need to run 4 RubberBand (even 3 if somehow - likely impossible currently - we manage not to scale the 4th) (CPU) and 4 stereo track in the cachingreader (Memory)
• With the stereo deck, you need one single rubber band and one stereo track in the caching reader

how to use them without stem GUI controls? [in sampler]

This is what this PR addresses - you load a stem track into the sampler (e.g the vocal). If you now want to also have the drum, you can load it (e.g in another sampler)

stem-parts in samplers are only usefull when the can have saved hotcues and loops.

AFAIK, this is already supported.

Cloning a part of the stem to another decjk is not useful, the complete track should be cloned with the stemsetting

Appreciate you may not benefit it, but I personally would. In any case, we are planning to support both, starting with your use case anyway.

I do have the impression that playing intensively with stems takes quiet some extra cpu, I am affraid the minimal configuration will have to be strong enough

This is one of the reason why cloning a specific stem track to a stereo deck would help! Beside the RubberBand optimisation, running 4 decks of stem remain pretty much impossible!

So we may add global flags CO to decide which Stems should be loaded

So you don't like the solution offered in that PR? May I just ask, what is the fundamental problem?
[...]

Load always the stereo down mix by default.

[...]
We could definitely add a setting that allow you to chose the default behaviour of LoadSelectedTrack (e.g Load as a stem deck, Load main mix as stereo, Load first stem as stereo, ...), but I would make a strong point on defaulting it to load a stem deck, as I have the same feeling than @JoergAtGithub on this - if I'm going the extra mile of getting a stem file, this is usually not to play it as stereo.

---

I really appreciate the discussion here, but this has drifted quite far away from what that PR is about! I have managed to achieve the stem project in an iterative manner, and I'm not planning to change the approach on this PR :)

So please, let's move the discussion to the epic and keep discussion here related to selective load and load behaviour only ! (cloning, splitting, cueing, ...)

[JoergAtGithub]

Above I listed 8 use cases for advanced stem loading COs. Since nobody brought up another use case relevant for loading, I guess it's complete. Of the list, the cases 1,2,3,5 & 6 are handled by the COs implemented in this PR. The uses cases 7 & 8 are also fullfilled, but by using the existing main deck stems constrols instead of loading into a dedicated deck mode.

What remains is the "Karaokee" use case 4 for samplers, which requires loading multiple stems, mix them at load time to use the sampler with low CPU effort.
I'm unsure how relevant this use case is, but to support it (in future), we could define the CO as follows:

[SamplerX],load_selected_track_stems with the following argument:

• -1 Load premastered stereo mix from stem file
• 0 Load with default mode (according deck type preferences)
• 0bnnnn Load the stems according the binary coding, and mix them at load time. e.g.:
  • 0x0001 for loading vocals only
  • 0x1000 for loading drum only
  • 0x1110 for loading instrumental only (Karaokee case)

The drawback would be, that this complex argument can't be mapped easily in the MIDI XML and would require the use of JavaScript.

[acolombier]

Sounds good @JoergAtGithub
I think it should be pretty straight forward to update the current selectedStemIdx to a selectedStemMask so I will look into this.

It might take me some times to work out the UI changes so I might make it CO only for now, and keep the UI with individual stem selection.

[daschuer]

It should be possible to map the complete fully qualified action to buttons of a controller.

Yes, I am just in doubt that we need it for every load action we have currently. The user won't have as many buttons he need for all dedicated actions so we also need to allow a way to use two buttons to achieve the action kind of intuitive, ideally.

And we should consider race conditions if we need to set multiple COs for executing a single operation, e.g. load buttons are pressed on two CDJs at the same time.

Not sure if this is s real issue. But I agree that we need to consider it.

If a user spend the effort to create a stem file, I doubt that playing the stereo mix is what he expects as default.

I don't think we should do this assumption for some reasons:

• Creating stems is not lossless (in most cases). So it might be desire to use the stems only during the transition and later play the track normally for a better quality and to save CPU.
• If we imagine in future on the fly stem creation this assumption becomes void. We may make it true if we show the stem version and stereo version in the library.

So we may add global flags CO to decide which Stems should be loaded

So you don't like the solution offered in that PR? May I just ask, what is the fundamental problem?

My fundamental problem is that I have no idea how to use these controls in a real mapping and how to do a GUI for it. Maybe we need a GUI feedback for that?
The other issue is that the new COs implies a workflow that feels unnatural for me coming from the current load button,s I have currently mapped.
I also don't see how we can do the transition in future from single to multiple stem mode during a mix.
My hope was to do it with one set of controls.

[daschuer]

@JoergAtGithub I can confirm your consideration about the use cases. But a plan how to map it on controllers and use it in the GUI unclear. I am also concerned about a future plan of switching the stem mode while a track is playing. Maybe fixing this, instead of dedicated loading controls also covers all use cases.

@acolombier I am not demanding a change of the engine implementation. It would be OK to do a first version that loads a track again when changing the stem mode. Does that make sense for you?

[JoergAtGithub]

I don't think, that it will sound good to switch from the mixed stereo to the 5th track in a stem file. This is because the later is mastered by the producer and the mixed stems are not.

This PR is primary about sampler decks, which have no stem controls. I don't understand, where a usecase for full stem support in sampler decks is. It will just consume so much CPU, that we get buffer underflows.

[acolombier]

The user won't have as many buttons he need for all dedicated actions

That's not including combo - thinking of usecase of the S4 Mk3, one could press the pad 1 and 4 at the same time, before pressing the load encoder, to load pre-mixed 1 and 4 to a stereo track.

Maybe we need a GUI feedback for that?

GUI feedback is already implemented.

(Load action in menu)

(Waveform draws drum only, with other track with transparency)

Appreciate we may add more, perhaps stem color on context menu, any other suggestion?

The other issue is that the new COs implies a workflow that feels unnatural for me coming from the current load buttons

Have you considered combo? (e.g press shift+X+1) I know this is not possible on the MIDI engine, but this is specific to it and its limitation, currently being reworked in engine v2
With HID+JS, this is certainly possible and something I have already implemented for FX control on the S4 (#13126)

It would be OK to do a first version that loads a track again when changing the stem mode.

I'm a bit confused.
That's what load_selected_track_stems with -1 or 15 would do, right? Or what LoadSelectedTrack would do when setting [Mixer Profile],default_stem_load with -1 or 15? Happy to add that to this PR yes, although no time for it at the moment.

I don't think, that it will sound good to switch from the mixed stereo to the 5th track in a stem file. This is because the later is mastered by the producer and the mixed stems are not.

Yes good point. As far as current stem implementation, it's impossible to load the pre-mixed stereo track. If one request a stereo track, then 1 to 4 are pre-mixed by the audio source. We won't be able to change that till we have DSP support, otherwise analyse like replaygain do not work and provide very poor result.

[daschuer]

I don't think, that it will sound good to switch from the mixed stereo to the 5th track in a stem file.

Why not? The first track is the original track so it has always the best sound. It is the original! Playing all four stems together is IMHO not recommended. However the transition from 4 stem to original mode will be notable because of the missing mastering info and the additions of the sounds not part of any of the four stems and should be finally users choice.

This PR is primary about sampler decks, which have no stem controls. I don't understand, where a usecase for full stem support in sampler decks is.

I acknowledge that the plan was to do this PR only for samplers. But it is now for all decks. Which is welcome.
Let's just finished this discussion about the best CO interface that is future proove.

What do we have?
The load action menu is already nice. This is independent from the CO interface.

The proposed CO interface is currently:
load_selected_track_stem_%1_and_play and
load_selected_track_stem_%1

I am in doubt that these controls will ever be used in a direct 1:1 mapping. Do you agree, or do you have a special use case in mind?

All controllers have already working mappings to load and clone tracks. My assumption is that the user expects that this remains the same for stem tracks.
So let's design the CO interface around that.

That's not including combo - thinking of usecase of the S4 Mk3, one could press the pad 1 and 4 at the same time, before pressing the load encoder, to load pre-mixed 1 and 4 to a stereo track.

That's a good start. What do we need for this?
I still think a global stem load mode CO will be useful. But let's consider alternatives. Maybe with GUI feedback or even with a GUI or keyboard control.

For the deck/sampler context menu the new load Actions are IMHO also very useful using the already loaded Track. This covers the use case the track was dropped to the deck and is not longer visible in the library. Can you confirm that?

[JoergAtGithub]

I don't think, that it will sound good to switch from the mixed stereo to the 5th track in a stem file.

Why not? The first track is the original track so it has always the best sound.

Yes, but because of the this difference a DJ needs to use two main decks, adjust pre-gain, effects like limiter until both decks sound similar, and than cross-fade from one to another deck. If you executed this transition automatic in one deck it will sound awful.

Note that no other DJ software has such a feature, they always mix the stems. The closesd functionality is in Traktor Pro 4 which loads the orignial track to a main deck, if you hold down Shift at track load.

I am in doubt that these controls will ever be used in a direct 1:1 mapping. Do you agree, or do you have a special use case in mind?

Why not, there are dozens of controllers with large pad arrays, intended to assign each button to a sampler. For example an F1, where you could select the stem(s) to load by the top row of 4 buttons (toggle on/off), than press the load button and assign the sampler.

I can't imagine a more common use case for stems n samplers, than prepare the samplers with beats or volcals that you need in your set. And than run them with a single button press during the set.

All controllers have already working mappings to load and clone tracks. My assumption is that the user expects that this remains the same for stem tracks. So let's design the CO interface around that.

The old COs will not change. We can't add arguments there for backward compatibility.

That's a good start. What do we need for this? I still think a global stem load mode CO will be useful. But let's consider alternatives. Maybe with GUI feedback or even with a GUI or keyboard control.

A global setting does not match here, because the decision which stems to load in a sampler will different each time you load a track. A mapping developer would have to call always both COs. Furtherore two COs that depends on each other are prone to race conditions. We have multi-threading here (e.g. 4 threads for 4 Traktor F1 decks), COs should execute stateless implementations only.

[daschuer]

If you executed this transition automatic in one deck it will sound awful.

It depends on the way the stem was created and it is IMHO up to the user to make the transition to the original not sounding awfull. I can also imagine a transition from single stem to original. But anyway, my point is that a seamless mode transition in one deck is a desired future feature and we need to consider it now in our CO design.

I am in doubt that these controls will ever be used in a direct 1:1 mapping. Do you agree, or do you have a special use case in mind?

Why not, there are dozens of controllers with large pad arrays, intended to assign each button to a sampler.

I think there are many other interesting 1:1 mappings on such controllers instead of using them for four load controls per sampler.

For example an F1, where you could select the stem(s) to load by the top row of 4 buttons (toggle on/off), than press the load button and assign the sampler.

That's exactly a mapping I had in mind. You example is not a 1:1 with the proposed controls. But it would be a perfect 1:1 mapping for a set of load controls.

For now the engine does not support to load two stems out of four for example. Not sure if we want to add the karaoke mode in futur or use full stem mode for it, but it cannot hurt to consider it in the CO interface.

A global setting does not match here, because the decision which stems to load in a sampler will different each time you load a track.

I agree

A mapping developer would have to call always both COs.

Not really, because the user does it. Even with the current CO solution the mapping needs to implement a state full solution that works the same.

I can confirm that the race condition may happen if the mapping does automatic loading in between. But for now I don't see a use case for such a automatic that sets the COs concurrently. I value the benefit more that the any existing mappings can use the feature without mandatory scripting

But anyway. I am happy that we have now a common understanding of the issue. We have two ideas on the table. Maybe we can find a optimum third idea?

[JoergAtGithub]

For now the engine does not support to load two stems out of four for example. Not sure if we want to add the karaoke mode in futur or use full stem mode for it, but it cannot hurt to consider it in the CO interface.

Did you miss the proposal in: #13268 (comment)

[daschuer]

Did you miss the proposal in: #13268 (comment)

No. The issue with this proposal is that it also requires scripting and rewrite of current load mappings.

Let's consider the concurrent issue. This can be an issue if the user has selected a load mode on one controller and than an automatic feature resets the decision. This can be a radio script loading a jingle to a sampler or such. Since such a script needs also select the track, it is probably the beast to offer a JS API to do it without COs.

The issue is that the selected track itself is also statefull and the load COs cannot be used.

So we may consider to select the stems to load like the track itself. This means reset the control to a reasonable default whenever a track is selected.

What do you think?