4 Stem Separation Mode only works once with CPU as Inference Device

[Rosenheim23]

Hello,

I recently came across this openvino plugin for Audacity.

I am using it to separate the drums from music tracks and have encountered the following problem:

If I select the CPU as Inference Device in the 4 Stem method, the separation works once. If I then do the same process again, whether with the same piece of music or a different one, I only get noise in the 4 separated stems. Only when I delete the .blob files from the cache in the profile directory does the separation work again for one time.

If the GPU is selected as the inference device, the separation always works.

I have tested this on 2 notebooks, one with an 8th generation i5 CPU and Windows 10 and one with an 11th generation i5 CPU and Windows 11.

I use the CPU because in my case the GPU is significantly slower.

Many thanks and best regards
Christian

[RyanMetcalfeInt8]

Hi @Rosenheim23, thanks for reaching out. This is strange behavior... I tried to reproduce it on my 11th Gen i7, but it seems to produce good separation results on back-to-back runs (i.e. when picking up pre-compiled blobs from cache dir).

Couple follow up questions though.

1. Can you give exact CPU model used? For example, in my case I'm using i7-1185G7.
2. Just curious what the format (mono/stereo, sample rate) of the input track is, and also the selected length. Can you also reproduce the issue for 'short' selections (say, ~10 seconds or so)?

Thanks!
Ryan

[Rosenheim23]

Hi Ryan,

Thanks for the reply!

1: The first notebook has a Core i5-8250U CPU and runs Windows 10 22H2. The second has a Core i5-1135G7 CPU and runs Windows 11 22H2. Both operating systems are German versions.

2: I use the plugin with stereo tracks in 44100 Hz/16-bit PCM format. The problem occurs regardless of the length, so it doesn't matter whether it's a complete 4-minute track or just a 10-second excerpt.

Perhaps it is also important that I use a self-compiled version of Audacity because I need the ASIO support.

When I mix the defective tracks together again, the music can be heard again, but with background noise.

The problem also occurs when I use the 2 stem method.

Thanks again and best regards
Christian

[RyanMetcalfeInt8]

Hi Christian,

Hmm, interesting. I highly doubt that using an ASIO-enabled custom build of Audacity would affect the behavior here... as it seems to be related to OpenVINO's caching mechanism. I have a very similar setup (tiger lake i7, Windows 11), and so I'm surprised we see different behavior.

Maybe to be 100% sure, it would be worth it to test on your side using the release builds of Audacity? But again, I'd be shocked to see any behavioral differences..

I'll also think on this a bit more, and try to come up with some way to half-split the problem..

Thanks,
Ryan

[Rosenheim23]

Hi Ryan,

I just installed an official version from the Audacity website and reset all settings. The problem also occurs with this.

Best regards
Christian

[RyanMetcalfeInt8]

Hi Christian,

Sorry for the late reply. Any chance you can post a brief video showing the problem? I still am unable to reproduce this...

Thanks,
Ryan

[Rosenheim23]

Hi Ryan,

No problem.

As a workaround, I wrote a script that clears the cache when Audacity is started. Since it only takes a few seconds to create the cache file, this is not a major limitation and the tool is perfectly usable.

Here is a .zip file of the recording:

Audacity_Screencapture.zip

Unfortunately the sound was not recorded. I have therefore attached the drum track from the first and second conversion.

If you need anything else, just let me know.

Best regards
Christian

[Rosenheim23]

PS: I also tried the whole thing again on a third notebook with an old i5-3210M CPU and a fresh Windows 10 64 bit installation.

The problem also occurs here.

Best regards
Christian

[RyanMetcalfeInt8]

Hi Christian,

Here is a .zip file of the recording:

Thank you for taking the time to do that. It's very helpful.

Out of curiosity, given that it's a German version of Windows, are there any 'special' characters in %appdata% path? I had heard recently that there was an OpenVINO bug around certain kinds of special characters (such as chinese characters) in the model path -- maybe it could be something similar?

Do you encounter weird caching behavior for other effects? For example, if you ran whisper transcription on the vocal track for 'Under Pressure', do you get consistent results for back-to-back runs?

Thanks!
Ryan

[Rosenheim23]

Hi Ryan,

There are no German special characters neither in the %appdata% path nor in the Audacity application path.

I hadn't used any of the other functions yet and just tried the whisper transcription as you suggested.

And yes, using the CPU shows the same behaviour. The whisper transcription works the first time I use it. With subsequent applications, I either get no result at all or just a jumble of characters. Only when I clear the cache does the whisper transcription work again.

If I select the GPU in OpenVINO Inference Device, the whisper transcription always works without deleting the cache in between.

Best regards
Christian

[RyanMetcalfeInt8]

Hi Christian,

Thanks for the quick reply. It's very useful to know that there seems to be some overall caching issue happening, as opposed to something specific to the music separation model.

I've submitted a question to OpenVINO issues page here (openvinotoolkit/openvino#21942) to understand if there are any known issues with model caching on OpenVINO 2023.1.

Thanks,
Ryan

[Rosenheim23]

Hi Ryan,

Thanks again for your work.

As I said, the plugin is already very useful for me as it is.

I had previously used an online service for separating the stems. That wasn't really any faster and the free version certainly didn't deliver any better results than your plugin.

I have now also subscribed to your question at the OpenVINO project to see if there is any news there.

If I can be of further help with any tests or you have any further questions, please contact me again.

Best Regards
Christian

[RyanMetcalfeInt8]

Thanks Christian! Glad it's still usable despite the caching bug.

At some point in the next week or so, I might compile the plugins against the latest OpenVINO release (2023.2), and post a new DLL package here to see if that resolves your issue (if you're willing to test).

And I'll keep trying to reproduce this on my side (I just tried on my wife's laptop 😆 -- 7th gen i5 -- but didn't hit the issue there either).

Best Regards,
Ryan

[Rosenheim23]

I am happy to test the new release and post my experiences here.

Best Regards
Christian

[akash07k]

Hi @RyanMetcalfeInt8
I too am impacted with issue, but on 2 Stem separation mode also.
I'll create a new issue for this though.

[RyanMetcalfeInt8]

but on 2 Stem separation mode also.

Note that under the hood, 2-stem and 4-stem run the same model (the 4-stem model). The 2-stem just applies "mix-and-render" for the drums, bass, and 'other instruments' tracks.

tl;dr -- Any OpenVINO caching issue would definitely affect both 2-stem and 4-stem.

[RyanMetcalfeInt8]

Hi @Rosenheim23, @akash07k,

Sorry for the delay in getting a test package built with latest OpenVINO (2023.2). Here it is, if you're still willing to try it out to see if it resolves the caching bug:

Audacity-OpenVINO-2023.2-Module-3.4.2-R1

Install it in the same way as original Audacity-OpenVINO-Module-3.4.2-R1.zip, but allow it to overwrite any existing files. You'll need to set mod-openvino to 'Enabled' again after install (and close / re-open Audacity).

Let me know if you have any questions -- looking forward to hearing the results.

Thanks again!
Ryan

[Rosenheim23]

Hi Ryan,

Thank you for providing the new version. I have just tried it out.

Unfortunately, my problem remains unchanged even with this version.

Best regards
Christian

[RyanMetcalfeInt8]

Hi Christian,

Thanks for the super quick turnaround on that! Just so I'm clear, which platform / configuration did you try it on?

Thanks,
Ryan

[Rosenheim23]

Hi Ryan,

I have tried it on my two newer laptops. (Intel Core i5-1135G7 CPU with Windows 11 22H2 and Intel Core i5-8250U CPU with Windows 10 22H2)

Best regards
Christian

[akash07k]

Hi Ryan,
Thanks for the newer build,
Unfortunately, I too am still unable to get it working. The problem remains the same regarding the caching stuff.

[RyanMetcalfeInt8]

Hi @Rosenheim23, @akash07k,

Thank you for trying. I think as a next step, I might come up with a more 'standalone' test to check caching mechanism in isolation (i.e. a very simple test.exe). I want to make sure it's indeed a bug purely within OpenVINO itself, as opposed to something that the Audacity plugin is doing wrong which could be triggering such behavior.

At the very least, I'm thinking about adding the ability to disable caching for the next official drop. Either through some kind environment variable (this is sort of ugly). Or, I can envision this kind of setting sitting nicely as a 'plugin-level' setting (as opposed to exposing it as a check-box for each individual feature).

Thanks,
Ryan

[RyanMetcalfeInt8]

Hi @Rosenheim23, @akash07k,

Hope you're both doing well. There is a new OpenVINO release that may resolve some caching bugs. I've placed an updated zip file for you to try (if you're still willing to help check), here: Audacity-OpenVINO-2023.3-Module-3.4.2-R1.zip

Thanks in advance!
Ryan

[akash07k]

Thanks @RyanMetcalfeInt8 ,
I'll try this new build and will let you know the results soon.

Hi @Rosenheim23, @akash07k,

Hope you're both doing well. There is a new OpenVINO release that may resolve some caching bugs. I've placed an updated zip file for you to try (if you're still willing to help check), here: Audacity-OpenVINO-2023.3-Module-3.4.2-R1.zip

Thanks in advance! Ryan

[Rosenheim23]

Hi @Rosenheim23, @akash07k,

Hope you're both doing well. There is a new OpenVINO release that may resolve some caching bugs. I've placed an updated zip file for you to try (if you're still willing to help check), here: Audacity-OpenVINO-2023.3-Module-3.4.2-R1.zip

Thanks in advance! Ryan

Hi Ryan,

thanks for the new version.

Unfortunately, the problem has not improved in my case. Sorry!

I used the plugin on a complete reinstall of Audacity, but currently only on my Windows 11 machine. I will also test it on my Windows 10 machine in the next few days and let you know here.

Best regards
Christian

[RyanMetcalfeInt8]

Hi Christian,

Thanks for following up so quickly. I will let OpenVINO team know that the issue is still observed with latest (2023.3) release.

Regards,
Ryan

[teoskoki]

Hi Christian,

Thanks for following up so quickly. I will let OpenVINO team know that the issue is still observed with latest (2023.3) release.

Regards, Ryan

Dear Ryan,
Thank you so much that you've done so much to solve this problem.
I'm writing here to tell you that I have THE SAME step by step issue. Totally the same. In 4 words - "It works only once".

And I've done everything that you've suggested. Including your last sulution. And it helped only to reset the problem. After I installed your updates, the separation worked only once, and that's it.

Could you please let me know if the team (and you) already found the solution.

PS: I have Intel Core i5-7300U CPU Windows 7 Professional sp1 x64

Thanks in advance

[teoskoki]

In your #68 I've found an additional solution - to remove %appdata%\audacity\openvino-model-cache folder before running music separation - it really works. If I delete the folder each time before using Audacity. At least it works! So, I'm glad. And thanks again.

If there is (or could be as a result of this unsolved problem) a plugin which deletes this folder each new time automatically, it would be great.

[RyanMetcalfeInt8]

Hi @Rosenheim23, @teoskoki, @akash07k,

Hope all is well with you. I just want to let you know that there is an updated version of these plugins available here: https://github.com/intel/openvino-plugins-ai-audacity/releases/tag/v3.5.0-R2

I wish I could say that I am confident that this 'cache' issue is resolved, but I still haven't been able to reproduce it on any of my running systems since it was first reported here. There have been plenty of other users that have hit it though, unfortunately.

Since we have upgraded to OpenVINO 2024.0, there's a chance that there was some bug fix included there that will resolve this issue... it would be very helpful to us if someone could let us know whether or not the same issue is present in this updated release.

Thanks very much!
Ryan

[Rosenheim23]

Hello Ryan,

I have just installed the latest Audacity version 3.5.0 including the corresponding Openvino plugin v3.5.0-R2.

Also with this version the problem described above remains unchanged.

Best regards
Christian

[RyanMetcalfeInt8]

Hi Christian,

Thank you for giving it a shot. Hmm, I'll have to figure out how to get to the bottom of that.

Thanks again,
Ryan

[Helmut002]

Hi Ryan, we know each other from my problem with the openCL.dll that You solved in a very great way.
I have the same problem with the openvino-model-cache folder. Before I start an openvino-session I must delete the blob-file. It is not necessary to delet the whole folder - only the blob must be deleted. It is also not necessary to close aud restart audacity.
This problem exists on my both machines. One older PC with win10 aun one new PC with win11.

[RyanMetcalfeInt8]

Hi @Helmut002,

Sorry to hear that you also are experiencing this issue. Just curious, are you running the German version of Windows as well? I know that Christian, who reported this issue, is using the German version.

Thanks,
Ryan

[Helmut002]

Hi Ryan,
in both pc's I use the german version.
GerätenamePCW11.docx
GerätenameHelmut.docx
And I use audacity 3.5.1 with the right version of openvino (audacity-win-v3.5.1-R2-64bit-OpenVINO-AI-Plugins)

[RyanMetcalfeInt8]

Hi @Rosenheim23, @Helmut002,

I am finally able to reproduce this issue. It turns out, that if I simply select Deutsch as the language to install,

then I am able to reproduce the issue, exactly as you (and others) have described. On the same system, if I uninstall Audacity, remove the cached blobs, and re-install in English -- then everything works fine. It's very, very strange.

Furthermore, if I have English installation, run Music Separation (to produce the cached blob), and then uninstall & install Deutsch version -- then Music Separation consistently runs okay (as it now is picking up the blobs saved from English installation).

I'll dig into this more..

[Rosenheim23]

Hi Ryan,

that's good news!

Because I do not use an installer but a self-compiled version of Audacity, I have done a few tests myself.

It seems to be enough to change the language to English in Audacity. After that, the plugin works flawlessly several times in a row without deleting the .blob file first, even if I change the language back to German after running the plugin for the first time.

Best regards
Christian

[RyanMetcalfeInt8]

Hi @Rosenheim23,

Ah, good to know! I narrowed it down to a locale issue -- Looks like the OpenVINO caching mechanism is sensitive to the numeric locale that is set. I'll submit this issue to the OpenVINO project, and in the meantime I'll implement a workaround for this to be included into the next patch release.

Looking forward to finally getting to resolve this one!

Thanks!
Ryan

[Rosenheim23]

Hi Ryan,

I just came across your latest version of the OpenVino plugin and gave it a quick test.

The R2.2 version works fine every time now, even without deleting the .blob file in between tests!

Perfect! Many, many thanks for your troubleshooting efforts.

Best regards
Christian

[RyanMetcalfeInt8]

Hi Christian,

Thank you for confirming that it's all working now with the latest release! I had intended to add a comment here to let you know that the new release was available... I must have forgot to do that, so I apologize..

Anyway, thanks for all of your testing (and patience) -- I think we can finally close this issue!

Best Regards,
Ryan