SIR is always INFINITY from the "mir_eval.separation.bss_eval_sources" function

[cocosci]

Hi there,
For the mir_eval.separation.bss_eval_sources(reference, estimate) function,
SDR and SAR are always the same and SIR is always infinity...

I wonder,
1> besides reference, estimate, do I need to offer the noise file to the function?
2> the input matrices are not bounded to [-1,+1]. Is that Okay?

[faroit]

1> besides reference, estimate, do I need to offer the noisy to the function?

what do you mean by "the noisy"?

2> the input matrices are not bounded to [-1,+1]. Is that Okay?

I don't think this would cause a problem since the metric is mostly based on cross correlations.

[cocosci]

The background noise that is added to the clean signal.
For each signal, I added 10 types of noise.

[cocosci]

According to the document,
reference_sources : matrix containing true sources
estimated_sources : matrix containing estimated sources

so how will the toolbox find the noise I added and calculate SIR? Thanks!

[faroit]

I don't know what your application is, but bss_eval is not perfectly suitable to evaluate denoising applications.

If you still want to do it, make sure to pass 2 signals to both reference and estimates, where true sources are [clean vocals, interfering noise] and estimates are [predicted vocals, predicted noise]

also see #256

[cocosci]

Yes, I am trying to evaluate my source separation/speech enhancement/ or denoising algorithm.
I thought it's equivalent to the bss toolbox, http://bass-db.gforge.inria.fr/bss_eval/, but with Python.
Am I wrong??

[faroit]

I thought it's equivalent to the bss toolbox, http://bass-db.gforge.inria.fr/bss_eval/, but with Python.
Am I wrong??

yes, it is a reimplementation of these tools. So do you get different result with mir_eval than with matlab? If not, I guess your problem is related to the usage of bss_eval for your application.

[cocosci]

I haven't tried the matlab version, but I did download it. For that Matlab function, it takes more parameters than the counterpart here.
From the handbook, it takes 3 or 4 parameters.
[SDR, SIR, SNR, SAR] = bss crit(s target, e interf, (e noise), e artif)

[cocosci]

In my case, say I have a clean file, "clean.wav"; noise, "noise.wav", (and the mixed.wav)
and the denoised file, "denoised.wav". They are with the same length. How to use your function to get SDR, SAR, SIR then?

[faroit]

From the handbook, it takes 3 or 4 parameters.
[SDR, SIR, SNR, SAR] = bss crit(s target, e interf, (e noise), e artif)

you need to compare with [SDR,SIR,SAR,perm]=bss_eval_sources(se,s)

[faroit]

In my case, say I have a clean file, "clean.wav"; noise, "noise.wav", (and the mixed.wav)
and the denoised file, "denoised.wav". They are with the same length. How to use your function to get SDR, SAR, SIR then?

as mentioned here: you need 4 sources. The mixture is not needed because bss_eval is designed for linear mixtures only.
In your example you only have 3 sources. To make it work you need to provided the estimated noise as well. If your algorithm doesn't provide it, you can use mix-denoised.

[cocosci]

Do we have the counterpart for bss_crit()?
What's the difference between these two? bss_eval_sources() and bss_crit()?

And why you say that mir-eval is not perfect for denoising application? BSS toolbox is designed to evaluate source separation, right? Thanks.

[faroit]

Do we have the counterpart for bss_crit()?

see https://github.com/craffel/mir_eval/blob/master/mir_eval/separation.py#L803

What's the difference between these two? bss_eval_sources() and bss_crit()?

Please read here

And why you say that mir-eval is not perfect for denoising application? BSS toolbox is designed to evaluate source separation, right?

sure you can use it, just make sure you provide all sources

[cocosci]

Thanks for your in-time and patient explanation!

[cocosci]

“If your algorithm doesn't provide it, you can use mix-denoised.” What do you mean by this?
Can I just use mixed_data - predicted_source to get predicted_noise?