Implement more transcription metrics

[justinsalamon]

In addition to precision, recall, and f-measure, MIREX also computes the Average Overlap Ratio (defined in this paper). Furthermore, it also computes all four metrics (p, r, f1, o) but with mapping the pitch values to chroma values (i.e. so that octave errors aren't penalized).

This means that MIREX computes 16 metrics in total:
(Precision, Recall, F-measure, AverageOR) x (with offsets, without offsets ) x (pitch, chroma) = 16

Currently we have:
(Precision, Recall, F-measure) x (with offsets, without offsets ) = 6

[craffel]

👍 Seems pretty straightforward to implement both, right?

[justinsalamon]

👍 Seems pretty straightforward to implement both, right?

Yup, I just have no cycles left for it right now. Maybe @rabitt would fancy taking a stab at this? ;) Otherwise I'm happy to revisit this some time in the future.

[rabitt]

Maybe @rabitt would fancy taking a stab at this? ;) Otherwise I'm happy to revisit this some time in the future.

@justinsalamon Sorry, no cycles for this right now :(

[justinsalamon]

Other than the missing MIREX metrics, there are 3 more sets of metrics I'm interested in - these are still p,r,f, but using only onset information, or only offset information, or only pitch information. This basically allows you to break down the results and gain further insight (e.g. some algorithms might be great at onset detection but terrible at pitch detection, etc.).

These metrics are not part of MIREX - would it still be in the scope of mir_eval? From an implementation perspective onset and offset only are very straight forward to implement - they are already computed for calculating the overall note hit matrix, so it would be relatively straight forward derivate work. For pitch only I need to give it some thought since matching pitch values in a completely time-agnostic fashion doesn't make much sense.

But assuming I figure out something sensible for each of the three, would this make sense for PR?

[craffel]

These metrics are not part of MIREX - would it still be in the scope of mir_eval?

Potentially, yes, but from a holistic view I'd say they are much lower priority than the rest of the MIREX metrics. If you want them though and are up for implementing them, feel free to make a PR.

For pitch only I need to give it some thought since matching pitch values in a completely time-agnostic fashion doesn't make much sense.

Yeah, I can't think of a way to do this which makes sense. The overlap ratio seems like a more sane metric for this, if you don't want to be separately paying attention to whether the onset or offset falls into a fixed threshold.

[justinsalamon]

Yeah, I can't think of a way to do this which makes sense. The overlap ratio seems like a more sane metric for this, if you don't want to be separately paying attention to whether the onset or offset falls into a fixed threshold.

That's actually a very good point - the overlap ratio with a very lean onset threshold would basically do exactly what I need for pitch.

OK, in which case my idea would be to implement match_note_onsets, match_note_offsets, and overlap_ratio. Or do you think it's better to implement the former two using optional arguments to the already existing match_notes method? It would probably be about 4 new lines of code.

[craffel]

Or do you think it's better to implement the former two using optional arguments to the already existing match_notes method?

You mean like onsets_only or offset_only? I think that could get a little hairy/confusing; I'd prefer if the relevant functionality was factored out (I think it already is?) and then just have separate metrics. Of course, you can also do whichever you think makes more sense and we can hash it out in the PR.

[justinsalamon]

I'm fine with implementing these as separate metrics, you're right in that it would keep things clearly delineated. I'll get onto the PR when I have a moment to breathe.

[craffel]

@justinsalamon if this was resolved with the closing of #189 , can you close? Thanks.

[justinsalamon]

Well, the chroma metrics are still not implemented, so if we want 100% coverage of the MIREX metrics I guess we should keep this open.

[craffel]

Ok, migrated to a separate issue #197.