Simple quality measurement tool

[avm]

In order to make Audiveris better, we need some way to measure how good it already is. This is an attempt to actually measure the recognition quality with the simplest possible metric: number of notes and rests correctly placed in the recognition result.

This requires Python 3 with the music21 library installed. One sample test case is included (actually taken from issue #78), along with its "ideal" recognition result.

The proposed workflow is this:

• before starting work on a recognition issue, add a minimal example of it to test/cases/ (source.png or source.pdf and target.xml);
• measure recognition quality with test$ ./diffscore.py -c cases;
• fix the issue in the code;
• measure quality again, checking that it went up. If it went down, it means that we have either broken something else or not fixed what we intended.

The quality score calculation is quite simplistic right now, just checking pitch and duration for the notes and rests. Later on, within the same framework and using existing test cases, we can start taking into account more stuff (keys, time signatures, measure durations, dynamics, etc).

[hbitteur]

Thanks Alexander. We are more and more aware of the need for such regression management, but failed to provide one for lack of high-level measurement tool.
Your proposal sounds very simple but may well be very useful, especially within the context of continuous integration, which is still to be set up.
@maximumspatium is the right person to work with you on this topic, so I took the liberty to assign him this issue. He may not be immediately available, but please don't give up.

[maximumspatium]

@maximumspatium is the right person to work with you on this topic, so I took the liberty to assign him this issue.

Thanks :)

He may not be immediately available, but please don't give up.

That's true. As a hobbyist programmer, I'm doing all this work in my spare time.
@avm Please be patient - I'll look into your proposal in the next days and give you a feedback.

[guillaumerose]

This regression test looks good to me. Since this project is in Java, it will be more natural to JUnit. Happy to do it if you are interested.

[hbitteur]

Of course this would be interesting for the project.
But the initial message was posted several years ago.
See what you can do to re-activate this.

[PeterGreth]

Hello everybody!

Thanks @avm for this nice contribution! I think tracking recognition quality over time/commits would indeed be very valuable 😃

I had a few spare hours and felt like I could contribute to this awesome project by providing the equivalent implementation in Java/Junit. @guillaumerose I hope you hadn't started yet - at least I could not find a branch or commits on your Audiveris fork.

For clarity, I created a separate PR: #563

Cheers
Peter