3.2.0 TestStretcher fails

[dvzrv]

Hi @cannam! When trying to upgrade to 3.2.0 for Arch Linux I ran into the issue of the test "Stretcher" failing.

8/8 Stretcher         FAIL            3.70s   (exit status 201 or signal 73 SIGinvalid)
>>> MALLOC_PERTURB_=65 /build/rubberband/src/build/tests --run_test=TestStretcher --log_level=message

Output from build/meson-logs/testlog.txt:

==================================== 8/8 =====================================
test:         Stretcher
start time:   09:12:21
duration:     3.70s
result:       (exit status 201 or signal 73 SIGinvalid)
command:      MALLOC_PERTURB_=65 /build/rubberband/src/build/tests --run_test=TestStretcher --log_level=message
----------------------------------- stdout -----------------------------------
Running 69 test cases...
../rubberband-3.2.0/src/test/TestStretcher.cpp(422): error: in "TestStretcher/sinusoid_slow_higher_realtime_finer": check expected - rms < maxUnder has failed [0.70710678118654757 - 0.58194117738335283 >= 0.10000000000000001]
../rubberband-3.2.0/src/test/TestStretcher.cpp(422): error: in "TestStretcher/sinusoid_fast_higher_realtime_finer": check expected - rms < maxUnder has failed [0.70710678118654757 - 0.58239831167261791 >= 0.10000000000000001]
../rubberband-3.2.0/src/test/TestStretcher.cpp(401): error: in "TestStretcher/sinusoid_fast_higher_realtime_finer_hqpitch": check positiveCrossings >= expectedCrossings - slack has failed [25 < 29]
../rubberband-3.2.0/src/test/TestStretcher.cpp(401): error: in "TestStretcher/sinusoid_fast_higher_realtime_finer_hcpitch": check positiveCrossings >= expectedCrossings - slack has failed [25 < 29]
../rubberband-3.2.0/src/test/TestStretcher.cpp(400): error: in "TestStretcher/sinusoid_slow_lower_realtime_finer_hqpitch": check positiveCrossings <= expectedCrossings + slack has failed [82 > 81]
../rubberband-3.2.0/src/test/TestStretcher.cpp(422): error: in "TestStretcher/sinusoid_slow_lower_realtime_faster_hqpitch": check expected - rms < maxUnder has failed [0.70710678118654757 - 0.50525537719079694 >= 0.20000000000000001]
../rubberband-3.2.0/src/test/TestStretcher.cpp(951): error: in "TestStretcher/impulses_slow_lower_realtime_finer": check peak1 > int(ceil(4840 * timeRatio)) has failed [38415 <= 38720]
../rubberband-3.2.0/src/test/TestStretcher.cpp(951): error: in "TestStretcher/impulses_slow_lower_realtime_finer_hcpitch": check peak1 > int(ceil(4840 * timeRatio)) has failed [38415 <= 38720]
../rubberband-3.2.0/src/test/TestStretcher.cpp(1122): error: in "TestStretcher/final_fast_samepitch_realtime_finer": check outcount >= nOut has failed [384 < 2000]
../rubberband-3.2.0/src/test/TestStretcher.cpp(1122): error: in "TestStretcher/final_fast_samepitch_realtime_finer_after": check outcount >= nOut has failed [384 < 2000]
../rubberband-3.2.0/src/test/TestStretcher.cpp(1122): error: in "TestStretcher/final_fast_higher_realtime_finer": check outcount >= nOut has failed [1108 < 2000]
../rubberband-3.2.0/src/test/TestStretcher.cpp(1122): error: in "TestStretcher/final_fast_higher_realtime_finer_after": check outcount >= nOut has failed [1108 < 2000]
../rubberband-3.2.0/src/test/TestStretcher.cpp(1122): error: in "TestStretcher/final_fast_lower_realtime_finer": check outcount >= nOut has failed [0 < 2000]
../rubberband-3.2.0/src/test/TestStretcher.cpp(1122): error: in "TestStretcher/final_fast_lower_realtime_finer_after": check outcount >= nOut has failed [0 < 2000]
----------------------------------- stderr -----------------------------------

*** 14 failures are detected in the test module "RubberBand"

==============================================================================


Summary of Failures:

8/8 Stretcher         FAIL            3.70s   (exit status 201 or signal 73 SIGinvalid)

Ok:                 7
Expected Fail:      0
Fail:               1
Unexpected Pass:    0
Skipped:            0
Timeout:            0

I'm not entirely sure what to do with this, maybe you have an idea?

[cannam]

Thanks for the report...

A similar failure has been reported at https://todo.sr.ht/~breakfastquay/rubberband/27 but because that was in a 32-bit build, I was waiting until I had a moment to dig out a 32-bit environment to test it.

Which was stupid of me, because now that I have this report as well, and I know Arch no longer does 32-bit builds, I can immediately reproduce it - it happens always when using libsamplerate instead of the default built-in resampler. Duh. I wonder why this is not checked in CI - this configuration is built but I guess the tests are not being run within it.

I'll look into this during the week. You could build with the built-in resampler (and the tests should then pass) but perhaps it's better to wait for a fix. The fix might just be to make the test less stringent, but I don't know for sure yet. Sorry about this.

[dvzrv]

No worries! Thanks for the prompt reply! :)
I can wait a bit :)

[cannam]

There's a 3.2.1 out now with fixes. I hope...

[nekopsykose]

everything is fixed, except this same thing still fails on 32-bit x86:

――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stdout:
Running 69 test cases...
../src/test/TestStretcher.cpp(951): error: in "TestStretcher/impulses_fast_samepitch_realtime_finer": check peak0 < int(ceil(200 * timeRatio)) has failed [100 >= 100]
stderr:
*** 1 failure is detected in the test module "RubberBand"
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――***

with mostly the same configuration

Rubber Band Library 3.2.1
  Directories
    prefix                   : /usr
    bindir                   : bin
    libdir                   : lib
    datadir                  : share
  Configuration
    FFT                      : FFTW
    Resampler                : libsamplerate
    Build type               : Release
    Architecture             : x86
  Build targets
    Static library           : YES
                               Name: rubberband
    Dynamic library          : YES
                               Name: rubberband
    JNI library              : NO
    LADSPA plugin            : YES
                               Name: ladspa-rubberband
    LV2 plugin               : YES
                               Name: lv2-rubberband
    Vamp plugin              : YES
                               Name: vamp-rubberband
    Command-line utility (R2): YES
                               Name: rubberband
    Command-line utility (R3): YES
                               Name: rubberband-r3
    Unit tests               : YES
                               Name: tests
  User defined options
    auto_features            : auto
    bindir                   : /usr/bin
    buildtype                : release
    datadir                  : /usr/share
    includedir               : /usr/include
    infodir                  : /usr/share/info
    libdir                   : /usr/lib
    libexecdir               : /usr/libexec
    localedir                : /usr/share/locale
    localstatedir            : /var
    mandir                   : /usr/share/man
    prefix                   : /usr
    sbindir                  : /usr/sbin
    sharedstatedir           : /var/lib
    sysconfdir               : /etc
    wrap_mode                : nodownload
    b_lto                    : true
    b_pie                    : true
    b_staticpic              : true
    fft                      : fftw
    resampler                : libsamplerate

there's also a failure on the ppc64le architecture:

Running 69 test cases...
../src/test/TestStretcher.cpp(958): error: in "TestStretcher/impulses_slow_lower_realtime_finer": check peak2 > int(ceil(9770 * timeRatio)) has failed [77399 <= 78160]
../src/test/TestStretcher.cpp(958): error: in "TestStretcher/impulses_slow_lower_realtime_finer_hcpitch": check peak2 > int(ceil(9770 * timeRatio)) has failed [77399 <= 78160]
stderr:
*** 2 failures are detected in the test module "RubberBand"

[cannam]

That's odd, I did test x86... wonder what I've missed there.

I have no means of testing ppc. As you can see in many cases the precise values are quite sensitive to the configuration of various mathematical libraries - and that's normal and acceptable - so the aim in these particular tests is to check that the results are being produced at all and "are plausible" rather than to check anything with precision. It's "are plausible" where the problems lie!

[nekopsykose]

to be specific, it's against musl libc, on alpine linux. the full log is https://gitlab.alpinelinux.org/alpine/aports/-/jobs/1005027

[nekopsykose]

As you can see in many cases the precise values are quite sensitive to the configuration of various mathematical libraries

oh certainly- initially it was using builtin fft/resampling and the tests passed then. they only fail in release mode without builtins (for ppc64le specifically; x86 just always fails)...

[dvzrv]

Just wanted to note that it fixed it for my use-case (x86_64). Thanks @cannam!

[triallax]

Perhaps this is fixed by df596f4?

[nekopsykose]

for 32-bit x86 yes, for ppc64le it's the same