Add bounds checks in SDL_qsort

[aikawayataro]

Description

Updating qsort implementation fixed only part of the non-transitive compare issue.
Using such a compare function should be considered a user code bug, but I believe it's better not to crash the whole program.
I set up a fuzzer and found a few unchecked memory reads and writes. With the proposed changes, qsort will not crash with invalid compare functions.
Fuzzer source code: fuzzer.zip

[madebr]

These checks are insufficient.
When adding extra tests to testqsort.c (madebr@1ea94b0), ci fails:
https://github.com/madebr/SDL/actions/runs/9597768806/job/26467538455

Feel free to use the commit in this pr to verify the fixes.

[aikawayataro]

@madebr You're accessing a zero-sized allocated block at line 61 when arraylen=0 😁 there's no memory corruptions from qsort.
As for your test of non-transitive compare, this usecase, which is considered invalid and should not be tested.

[DanielGibson]

As for your test of non-transitive compare, this usecase, which is considered invalid and should not be tested.

It's not valid, but testing it to make sure it at least doesn't crash makes sense (that's what these changes are about, after all).
But you can't expect the data to be sorted afterwards, so verifying any order doesn't make sense (even the "incorrect" order after running qsort with invalid comparator on a given array might not be deterministic in the long term, when the qsort implementation is updated again; and it might even be different depending on wordsize and whatever potentially platform-dependent special cases that implementation handles).

(No idea what exactly goes wrong in the qsort test, unless I missed something the log only mentions an incorrect exit code?)

[madebr]

@madebr You're accessing a zero-sized allocated block at line 61 when arraylen=0 😁 there's no memory corruptions from qsort. As for your test of non-transitive compare, this usecase, which is considered invalid and should not be tested.

Whoops! Good catch.
I think my changes still make sense, with a fix for the bug you noticed.

    if (arraylen > 0) {
        prev = nums[0];
    }

[aikawayataro]

It's not valid, but testing it to make sure it at least doesn't crash makes sense (that's what these changes are about, after all).
But you can't expect the data to be sorted afterwards, so verifying any order doesn't make sense (even the "incorrect" order after running qsort with invalid comparator on a given array might not be deterministic in the long term, when the qsort implementation is updated again; and it might even be different depending on wordsize and whatever potentially platform-dependent special cases that implementation handles).

That's what I meant to say, we shouldn't test the order, just the function.

(No idea what exactly goes wrong in the qsort test, unless I missed something the log only mentions an incorrect exit code?)

Because of a bug in the test itself, there's a segfault.

I think my changes still make sense, with a fix for the bug you noticed.

Test runs just fine with your fix (there was a note about a non-existent bug I found, apologies).
Also your arraylen with invalid compare function is too big because it takes crazy amounts of cpu compared to test with valid one. I guess I will add other arraylen values that will cover the whole qsort without very large values. We should also test aligned and unaligned branches. I'll add a test with these remarks in mind.

[aikawayataro]

I refactored the test, but honestly I think it does not look reasonable. The qsort used is in fact 3 qsorts for different cases. To test them all requires a lot of hackery (ignore failing build I can't figure out right pointer type for const array pointer).
What I believe is that we should not use qsort_aligned and qsort_words but use only plain unaligned version, which works in any case.