std.string.indexOf wrong result with bad unicode

[dlangBugzillaToGithub]

dlang-bugzilla (@CyberShadow) reported this on 2018-11-23T22:39:35Z

Transfered from https://issues.dlang.org/show_bug.cgi?id=19428

Description

//////////////////// test.d ///////////////////
import std.algorithm.comparison;
import std.range;
import std.string;

void main()
{
    assert(indexOf(
            only('\uFFFD', '\uFFFD', '\uFFFD'),
            "\x83\x84\x85",
            CaseSensitive.yes) == -1);
}
///////////////////////////////////////////////

Looks like it's replacing bad Unicode with replacement characters under the hood.

This becomes worse when something causes the same thing to happen to the haystack, as in this unit test:

https://github.com/dlang/phobos/blob/9bfc82130c0e4af4d1dc95bb261570c6e4f6f5d8/std/string.d#L887-L903

Note that this unittest is incorrectly annotated as nothrow/@nogc. We can't use the kind of decoding that substitutes errors with replacement characters, as that will introduce bugs like these.