String-like extension methods to ReadOnlySpan<char> Epic #22434

stephentoub · 2017-06-23T00:29:42Z

As we look to support more efficient parsing/formatting/usage of ReadOnlySpan<char> as slices of System.Strings, we're planning to add a variety of APIs across corefx that operate with ReadOnlySpan<char> (https://github.com/dotnet/corefx/issues/21281). But for such APIs to be truly useful, and for ReadOnlySpan<char> to be generally helpful as a string-like type, we need a set of extension methods on ReadOnlySpan<char> that mirror the corresponding operations on string, e.g. Equals with various kinds of string comparisons, Trim, Replace, etc. We need to define, implement, test, and ship a core set of these (more can of course be added in the future), e.g.

Edit by @ahsonkhan - Updated APIs:

// All these do ordinal comparisons, and hence do not rely on StringComparison, and can live in System.Memory.dll
public static class MemoryExtensions
{
    // If we decide to add overloads to Trim in the future that are non-ordinal and take StringComparison 
    // (similar to https://github.com/dotnet/corefx/issues/1244), they will be .NET Core specific.
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);

    public static bool IsWhiteSpace(this ReadOnlySpan<char> span);

    public static bool Remove(this ReadOnlySpan<char> source, int startIndex, int count, Span<char> destination);

    // Does exactly what string does today, i.e. ordinal, case-sensitive, culture-insensitive comparison
    public static bool Replace(this ReadOnlySpan<char> source, ReadOnlySpan<char> oldValue, ReadOnlySpan<char> newValue, Span<char> destination, out int bytesWritten);
    
    // To me, these are complementary to the Trim APIs and hence we should add them.
    public static bool PadLeft(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination);
    public static bool PadLeft(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination, char paddingChar);
    public static bool PadRight(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination);
    public static bool PadRight(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination, char paddingChar);
}

// Live in CoreLib and only available on .NET Core, exposed from System.Memory.dll
// Atm, this class in corelib is called 'Span' and contains the .NET Core specific implementation of the extension methods
public static class MemoryExtensions
{
    public static bool Equals(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static int Compare(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool Contains(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool StartsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool EndsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool Replace(this ReadOnlySpan<char> source, ReadOnlySpan<char> oldValue, ReadOnlySpan<char> newValue, StringComparison comparison, Span<char> destination, out int bytesWritten);

    public static bool ToUpper(this ReadOnlySpan<char> source, Span<char> destination);
    public static bool ToUpper(this ReadOnlySpan<char> source, Span<char> destination, CultureInfo culture);
    public static bool ToUpperInvariant(this ReadOnlySpan<char> source, Span<char> destination);
    public static bool ToLower(this ReadOnlySpan<char> source, Span<char> destination);
    public static bool ToLower(this ReadOnlySpan<char> source, Span<char> destination, CultureInfo culture);
    public static bool ToLowerInvariant(this ReadOnlySpan<char> source, Span<char> destination);
}

Original Proposal

public static class SpanExtensions
{
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span);
    public static bool Equals(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static int Compare(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool StartsWith(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static ReadOnlySpan<char> Remove(this ReadOnlySpan<char> source, int startIndex, int count); // this will need to allocate if any chars are removed from the middle
    ...
}

The text was updated successfully, but these errors were encountered:

jnm2 · 2017-09-15T14:40:09Z

I'm particularly interested in IndexOf(char) and LastIndexOf(char) so far.

safakgur · 2017-10-27T09:01:26Z

Some of these extensions may also be useful for types other than char:

public static bool Equals<T>(this ReadOnlySpan<T> source, ReadOnlySpan<T> value)
    where T : struct, IEquatable<T>; // Non-allocating Enumerable.SequenceEquals

public static bool StartsWith(this ReadOnlySpan<T> source, ReadOnlySpan<T> value)
    where T : struct, IEquatable<T>;

// and overloads with IEqualityComparer<T> instead of the type constraint.

ahsonkhan · 2017-12-12T01:46:53Z

I'm particularly interested in IndexOf(char) and LastIndexOf(char) so far.

We have IndexOf already. Does that your scenario? https://github.com/dotnet/corefx/blob/master/src/System.Memory/ref/System.Memory.cs#L93

Some of these extensions may also be useful for types other than char:
Equals
StartsWith

We have StartsWith already. Also, is the semantics of Equals the same as SequenceEqual? If so, that exists as well.
https://github.com/dotnet/corefx/blob/master/src/System.Memory/ref/System.Memory.cs#L137

ahsonkhan · 2017-12-12T01:55:23Z

What about adding these as well?

public static bool EndsWith(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
public static bool IsWhiteSpace(this ReadOnlySpan<char> value);
public static ReadOnlySpan<char> Replace(this Span<char> oldValue, ReadOnlySpan<char> newValue); // this will need to allocate if the length of the chars being removed is not equal to the length of the chars being added
public static ReadOnlySpan<char> ToUpper(this ReadOnlySpan<char> span);
public static ReadOnlySpan<char> ToLower(this ReadOnlySpan<char> span);
public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span);
public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span);

ahsonkhan · 2017-12-12T19:10:38Z

Updated proposal here: https://github.com/dotnet/corefx/issues/21395#issuecomment-357410425

Proposed API Additions

public static class MemoryExtensions
{
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span);
    public static bool Equals(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static int Compare(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
   
    // this will need to allocate if any chars are removed from the middle
    public static ReadOnlySpan<char> Remove(this ReadOnlySpan<char> source, int startIndex, int count); 
    
    public static bool StartsWith(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool EndsWith(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool IsWhiteSpace(this ReadOnlySpan<char> value);
    
    // this will need to allocate if the length of the chars being removed is not equal to the length of the chars being added
    public static ReadOnlySpan<char> Replace(this Span<char> oldValue, ReadOnlySpan<char> newValue); 
    
    public static ReadOnlySpan<char> ToUpper(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> ToLower(this ReadOnlySpan<char> span);
}

jkotas · 2017-12-12T19:23:26Z

Are these APIs going to be present in both the OOB and inbox version of System.Memory? What is the OOB implementation going to look like?

The ToUpper and ToLower APIs are also allocating. Should we rather change the signatures to be non-allocating?

ahsonkhan · 2017-12-12T22:52:23Z

Are these APIs going to be present in both the OOB and inbox version of System.Memory? What is the OOB implementation going to look like?

I was thinking, similar to APIs like StartsWith, SequenceEqual, etc. (https://github.com/dotnet/corefx/blob/master/src/System.Memory/src/System/MemoryExtensions.cs#L281), that there will only be an OOB version. What would be the value of having inbox versions as well?

The ToUpper and ToLower APIs are also allocating. Should we rather change the signatures to be non-allocating?

Yes. I will work on the API signatures (adding destination span that is passed in as well), to try to get them to be non-allocating.

jkotas · 2017-12-12T23:18:19Z

The problem with ToLower, ToUpper and other similar globalization APIs is that it is impossible to implement them efficiently in the OOB version. The implementation would always have to allocate the strings from the Span first. It make them kind of pointless to have in the OOB System.Memory. Have you though about having these APIs in System.Runtime or similar contract instead?

benaadams · 2017-12-12T23:39:52Z

Wouldn't work for string (as immuatable); but for regular Span<char>?

public static void ToUpper(this Span<char> span);
public static void ToLower(this Span<char> span);

Tragetaschen · 2017-12-19T11:44:40Z

@benaadams
string.ToUpper depends on the current culture and there's no one-to-one mapping between characters. For example the one-character, lowercase ß in German becomes the two-character SS (groß => GROSS)

theolivenbaum · 2017-12-19T20:54:49Z

any thoughts on supporting running Regex on ReadOnlySpan?

KrzysztofCwalina · 2017-12-20T04:46:23Z

I think APIs that don't specify otherwise (including the ToUpper/Lower) should be culture invariant. If We want to provide culture aware APIs, they should take an explicit CultureInfo argument.

svick · 2017-12-20T15:03:31Z

@rafael-aero There's a separate issue for that: https://github.com/dotnet/corefx/issues/24145.

Tornhoof · 2017-12-21T01:39:18Z

FWIW, @Tragetaschen

For example the one-character, lowercase ß in German becomes the two-character SS (groß => GROSS)

This is not correct anymore, there is a ẞ in Unicode for years now, U+1E9E. It's part of german Orthography since July 2017. https://en.wikipedia.org/wiki/Capital_ẞ
I Imagine there are still other examples though.

SomeAnonDev · 2017-12-27T14:17:29Z

It would be great if we could also switch on Spans of char as if they were strings.

jnm2 · 2017-12-27T16:22:07Z

@SomeAnon42 Special-casing ReadOnlySpan<char> the way string is special-cased would be a language-level change. (https://github.com/dotnet/csharplang)

bbowyersmyth · 2017-12-27T21:49:51Z

@Tragetaschen @benaadams CoreCLR already operates under the assumption that the case change produces the same length string. https://github.com/dotnet/coreclr/blob/master/src/mscorlib/shared/System/Globalization/TextInfo.Windows.cs#L26-L76
This may be a bug. If not, it would be consistent for span to do the same thing.

ahsonkhan · 2018-01-12T20:51:39Z

Updated proposal here: https://github.com/dotnet/corefx/issues/21395#issuecomment-357410425

Added:

Contains
Non-allocating alternatives
Additional overloads

Updated Proposed API Additions

public static class MemoryExtensions
{
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    
    public static bool Equals(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool Contains(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static int Compare(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
   
    // this will need to allocate if any chars are removed from the middle
    public static ReadOnlySpan<char> Remove(this ReadOnlySpan<char> source, int startIndex, int count);
    public static ReadOnlySpan<char> Remove(this ReadOnlySpan<char> source, int startIndex, int count);
    // non-allocating alternative:
    public static bool Remove(this ReadOnlySpan<char> source, int startIndex, Span<char> result);
    public static bool Remove(this ReadOnlySpan<char> source, int startIndex, int count, Span<char> result); 
    
    public static bool StartsWith(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool EndsWith(this ReadOnlySpan<char> source, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool IsWhiteSpace(this ReadOnlySpan<char> value);
    
    // this will need to allocate if the length of the chars being removed is not equal to the length of the chars being added
    public static ReadOnlySpan<char> Replace(this Span<char> oldValue, ReadOnlySpan<char> newValue);
    // non-allocating alternative:
    public static bool Replace(this Span<char> oldValue, ReadOnlySpan<char> newValue, Span<char> result);
    
    public static ReadOnlySpan<char> ToUpper(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> ToUpperInvariant(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> ToLower(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> ToLowerInvariant(this ReadOnlySpan<char> span);

    // these will allocate
    public static ReadOnlySpan<char> PadLeft(this ReadOnlySpan<char> span, int totalWidth);
    public static ReadOnlySpan<char> PadLeft(this ReadOnlySpan<char> span, int paddingChar);
    public static ReadOnlySpan<char> PadRight(this ReadOnlySpan<char> span, int totalWidth);
    public static ReadOnlySpan<char> PadRight(this ReadOnlySpan<char> span, int paddingChar);
    // non-allocating alternative
    public static bool PadLeft(this ReadOnlySpan<char> span, int totalWidth, Span<char> result);
    public static bool PadLeft(this ReadOnlySpan<char> span, int paddingChar, Span<char> result);
    public static bool PadRight(this ReadOnlySpan<char> span, int totalWidth, Span<char> result);
    public static bool PadRight(this ReadOnlySpan<char> span, int paddingChar, Span<char> result);
}

jkotas · 2018-01-12T21:30:34Z

Are the signatures of Replace methods right? They seem to be missing an argument. Also, why is the allocating variant of Replace returning ReadOnlySpan? Shouldn't it rather return writeable span?

terrajobst · 2018-01-12T21:49:43Z

Video

We don't think this is ready yet. @ahsonkhan, please redesign the APIs to avoid allocations by returning new spans. Instead, they should follow our TryXxx pattern. But we have to keep in mind that many times folks will start with ReadOnlySpan<char> from a string where in-place isn't possible. It would help to have some sample code that shows how these APIs are used.

ahsonkhan · 2018-01-13T05:03:15Z

Update:

Removed the allocating versions of the API
Fixed Replace
Fixed argument names and return values of the non-allocating versions of the APIs

Most of the APIs that are designed to avoid allocations do not need to return bytesWritten since that will be known to the user before the call. For example, we know that Remove will write exactly source.Length - count chars to the destination span. The exception to this is Replace, which is why it needs bytesWritten.

Proposed API

public static class MemoryExtensions
{
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span) { throw null; }
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, char trimChar) { throw null; }
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars) { throw null; }
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span) { throw null; }
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, char trimChar) { throw null; }
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars) { throw null; }
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span) { throw null; }
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, char trimChar) { throw null; }
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars) { throw null; }
    
    public static bool Equals(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison) { throw null; }
    public static int Compare(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison) { throw null; }
    public static bool Contains(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison) { throw null; }

    public static bool StartsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison) { throw null; }
    public static bool EndsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison) { throw null; }
    public static bool IsWhiteSpace(this ReadOnlySpan<char> span) { throw null; }


    // APIs designed to avoid allocations:

    // do we need bytesWritten? It should be source.Length - count on success
    public static bool Remove(this ReadOnlySpan<char> source, int startIndex, int count, Span<char> destination) { throw null; }
    
    public static bool Replace(this ReadOnlySpan<char> source, ReadOnlySpan<char> oldValue, ReadOnlySpan<char> newValue, Span<char> destination, out int bytesWritten) { throw null; }
    
    // do we need bytesWritten? It should be source.Length on success
    public static bool ToUpper(this ReadOnlySpan<char> source, Span<char> destination) { throw null; }
    public static bool ToUpperInvariant(this ReadOnlySpan<char> source, Span<char> destination) { throw null; }
    public static bool ToLower(this ReadOnlySpan<char> source, Span<char> destination) { throw null; }
    public static bool ToLowerInvariant(this ReadOnlySpan<char> source, Span<char> destination) { throw null; }

    // do we need bytesWritten? It should be totalWidth on success or source.Length if totalWidth < source.Length
    public static bool PadLeft(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination) { throw null; } 
    public static bool PadLeft(this ReadOnlySpan<char> source, int totalWidth, char paddingChar, Span<char> destination) { throw null; }
    public static bool PadRight(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination) { throw null; }
    public static bool PadRight(this ReadOnlySpan<char> source, int totalWidth, char paddingChar, Span<char> destination) { throw null; }
}

Sample Usage

Approach 1 to use PadLeft:

ReadOnlySpan<char> span; // input
int totalWidth = 50; // known value, assume >= span.Length
Span<char> destination = new char[totalWidth]; //if set to less than totalWidth, PadLeft will return false
if (span.PadLeft(totalWidth, destination))
{
    // destination has some pad characters (if there is extra space) followed by the entire input
    // input is at the end of the destination buffer
}
else
{
    // resize destination to be >= totalWidth
    var result = span.PadLeft(totalWidth, destination);
    Debug.Assert(result);
}
// only if destination.Length > totalWidth, it will be necessary to slice to get the precise segment.
destination = destination.Slice(0, totalWidth);

Approach 2 to use PadLeft:

ReadOnlySpan<char> span; // input
int totalWidth = x; // unknown value, x >= 0, could be less than span.Length
Span<char> destination = new char[y] // y could be less than x

if (source.Length > totalWidth) return; // user can choose to do nothing, or call PadLeft which will just do a copy

if (destination.Length < totalWidth) destination = new char[totalWidth];
span.PadLeft(totalWidth, destination); // no need to slice, the destination is of the right size.

Sample usage for Replace:

string testStr = "abcdefghibclmbcp";
ReadOnlySpan<char> source = testStr.AsReadOnlySpan();
Span<char> destination = new char[source.Length];

// oldValue.Length >= newValue.Length
source.Replace("a".AsReadOnlySpan(), "z".AsReadOnlySpan(), destination, out int bytesWritten)
// not necessary here
destination = destination.Slice(0, bytesWritten);


// oldValue.Length < newValue.Length
while (!source.Replace("z".AsReadOnlySpan(), "ab".AsReadOnlySpan(), destination, out int bytesWritten))
{
    // enlarge destination
    // maximum size required: string.Length + (string.Length/oldValue.Length * (newValue.Length - oldValue.Length))
    // maximum size required: 16 + (16/1 * (2 - 1)) = 16 + 16 = 32
    // growing the destination to 32 will gaurantee success in this case
}
destination = destination.Slice(0, bytesWritten);

Open Questions:

Should these methods be generic? If not, should we provide byte specific overloads?
When should we provide the bytesWritten overload? The APIs are currently designed to only provide it if it cannot be calculated, or already known, by the caller (in constant time) before the method is called.

terrajobst · 2018-01-16T20:05:20Z

Video

// Approved, but needs to go somewhere else due to globalization

public static class MemoryExtensions
{
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    
    public static bool Equals(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static int Compare(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool Contains(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);

    public static bool StartsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool EndsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool IsWhiteSpace(this ReadOnlySpan<char> span);
}

// Needs more work

public static class MemoryExtensions
{
    // APIs designed to avoid allocations:

    public static bool Remove(this ReadOnlySpan<char> source, int startIndex, int count, Span<char> destination);
    
    public static bool Replace(this ReadOnlySpan<char> source, ReadOnlySpan<char> oldValue, ReadOnlySpan<char> newValue, Span<char> destination, out int bytesWritten);
    
    // do we need bytesWritten? It should be source.Length on success
    public static bool ToUpper(this ReadOnlySpan<char> source, Span<char> destination);
    public static bool ToUpperInvariant(this ReadOnlySpan<char> source, Span<char> destination);
    public static bool ToLower(this ReadOnlySpan<char> source, Span<char> destination);
    public static bool ToLowerInvariant(this ReadOnlySpan<char> source, Span<char> destination);
}

// We probably don't want these    

public static class MemoryExtensions
{
    public static bool PadLeft(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination); 
    public static bool PadLeft(this ReadOnlySpan<char> source, int totalWidth, char paddingChar, Span<char> destination);
    public static bool PadRight(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination);
    public static bool PadRight(this ReadOnlySpan<char> source, int totalWidth, char paddingChar, Span<char> destination);
}

benaadams · 2018-01-16T20:08:37Z

We probably don't want these
PadLeft

Time for a NuGet span left-pad? Will become the most popular library and the world will come to depend on it - mwhahahaha!

benaadams · 2018-01-16T20:12:44Z

needs to go somewhere else due to globalization

What about the Ordinal versions?

public static ReadOnlySpan<T> Trim(this ReadOnlySpan<T> span, T trimElement);
public static ReadOnlySpan<T> Trim(this ReadOnlySpan<T> span, ReadOnlySpan<T> trimElements);
// etc

ahsonkhan · 2018-01-16T22:27:32Z

// Needs more work

I understand that we need to consider StringComparison overload for Replace, but what additional work/design decisions are there for the Remove API? It doesn't have any concerns regarding comparison/globalization/etc.

ahsonkhan · 2018-01-20T03:25:37Z

// All these do ordinal comparisons, and hence do not rely on StringComparison, and can live in System.Memory.dll
public static class MemoryExtensions
{
    // If we decide to add overloads to Trim in the future that are non-ordinal and take StringComparison 
    // (similar to https://github.com/dotnet/corefx/issues/1244), they will be .NET Core specific.
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);

    public static bool IsWhiteSpace(this ReadOnlySpan<char> span);

    public static bool Remove(this ReadOnlySpan<char> source, int startIndex, int count, Span<char> destination);

    // Does exactly what string does today, i.e. ordinal, case-sensitive, culture-insensitive comparison
    public static bool Replace(this ReadOnlySpan<char> source, ReadOnlySpan<char> oldValue, ReadOnlySpan<char> newValue, Span<char> destination, out int bytesWritten);
    
    // To me, these are complementary to the Trim APIs and hence we should add them.
    public static bool PadLeft(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination);
    public static bool PadLeft(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination, char paddingChar);
    public static bool PadRight(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination);
    public static bool PadRight(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination, char paddingChar);
}

// Live in CoreLib and only available on .NET Core, exposed from System.Memory.dll
// Atm, this class in corelib is called 'Span' and contains the .NET Core specific implementation of the extension methods
public static class MemoryExtensions
{
    public static bool Equals(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static int Compare(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool Contains(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool StartsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool EndsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool Replace(this ReadOnlySpan<char> source, ReadOnlySpan<char> oldValue, ReadOnlySpan<char> newValue, StringComparison comparison, Span<char> destination, out int bytesWritten);

    public static bool ToUpper(this ReadOnlySpan<char> source, Span<char> destination, CultureInfo culture = CultureInfo.CurrentCulture);
    public static bool ToUpperInvariant(this ReadOnlySpan<char> source, Span<char> destination);
    public static bool ToLower(this ReadOnlySpan<char> source, Span<char> destination, CultureInfo culture = CultureInfo.CurrentCulture);
    public static bool ToLowerInvariant(this ReadOnlySpan<char> source, Span<char> destination);
}

Joe4evr · 2018-01-22T07:16:59Z

Can I throw in another suggestion? I'd really like to see some ability to split a ReadOnlySpan<char>. Obviously, you can't return a collection of Spans directly, but isn't that what Memory<T> is for?

// equivalent to the overloads of 'String.Split()'
public static IReadOnlyList<ReadOnlyMemory<char>> Split(this ReadOnlySpan<char> span, char[] seperator);
public static IReadOnlyList<ReadOnlyMemory<char>> Split(this ReadOnlySpan<char> span, char[] seperator, int count);
public static IReadOnlyList<ReadOnlyMemory<char>> Split(this ReadOnlySpan<char> span, char[] seperator, int count, StringSplitOptions options);
public static IReadOnlyList<ReadOnlyMemory<char>> Split(this ReadOnlySpan<char> span, char[] seperator, StringSplitOptions options);
public static IReadOnlyList<ReadOnlyMemory<char>> Split(this ReadOnlySpan<char> span, string[] seperator, int count, StringSplitOptions options);
public static IReadOnlyList<ReadOnlyMemory<char>> Split(this ReadOnlySpan<char> span, string[] seperator, StringSplitOptions options);

The reason for choosing IReadOnlyList<T> is to make the resulting collection indexable, just like string[]. It would be nice if the implementation is ImmutableArray<T>, but I'm not sure if that's a concern for distribution and such.

ahsonkhan · 2018-01-22T09:36:33Z

@Joe4evr, out of curiosity, do you have a scenario atm where these APIs would be useful? If so, can you please show the code sample?

I would replace the char[] overloads with ReadOnlySpan<char>. However, I am not sure about adding the split APIs, in general, given they have to allocate. Is there a way to avoid allocating? Also, given these are string-like APIs for span, it is strange to have an overload that takes a string[]. Maybe all these would fit better on ReadOnlyMemory instead, especially given the return type.

Joe4evr · 2018-01-22T16:35:42Z

My scenario is taking a relatively big string of user input and then parsing that to populate a more complex object. So rather than take the whole string at once, I'd like to parse it in pieces at a time. It'd be pretty nice if this can be facilitated by the Span/Memory<T> APIs so that this code won't have to allocate an extra 30-40 tiny strings whenever it runs.

Admittedly, I only started on this particular case earlier today, mostly to experiment and find out how much I could get out of the Span APIs at this time.

Maybe it was a bit naive of me to expect a collection like I did, but I'd at least like to see some API to deal with this scenario a little easier, because I'll probably not be the only one looking to split a span up into smaller chunks like this.

stephentoub · 2018-01-22T16:41:31Z

Splitting support would be good, but I don't think it would look like the proposed methods; as @ahsonkhan points out, that would result in a lot of allocation (including needing to copy the whole input string to the heap, since you can't store the span into a returned interface implementation).

I would instead expect a design more like an iterator implemented as a ref struct, e.g.

public ref struct CharSpanSplitter
{
    public CharSpanSplitter(ReadOnlySpan<char> value, char separator, StringSplitOptions options);
    public bool TryMoveNext(out ReadOnlySpan<char> result);
}

ahsonkhan · 2018-01-22T19:11:04Z

Can I throw in another suggestion? I'd really like to see some ability to split a ReadOnlySpan<char>.

I would instead expect a design more like an iterator implemented as a ref struct

@Joe4evr, would it be fine for me to split the conversation related to the split API into a new issue, especially since it would require additional design?

Edit: Created https://github.com/dotnet/corefx/issues/26528

jkotas · 2018-01-23T01:52:16Z

bool Equals(this ReadOnlySpan span, ReadOnlySpan value, StringComparison comparison);

StringComparison is just a conveniece wrapper over System.Globalization.CompareInfo. Do you also need new overloads that take Span on System.Globalization.CompareInfo to make this work? Same for all other stuff that deals with globalization.

public static bool ToUpper(this ReadOnlySpan source, Span destination, CultureInfo culture = CultureInfo.CurrentCulture);

CultureInfo.CurrentCulture cannot use a property as the default value. Default value has to be a constant.

Joe4evr · 2018-01-23T01:56:58Z

@stephentoub Fair enough. Glad I at least got the ball rolling on the matter.

ahsonkhan · 2018-01-23T02:02:56Z

CultureInfo.CurrentCulture cannot use a property as the default value. Default value has to be a constant.

Added the proper overload without optional arguments (updated the first post).

StringComparison is just a conveniece wrapper over System.Globalization.CompareInfo.

How so? StringComparison is just an enum with culture specific options and intersects with some of CompareOptions.

Do you also need new overloads that take Span on System.Globalization.CompareInfo to make this work?

~~I don't know what you mean by this. Are you asking if we would need additional overloads that take CompareInfo instead of StringComparison as arguments?~~

Edit: I think I get what you mean. We may need to expose the compare APIs on CompareInfo that take Span, publicly. This is used by Equals/Compare (and can probably remain internal):
https://github.com/dotnet/coreclr/blob/master/src/mscorlib/shared/System/Globalization/CompareInfo.cs#L393

We would need to add additional span-based overloads for IsPrefix/IsSuffix/IndexOf.

jkotas · 2018-01-23T02:23:29Z

This is used by Equals/Compare (and can probably remain internal

It would be odd for the overloads to remain internal. We have made the Span overloads public in all other similar cases - Stream, BinaryReader/Writer, ... .

khellang · 2018-01-23T14:11:13Z

I would instead expect a design more like an iterator implemented as a ref struct

@stephentoub @Joe4evr Why stop with a "string splitter"? I'd love to see something like Google Guava's CharMatcher, which could be much more useful (might be a case for a separate proposal, though.)

terrajobst · 2018-01-23T19:40:41Z

Video

We can expose all these APIs from MemoryExtensions, even the ones that need globalization APIs as this just means we can't expose them (yet).
We'll not expose APIs that make StringComparison or CultureInfo implicit. We might get feedback, but we think forcing the developer to make a decision is less prone to errors than selecting a default.
We've removed Replace as we don't think the API interaction / pattern works out. It need more thought/design.

public static class MemoryExtensions
{
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> Trim(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimStart(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, char trimChar);
    public static ReadOnlySpan<char> TrimEnd(this ReadOnlySpan<char> span, ReadOnlySpan<char> trimChars);

    public static bool IsWhiteSpace(this ReadOnlySpan<char> span);

    public static void Remove(this ReadOnlySpan<char> source, int startIndex, int count, Span<char> destination);    
    public static void PadLeft(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination);
    public static void PadLeft(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination, char paddingChar);
    public static void PadRight(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination);
    public static void PadRight(this ReadOnlySpan<char> source, int totalWidth, Span<char> destination, char paddingChar);

    // Those need access to globalization APIs. We'll also expose them from
    // the .NET Framework OOB (slow span). They will try to extract the string
    // from the underlying span (because slow span stores it) -- or -- allocate
    // a new string. This avoids bifurcating the API surface.

    public static bool Contains(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool EndsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool Equals(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static bool StartsWith(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static int CompareTo(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);
    public static int IndexOf(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);

    public static void ToLower(this ReadOnlySpan<char> source, Span<char> destination, CultureInfo culture);
    public static void ToLowerInvariant(this ReadOnlySpan<char> source, Span<char> destination);
    public static void ToUpper(this ReadOnlySpan<char> source, Span<char> destination, CultureInfo culture);
    public static void ToUpperInvariant(this ReadOnlySpan<char> source, Span<char> destination);
}

tarekgh · 2018-01-24T00:36:41Z

one last note regarding

public static int IndexOf(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison);

I want to have this API out the matched length of the searched characters as it is not always equal to the length of the value span.

Example: imagine the source span have the characters A ̀ and we are searching for Á with culture aware operation. IndexOf will return the index of the match and will return the matched length is 2 and not just 1.

so I suggest the API will be

public static int IndexOf(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, StringComparison comparison, out int matchedLength);

The benefit having the outmatched length is:

It will help in the scenario of searching more than one time inside the span. when hit first match, you can know where you can start the next search.
As we have removed the Replace API from the proposal, we are giving people the way to implement their Replace APIs

ahsonkhan · 2018-01-24T06:26:31Z

Given that the proposed API can result in correctness issues, I would agree that we need the out int matchedLength parameter for the non-ordinal IndexOf.

Change:

public static int IndexOf(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, 
                                       StringComparison comparison);

To:

public static int IndexOf(this ReadOnlySpan<char> span, ReadOnlySpan<char> value, 
                                       StringComparison comparison, out int matchedLength);

KrzysztofCwalina · 2018-01-24T17:06:08Z

I want to have this API out the matched length of the searched characters as it is not always equal to the length of the value span.

Then maybe we want a Ordinal overload without this out parameter?

Example: imagine the source span have the characters A ̀ and we are searching for Á with culture aware operation. IndexOf will return the index of the match and will return the matched length is 2 and not just 1.

Could we add the example to docs (remarks) of the method?

tarekgh · 2018-01-24T17:53:37Z

Then maybe we want an Ordinal overload without this out parameter?

I think matchedLength would be useful even in the ordinal case too. if you write some code to do a repeated search inside the span would just work for any case (either ordinal or non-ordinal). if we have the ordinal overload, the user has to use Span.Length to perform the subsequent operation.
I don't have a strong feeling though about that :-)

here is some code sample for IndexOf

            // string has A character followed by the COMBINING GRAVE ACCENT ̀  which is equivalent to À (0x00C0)
            string source = "A\u0300 Some Other text A\u0300"; 
            string value = "\u00C0"; // À

            ReadOnlySpan<char> sourceSpan = source.AsReadOnlySpan();
            ReadOnlySpan<char> valueSpan = value.AsReadOnlySpan();

            while (true)
            {
                int index = sourceSpan.IndexOf(valueSpan, StringComparison.InvariantCulture, out int matachedLength);
                if (index < 0 || index + matachedLength >= sourceSpan.Length)
                {
                    break; // done searching
                }
                sourceSpan = sourceSpan.Slice(index + matachedLength); // Slice for next search
            }

ahsonkhan · 2018-01-25T19:47:38Z

Then maybe we want a Ordinal overload without this out parameter?

We already have that, essentially, with IndexOf<T>
https://github.com/dotnet/corefx/blob/master/src/System.Memory/ref/System.Memory.cs#L42

public static int IndexOf<T>(this ReadOnlySpan<T> span, ReadOnlySpan<T> value) where T : IEquatable<T>

KrzysztofCwalina · 2018-01-25T21:48:04Z

We already have that, essentially, with IndexOf

Ah, makes sense.

justinvp · 2018-02-06T06:45:16Z

The name of the StringComparison parameters should be comparisonType, not comparison, to be consistent with the existing naming used everywhere else.
Some extension methods use source while others use span as the name of the this parameter. It'd be nice to be consistent with the naming.

ahsonkhan · 2018-02-06T20:31:05Z

Some extension methods use source while others use span as the name of the this parameter. It'd be nice to be consistent with the naming.

The APIs where this is source have another span parameter as destination. The other ones don't. That is the reason for the inconsistency. If we feel strongly about the parameter renaming, I can do it in all methods. We should probably address this together with other inconsistencies like https://github.com/dotnet/corefx/issues/26894 (and dotnet/corefx#26694 (comment)).

ahsonkhan · 2018-02-15T02:11:11Z

Based on the discussion here, we want to make an adjustment to the following APIs to return an int (number of characters written to the destination). Also, instead of throwing when the destination is too small, we would return -1 isntead.

Changing return type from void to int.

public static int ToLower(this ReadOnlySpan<char> source, Span<char> destination, CultureInfo culture);
public static int ToLowerInvariant(this ReadOnlySpan<char> source, Span<char> destination);
public static int ToUpper(this ReadOnlySpan<char> source, Span<char> destination, CultureInfo culture);
public static int ToUpperInvariant(this ReadOnlySpan<char> source, Span<char> destination);

ahsonkhan · 2018-02-28T06:53:31Z

Done

String-like Span extension methods - Trim / IsWhiteSpace corefx#26560
Add ROSpan StartsWith and EndsWith string-like APIs with StringComparison corefx#26880 / Add ReadOnlySpan string-like StartsWith and EndsWith API with globalization support coreclr#16223 / Fix impl of ReadOnlySpan StartsWith/EndsWith for Unix. coreclr#16418
Add ROSpan ToLower and ToUpper string-like APIs with CultureInfo corefx#27193 / Add ReadOnlySpan string-like ToLower/ToUpper API with globalization support coreclr#16379 / Fix impl of ReadOnlySpan ToLower/ToUpper for Unix. coreclr#16496
Add ROSpan Equals/CompareTo/IndexOf/Contains string-like APIs with StringComparison corefx#27319 / Add ReadOnlySpan string-like Equals/CompareTo/IndexOf/Contains API with globalization support coreclr#16467

KrzysztofCwalina · 2018-02-28T16:48:39Z

Thanks! Nice addition that nicely rounds up what can be done with Spans.

stephentoub assigned KrzysztofCwalina Jun 23, 2017

joshfree unassigned KrzysztofCwalina Dec 6, 2017

danmoseley changed the title ~~Need core set of ReadOnlySpan<char> string-like extension methods~~ String-like extension methods to ReadOnlySpan<char> Epic Jan 3, 2018

ahsonkhan self-assigned this Jan 17, 2018

ahsonkhan closed this as completed Feb 28, 2018

msftgits transferred this issue from dotnet/corefx Jan 31, 2020

msftgits added this to the 2.1.0 milestone Jan 31, 2020

KrzysztofCwalina mentioned this issue Jan 31, 2020

Complete MemoryExtensions Feature #23963

Closed

5 tasks

daveaglick mentioned this issue Feb 1, 2020

ReadOnlySpan<char> Replace extensions #29758

Open

msftbot bot locked as resolved and limited conversation to collaborators Dec 22, 2020

String-like extension methods to ReadOnlySpan<char> Epic #22434

String-like extension methods to ReadOnlySpan<char> Epic #22434

Comments

stephentoub commented Jun 23, 2017

jnm2 commented Sep 15, 2017

safakgur commented Oct 27, 2017 • edited

ahsonkhan commented Dec 12, 2017

ahsonkhan commented Dec 12, 2017

ahsonkhan commented Dec 12, 2017 • edited

Proposed API Additions

jkotas commented Dec 12, 2017 • edited

ahsonkhan commented Dec 12, 2017 • edited

jkotas commented Dec 12, 2017 • edited

benaadams commented Dec 12, 2017

Tragetaschen commented Dec 19, 2017

theolivenbaum commented Dec 19, 2017

KrzysztofCwalina commented Dec 20, 2017

svick commented Dec 20, 2017

Tornhoof commented Dec 21, 2017 • edited

SomeAnonDev commented Dec 27, 2017

jnm2 commented Dec 27, 2017

bbowyersmyth commented Dec 27, 2017

ahsonkhan commented Jan 12, 2018 • edited

Updated Proposed API Additions

jkotas commented Jan 12, 2018 • edited

terrajobst commented Jan 12, 2018 • edited

ahsonkhan commented Jan 13, 2018 • edited

Proposed API

Sample Usage

Open Questions:

terrajobst commented Jan 16, 2018 • edited

benaadams commented Jan 16, 2018

benaadams commented Jan 16, 2018

ahsonkhan commented Jan 16, 2018

ahsonkhan commented Jan 20, 2018 • edited

Joe4evr commented Jan 22, 2018

ahsonkhan commented Jan 22, 2018 • edited

Joe4evr commented Jan 22, 2018

stephentoub commented Jan 22, 2018

ahsonkhan commented Jan 22, 2018 • edited

jkotas commented Jan 23, 2018

Joe4evr commented Jan 23, 2018

ahsonkhan commented Jan 23, 2018 • edited

jkotas commented Jan 23, 2018 • edited

khellang commented Jan 23, 2018 • edited

terrajobst commented Jan 23, 2018 • edited

tarekgh commented Jan 24, 2018 • edited

ahsonkhan commented Jan 24, 2018

KrzysztofCwalina commented Jan 24, 2018

tarekgh commented Jan 24, 2018 • edited

ahsonkhan commented Jan 25, 2018 • edited

KrzysztofCwalina commented Jan 25, 2018

justinvp commented Feb 6, 2018

ahsonkhan commented Feb 6, 2018 • edited

ahsonkhan commented Feb 15, 2018

ahsonkhan commented Feb 28, 2018

KrzysztofCwalina commented Feb 28, 2018

safakgur commented Oct 27, 2017 •

edited

ahsonkhan commented Dec 12, 2017 •

edited

jkotas commented Dec 12, 2017 •

edited

ahsonkhan commented Dec 12, 2017 •

edited

jkotas commented Dec 12, 2017 •

edited

Tornhoof commented Dec 21, 2017 •

edited

ahsonkhan commented Jan 12, 2018 •

edited

jkotas commented Jan 12, 2018 •

edited

terrajobst commented Jan 12, 2018 •

edited

ahsonkhan commented Jan 13, 2018 •

edited

terrajobst commented Jan 16, 2018 •

edited

ahsonkhan commented Jan 20, 2018 •

edited

ahsonkhan commented Jan 22, 2018 •

edited

ahsonkhan commented Jan 22, 2018 •

edited

ahsonkhan commented Jan 23, 2018 •

edited

jkotas commented Jan 23, 2018 •

edited

khellang commented Jan 23, 2018 •

edited

terrajobst commented Jan 23, 2018 •

edited

tarekgh commented Jan 24, 2018 •

edited

tarekgh commented Jan 24, 2018 •

edited

ahsonkhan commented Jan 25, 2018 •

edited

ahsonkhan commented Feb 6, 2018 •

edited