Investigate using Span<char> for tokenizer/readers etc

[pjoiner]

For netstandard2.0 support use System.Memory

[pjoiner]

See if this can help dotnet/aspnetcore#46098

[pjoiner]

Also, see this dotnet/aspnetcore#49714. Although not sure we'll be able to use this in netstandard

[pjoiner]

So the issue here is that ReadOnlySpan cannot be used in an iterator block

See https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-messages/cs4013?f1url=%3FappId%3DDev16IDEF1%26l%3DEN-US%26k%3Dk(CS4013)%26rd%3Dtrue

Need to investigate if greedy parsing of every line using ReadOnlySpan is more efficient than lazy parsing using string.

[pjoiner]

So just converting to ReadOnlySpan and using slice ToString makes no difference

|         Method |            Runtime |     Mean |     Error |    StdDev |
|--------------- |------------------- |---------:|----------:|----------:|
| TokenizeString |           .NET 5.0 | 1.428 us | 0.3652 us | 0.0200 us |
| TokenizeString |           .NET 6.0 | 1.174 us | 0.0360 us | 0.0020 us |
| TokenizeString |           .NET 7.0 | 1.241 us | 1.0689 us | 0.0586 us |
| TokenizeString | .NET Framework 4.8 | 1.449 us | 0.4930 us | 0.0270 us |