Improve parsing performance

[RoosterDragon]

• In LinguiniParser, ZeroCopyReader, ZeroCopyUtil: change helpers from dealing in ReadOnlyMemory to char. In inner loops this removes a lot of redundant checks for a span of length 1 and inlines more readily.
• In ZeroCopyReader add helpers for SeekEol, IndexOfAnyChar. These use existing helper methods of Span that are SIMD accelerated to search for indexes quickly. These provide significant speedups compared to incrementing the index in a loop.
• In FluentBundle.AddResourceOverriding, provide a specialized 'override' method to avoid two dictionary lookups and allocating an empty list.

---

This makes parsing performance 1.5x faster. On my machine, the following test project runs in 2.9 seconds, compared to 4.5 seconds previously.

Test Project

using System;
using System.Diagnostics;
using System.Globalization;
using System.IO;
using System.Linq;
using Linguini.Bundle.Builder;
using Linguini.Syntax.Parser;

namespace ConsoleApp
{
    internal class Program
    {
        static void Main()
        {
            var sw = new Stopwatch();
            for (var i = 0; i < 10000; i++)
            {
                Setup(out var culture, out var streams);

                sw.Start();
                var bundle = LinguiniBuilder.Builder()
                    .CultureInfo(culture)
                    .SkipResources()
                    .SetUseIsolating(false)
                    .UseConcurrent()
                    .UncheckedBuild();

                foreach (var stream in streams)
                {
                    using var reader = new StreamReader(stream);
                    var parser = new LinguiniParser(reader);
                    var resource = parser.Parse();

                    bundle.AddResourceOverriding(resource);
                }
                sw.Stop();
            }

            sw.Stop();

            Console.WriteLine($"Done in {sw.Elapsed.TotalSeconds:0.00s}");
        }

        static void Setup(out CultureInfo culture, out MemoryStream[] streams)
        {
            culture = new CultureInfo("en");
            var paths = new[]
            {
                @"your-test-files-here.ftl",
            };
            streams = paths.Select(p => new MemoryStream(File.ReadAllBytes(p))).ToArray();
        }
    }
}

I used the following test files, but feel free to try with whatever you have to hand.

• https://github.com/OpenRA/OpenRA/blob/06df75ffee07c074f8ded8854cc9892db2d73cd2/mods/common/languages/en.ftl
• https://github.com/OpenRA/OpenRA/blob/06df75ffee07c074f8ded8854cc9892db2d73cd2/mods/common/languages/rules/en.ftl
• https://github.com/OpenRA/OpenRA/blob/06df75ffee07c074f8ded8854cc9892db2d73cd2/mods/ra/languages/rules/en.ftl

[Ygg01]

Seems to be failing tests.

Also, some benchmarking would be nice. I can add some Benchmarks later (although benchmarking is a bit of a dark science - easy to get wrong).

[Ygg01]

Here are results for current master:

BenchmarkDotNet=v0.13.5, OS=Windows 10 (10.0.19044.2965/21H2/November2021Update)
AMD Ryzen 9 5900X, 1 CPU, 24 logical and 12 physical cores
.NET SDK=7.0.102
[Host] : .NET 5.0.12 (5.0.1221.52207), X64 RyuJIT AVX2 DEBUG

Toolchain=InProcessEmitToolchain

Master:

Method	N	Mean	Error	StdDev
BenchmarkParser	1	3.009 ms	0.0236 ms	0.0197 ms
BenchmarkParser	25	74.498 ms	0.3450 ms	0.3227 ms
BenchmarkParser	50	150.409 ms	0.7432 ms	0.6952 ms

Perf branch:

Method	N	Mean	Error	StdDev
BenchmarkParser	1	1.337 ms	0.0124 ms	0.0110 ms
BenchmarkParser	25	32.971 ms	0.4621 ms	0.4096 ms
BenchmarkParser	50	65.290 ms	0.7167 ms	0.6354 ms

All in all those are some nice speedups.

[Ygg01]

Ok, expect a release soonish. Need to write changelog.

[RoosterDragon]

@Ygg01 looks like the action failed partway through pushing to nuget, just a ping in case you hadn't noticed: https://github.com/Ygg01/Linguini/actions/runs/5237132620

[Ygg01]

@Ygg01 looks like the action failed partway through pushing to nuget, just a ping in case you hadn't noticed: https://github.com/Ygg01/Linguini/actions/runs/5237132620

Yeah, I manually deployed them. I need to sort that deployment script. Look at repo readme , the versions of Linguini.Shared and Linguini.Syntax are updated to v0.5.0.

[RoosterDragon]

May I request Linguini.Bundle as well?