.NET 11 Migration Plan

[jordansrowles]

• Once .NET 11 drops, work will begin to implement the features of the release.
• LeanCorpus may drop support for .NET 10 mid-way between .NET 11 and 12, regardless once 12 previews start coming 10 will be dropped.
• Most of this should remain conditional net11.0 work until .NET 10 is dropped. Don't make the common path worse just to use a shiny API.

So what we gettin?

• Deprecate ZstdSharp compression sidecar for >= .NET 11
• Source-gen polymorphic JSON
• JSON Lines streaming
• AVX-VNNI-512 support
  • Revisit SIMD layer in a spike/experiment
  • Wider ARM/SVE support
• Runtime Async
• Partial numeric parsing
• Rowles.Text Unicode improvements
• Ordinal casing
• Stream adapters / fewer pointless copies
• Hard-link backup/snapshot optimisation
• BFloat16 vector experiment
• NativeAOT/runtime improvements
• System.IO.Pipelines improvements
• Process API cleanup
• Source generator improvements
• Test/devops improvements

Deprecate ZstdSharp compression sidecar for >= .NET 11

• Use native BCL Zstandard through ZstandardStream, ZstandardEncoder and ZstandardDecoder
• Will still need the sidecar for .NET 10
• Do not assume byte-for-byte sameness. Verify format compatibility, dictionary behaviour, options and checksum handling
• Test dictionary compression as well as normal stored-field compression
• Preview 7 also fixed multi-frame decompression and decompression window handling, so test those explicitly
• Keep parity tests between BCL Zstandard and ZstdSharp while both implementations exist

Source-gen polymorphic JSON

• Auto wiring of closed hierarchy classes without doing the [JsonDerivedType] dance
• Need language preview features for closed
• Currently will be useful for the future server, which uses the [JsonDerivedType] approach
• Source generator discovers the permitted derived types and generates the polymorphic metadata
• Explicit [JsonDerivedType] should still win where we need stable/manual discriminators
• Keep this mostly in server/internal contracts for now. Don't expose preview C# language features through the main public API without a good reason
• Preview 7 also fixes some source-gen edge cases around [JsonInclude] and nullable byte[]

JSON Lines streaming

• System.Text.Json can serialise IAsyncEnumerable<T> directly as JSON Lines/NDJSON
• Works with source-generated JsonTypeInfo<T>, so stays aligned with NativeAOT
• Good fit for:
  • bulk document import/export
  • streaming search results
  • server telemetry/events
  • administrative exports
  • benchmark/result streams
• Could make NDJSON a first-class server ingestion format rather than bolting it on later

AVX-VNNI-512 support

• Exposes AvxVnni.V512 including byte/sbyte dot-product multiply-and-accumulate ops over Vector512<int>
• We have Int8DistanceComputer that manually converts quantised bytes into floats before doing vector multiply
• For machines that support AVX-512 VNNI, we can have a specialised path using instructions like VPDPBUSD
• Making the hierarchy something like
  • AVX-VNNI-512
  • AVX2/Vector256
  • AdvSimd
  • generic Vector<T>
  • scalar
• Will need benchmarks, but my machines don't support 256/512 registers, will need to temp rent a VPS again
• Don't let a VNNI fast path turn the vector layer into architecture-specific spaghetti. Keep dispatch centralised

Revisit SIMD layer in a spike/experiment

• Preview 6 added cross-lane operations across Vector128/256/512 including Zip, Unzip, Reverse, Concat* and new construction/composition primitives
• Int8DistanceComputer currently uses Vector256, a stackallocated float buffer and scalar population before loading the vector
• Audit
  • Int8 distance
  • quantisation/dequantisation
  • packed ints
  • numeric aggregators
  • postings intersections
  • Roaring operations
• Look specifically for manual lane construction, scalar extraction/insertion and temporary stack buffers
• Only keep replacements that benchmark better. Cleaner SIMD code that runs slower is still worse

Wider ARM/SVE support

• Vector<T> can use the actual SVE vector width on supported Arm64 hardware rather than effectively being stuck at 128 bits
• Gives portable SIMD code a better chance of scaling on future ARM servers without us writing another complete intrinsic implementation
• Another reason to keep a generic Vector<T> path between architecture-specific intrinsics and scalar
• Add ARM/SVE CI/benchmark coverage when practical, even if initially through rented hardware

Runtime Async

• Now has tiered compilation, previously effectively only being tier 0
• Improved completed Task/ValueTask fast paths, tail-awaits, Task.Yield(), continuation handling and Task/ValueTask adaptation
• Good candidate for indexing, server and other high-throughput async paths

<PropertyGroup Condition="'$(TargetFramework)' == 'net11.0'">
  <Features>runtime-async=on</Features>
</PropertyGroup>

Do benchmark comparisons:

• net10
• net11 normal async
• net11 runtime-async
• net11 NativeAOT runtime-async

Benchmark both hot and boring paths:

• completed operations
• genuinely asynchronous I/O
• cancellation
• exceptions
• long ingestion pipelines
• high-concurrency server requests

Don't enable it globally just because the microbenchmarks look good.

Rowles.Text Unicode improvements

• RunePosition enumerates UTF-8/UTF-16 while giving you the rune's start position and encoded length
• Expands Rune support across String, StringBuilder, TextWriter and casing APIs
• Thai and CJK tokenisers are rune-sensitive analysers
• RunePosition is good because correct token offsets around supplementary Unicode chars become much easier
• Also gained UTF validation helpers such as
  • Utf16.IsValid
  • Utf16.IndexOfInvalidSubsequence
  • Utf8.IndexOfInvalidSubsequence
• Audit anywhere we're manually advancing UTF-16 indices or validating malformed input
• Add supplementary-plane and malformed UTF test cases while touching this

Ordinal casing

string.ToUpperOrdinal();
string.ToLowerOrdinal();
// with Rune and Span support
Rune....();
Span....();

• Mapping is designed to correspond to OrdinalIgnoreCase, with span forms for no allocation
• Potentially useful for:
  • normalisation filters
  • case-insensitive term handling
  • tokenisers
  • dicts
  • FST input prep
  • identifiers/field-name normalisation
• Combine with the Rune work from earlier releases for a Rowles.Text .NET 11 Unicode/casing improvement release
• Be careful not to accidentally change linguistic casing semantics where a language-specific analyser expects them

Partial numeric parsing

• INumberBase<T>.TryParsePartial parses a numeric prefix from a span and returns how much input it consumed
• Could improve:
  • "age:[18 TO 40]"
  • "price:19.95^2.0"
  • "lat,lon"
  • CSV/TSV import
  • numeric query parsing
• Main benefit is avoiding slicing/copying strings around each number
• Worth using in parsers where we already walk a larger input span character by character

Stream adapters / fewer pointless copies

• .NET 11 adds stream wrappers over existing data, including:

  • ReadOnlyMemoryStream
  • WritableMemoryStream
  • ReadOnlySequenceStream
  • StringStream

• Audit places that currently do things like:

  • ReadOnlyMemory<byte> -> MemoryStream
  • ReadOnlySequence<byte> -> byte[] -> Stream
  • string -> UTF-8 byte[] -> Stream

• Mostly a plumbing cleanup, but exactly the sort of pointless allocation LeanCorpus should avoid

• Particularly relevant around codecs, stored fields, server transport and compression

Hard-link backup/snapshot optimisation

• .NET 11 adds first-class hard-link APIs through File.CreateHardLink / FileInfo.CreateAsHardLink
• LeanCorpus segment files are immutable after commit, which makes them a good fit
• Potential backup modes:
  • Copy
  • HardLinkWhenPossible
  • Auto
• Auto can hard-link unchanged segment files on the same filesystem and fall back to copies otherwise
• Could make local backups/snapshots extremely cheap for large indexes
• Need to be very explicit in docs that a hard-linked backup is not an independent physical copy until files diverge
• Never make correctness depend on hard links being supported

BFloat16 vector experiment

• .NET 11 adds proper System.Numerics.BFloat16 support
• Could sit between Float32 and Int8 as another vector representation:
  • Float32
  • BFloat16
  • Int8
  • BBQ/1-bit
• Half the storage of Float32 while retaining a Float32-like exponent range, but with much less precision
• Benchmark:
  • recall
  • storage
  • indexing throughput
  • search throughput
  • rerank quality
• Do not add another vector encoding just for checkbox parity. Only keep it if it finds a useful point in the trade-off curve

NativeAOT/runtime improvements

• Preview 7 improves NativeAOT generic virtual dispatch
• ILC compilation itself has also improved
• Mostly free wins for us because the core already aims to be AOT-compatible
• Still worth adding net10 vs net11 NativeAOT size/startup/runtime comparisons
• Keep watching for runtime changes that let us delete AOT-specific workarounds

System.IO.Pipelines improvements

• Preview 7 reduces contention around internal pipe buffer rent/return paths and improves continuation locality
• Mostly a free server-side improvement
• Doesn't need a LeanCorpus feature by itself, but should show up in future server benchmarks
• Prefer using standard pipelines cleanly rather than trying to outsmart improvements the runtime now does for us

Process API cleanup

.NET 11 adds higher-level process helpers such as:

Process.Run(...);
Process.RunAsync(...);
Process.RunAndCaptureText(...);
Process.RunAndCaptureTextAsync(...);
Process.ReadAllLinesAsync(...);

Plus:

ProcessStartInfo.KillOnParentExit

• Useful for benchmark/profiling tooling that invokes:

  • dotnet-trace
  • dotnet-counters
  • dotnet-gcdump
  • child benchmark processes
  • external verification tools

• KillOnParentExit is especially useful for stopping diagnostic children being orphaned

• Doesn't justify replacing PowerShell devops with C# again. Use it where C# already owns the process

Source generator improvements

• StringBuilder.MoveChunks can move a builder's internal chunk chain rather than copying everything through ToString()
• DocumentMapEmitter and friends currently build generated output through StringBuilder
• Could be useful if the generator pipeline can pass the resulting chunks further before finally producing Roslyn SourceText
• Don't redesign the generator around this unless profiling shows generated-source construction matters
• Also review whether newer source-gen JSON capabilities can remove custom generated glue in server DTOs

Test/devops improvements

• MSBuild server is enabled by default, keeping a warm worker around between repeated builds
• Useful for normal dotnet build / dotnet test loops
• Doesn't magically make file-based C# devops scripts instant, so keep the PowerShell structure

New test controls include things like:

dotnet test --timeout 90s
dotnet test --maximum-failed-tests 5

• Useful for:
  • chaos tests that wedge
  • CI fail-fast
  • architecture/source-gen suites
  • avoiding wasting 20 minutes after something is obviously broken
• Add sane suite-level timeouts and failure caps through devops rather than making contributors remember the flags

Generic random APIs

• Generic Random.NextInteger<T> / NextBinaryFloat<T> APIs can simplify fuzz, chaos and benchmark input generation
• Low priority
• Use opportunistically when touching generators rather than doing a repo-wide refactor

Dont

• Don't turn net11.0 into a completely separate implementation of the engine
• Don't expose preview C# features in the stable public API unless they genuinely improve the model
• Don't add BFloat16, AVX-512 or Runtime Async as marketing checkboxes. Benchmark them
• Don't replace portable SIMD with intrinsics everywhere
• Don't remove .NET 10 fallbacks until .NET 10 support is actually dropped
• Don't assume new BCL implementations are automatically faster than the mature code we already use

The general theme should be: delete dependencies, delete allocations, improve Unicode correctness, improve AOT/server behaviour and add specialised hardware paths where they actually win.