Inlining and structs in C#

[giscus[bot]]

Inlining and structs in C#

In this - somewhat technical and barely usable - blog post, we will have a look at inlining and structs in C#. And how they can optimize performance in some interesting ways.

https://steven-giesel.com/blogPost/e89d7156-f3fd-4152-b78a-cb908bc43226/inlining-and-structs-in-c

[toupswork]

Another reason why they are sort of evil is because the compiler is unable to use a number of optimizations that depend on a value not changing in the middle of an operation; otherwise a stale read is possible. And this potential for a value change is not just multi threading scenarios. There are many examples where a compiler can't rule out that a single method won't cause a value change as a side effect. Examples of compiler optimizations that can't be used include hoisting, enregistering values, simplification of arithmetic and others. And for a mutable struct, passing a value by in creates a defensive copy.

[toupswork]

Just to clarify, there some situations where the compiler can use them on a mutable struct. The odds of it be able to do so decrease when the variable is passed by ref, in, or out.

[linkdotnet]

But with in - don’t you tell the compiler that you don’t wanna mutate the struct? Or better: the compiler would disallow anyway scenario where this could happen.

or am I missing something obvious here?

[toupswork]

Yes. An in parameter carries everything offered by readonly ref, such as you can't reassign it, but in addition to that, it provides a semantic guarantee that the mutable struct will not have its internal state mutated by the callee, even indirectly. For example, calling a method on the passed-in struct that mutates it or repassing it to another method. But here's the thing: The JITs algorithm can't easily determine whether calling a method or property on the struct or repassing it to another method won't mutate it. That's where the defensive copy comes in, which by it's nature is a guarantee of that.

[linkdotnet]

Ahhh - okay. Thanks for the explanation. Always appreciated to get some deeper insights into the workings.

I hope the compiler is smart enough in many cases because otherwise it would defeat a bit the purpose of the in keyword. Especially if you have larger structs and you don’t wanna have the penalty of copying the struct onto the new stackframe.

[toupswork]

The in parameter is one of the most confusing features. There's multiple reasons why they added it, including backwards compatibility.

[toupswork]

By the way, I found a niche use case for InlineArray structs that I never considered before. I wanted to see if this crossed your mind before.
So when they're discussed, it's almost always something where they're looked as using directly. Then people talk about how if theyre too big, then stack overflow. Or that you should pass them by ref to prevent getting expensive copies. Then how it can be boxed in non-obvious ways such as across async boundaries.
But if you use it as a field on a class, you end up with a different scenario. For context, a string is not a character array in disguise nor does it contain a field that is a character array. When it's created, the characters in the string are actually part of the in-memory layout of a string instance. So the size of the string is not a matter of adding up field sizes and padding; rather the instance has a variable size. That's exactly the same effect you get with a class having an inline array as a class field (except the variable size part). The class instance in memory has that inline array field as a buffer inside the instance in memory. So not only do you get better data density and locality performance benefits, but because it's a reference type, you don't get hit with a perfomance penalty if you use it inside an async method or iterator befause the address of class instance is saved to the heap. And that's exactly what I did. I created an IAsyncEnumerable iterator method that used that class as a reusable buffer, where first it reads each item from the IAsyncEnumerable source, immediately added it to the inline array on the class, then once it became full, I did a quick foreach / yield retun on it, then reset the index back to 0 and went back to filling the buffer. The performance improvements were fantastic

public sealed class DbStreamQuery(IDapperProvider dapperProvider, ServiceResultStatusAccessor accessor, ILogger<DbStreamQuery> logger): IDbStreamQuery
{
  public IAsyncEnumerable<T> StreamQuery<T>(EmbeddedQuery query, CancellationToken ct) where T : class => StreamQuery<T>(query, Size128, ct);
  [MethodImpl(MethodImplOptions.AggressiveOptimization)]
  public IAsyncEnumerable<T> StreamQuery<T>(EmbeddedQuery query, BufferSize bufferSize, CancellationToken ct) where T : class => bufferSize switch
  {
    Size128 => StreamQueryWithBuffer<T, Buffer128<T>, InlineArray128<T>>(query, ct),
    Size256 => StreamQueryWithBuffer<T, Buffer256<T>, InlineArray256<T>>(query, ct),
    Size512 => StreamQueryWithBuffer<T, Buffer512<T>, InlineArray512<T>>(query, ct),
    Size64 => StreamQueryWithBuffer<T, Buffer64<T>, InlineArray64<T>>(query, ct),
    _ => throw new ArgumentOutOfRangeException(nameof(bufferSize), bufferSize, null)
  };

  [MethodImpl(MethodImplOptions.AggressiveOptimization)]
  private async IAsyncEnumerable<T> StreamQueryWithBuffer<T, TBuffer, TArray>(EmbeddedQuery query, [EnumeratorCancellation] CancellationToken ct)
    where TBuffer : class, IBuffer<TArray, TBuffer, T>, new()
    where TArray : struct, IInlineArray<TArray, T>
    where T : class
  {
    var buffer = new TBuffer();

    await using var connection = dapperProvider.GetEscherDbConnection();
    await using var e = dapperProvider.GetStreamFromQuery<T>(connection, query).GetAsyncEnumerator(ct);

    var hasMore = true;

    loadBuffer:
    try
    {
      // Keep loading buffer in quick succession until full or no more items
      while (!buffer.IsFull)
      {
        hasMore = await e.MoveNextAsync();

        // No more items -- flush what we have in buffer
        if (!hasMore) break;

        // should always return `true` since we check for IsFull above
        var wasAdded = buffer.TryAdd(e.Current);

        Debug.Assert(wasAdded, "buffer.TryAdd(e.Current) returned false. This should not happen.");
      }
    }
    catch (Exception ex)
    {
      logger.LogEscherQueryFailed(query, ex);
      accessor.ServiceResultStatus = ResultStatus.Failed;
      throw;
    }

    // Empty the buffer of all items before loading it again
    foreach (var item in buffer) yield return item;

    if (hasMore)
    {
      // Arrives here if the buffer was full but there are still more items to read
      buffer.Clear();
      // Continue loading the buffer
      goto loadBuffer;
    }
  }
}

[InlineArray(ArraySize)] 
 public struct InlineArray512<T> : IInlineArray<InlineArray512<T>, T> 
 { 
   const int ArraySize = 512; 
   public static int Size => ArraySize; 
  
   private T _element0; 
  
   [UnscopedRef, MethodImpl(AggressiveInlining)] 
   public ref T At(int i) => ref this[i]; 
  
   public ref T Head 
   { 
     [UnscopedRef, MethodImpl(AggressiveInlining)] 
     get => ref DangerousGetElementAt(0); 
   } 
  
   public ref T Tail 
   { 
     [UnscopedRef, MethodImpl(AggressiveInlining)] 
     get => ref DangerousGetElementAt(ArraySize - 1); 
   } 
  
   [MethodImpl(AggressiveInlining)] 
   public Span<T> AsSpan() => MemoryMarshal.CreateSpan(ref Head, ArraySize); 
  
   [EditorBrowsable(EditorBrowsableState.Never)] 
   public ref T DangerousGetElementAt(int index) => ref Unsafe.Add(ref Unsafe.AsRef(in _element0), index); 
 }

public sealed class Buffer512<T> : IBuffer<InlineArray512<T>, Buffer512<T>, T>, IBuffer<T>
{
  private InlineArray512<T> _inlineBuffer;

  public int MaxSize { get; } = InlineArray512<T>.Size;
  public int Count { get; private set; }

  public bool IsFull
  {
    [MethodImpl(AggressiveInlining)]
    get => Count >= MaxSize;
  }

  [MethodImpl(AggressiveInlining)]
  public ref InlineArray512<T> GetInternalBufferByRef() => ref _inlineBuffer;

  [MethodImpl(AggressiveInlining)]
  public void Clear() => Count = 0;

  [MethodImpl(AggressiveInlining)]
  public bool TryAdd(T item)
  {
    if (IsFull) return false;
    _inlineBuffer.At(Count++) = item;
    return true;
  }

  [MethodImpl(AggressiveInlining)]
  public ref T DangerousGetReference(int index) => ref _inlineBuffer.DangerousGetElementAt(index);

  [MethodImpl(AggressiveInlining)]
  public Span<T> AsSpan() => _inlineBuffer.AsSpan()[..Count];

  [MethodImpl(AggressiveInlining)]
  public BufferEnumerator<T> GetEnumerator() => new(this);
}

[linkdotnet]

Interesting - currently not at home and have to check when I am back next week. Especially with the memorylayout and sharplab.io to get a better understanding on how they behave in those scenarios (strings as well as inlinearrays).