Remove UTF-8 byte[] APIs

- Drop byte[] overloads across Tomlyn (lexer/parser/syntax/reader/serializer) to avoid implying native UTF-8 byte support

- Keep Stream overloads for convenient file IO (UTF-8 only) and clarify behavior in docs

- Remove Encoding.UTF8.GetString usage and obsolete UTF-8 decode helpers

Author: xoofx

site/docs/getting-started.md

@@ -66,7 +66,7 @@ var toml = TomlSerializer.Serialize(new ServerConfig { Host = "example.com", Por
 ## Streams (UTF-8)
 
 > [!TIP]
-> Prefer `Stream` or `byte[]` overloads for files - they avoid allocating an intermediate `string` and read UTF-8 directly.
+> `Stream` overloads are for convenience. Tomlyn reads the entire stream into memory before parsing (see [Performance](performance.md)).
 
 Tomlyn provides `Stream` and `TextReader` overloads to avoid manual `StreamReader`/`StreamWriter` boilerplate:
 
@@ -83,13 +83,6 @@ using var output = File.Create("config_out.toml");
 TomlSerializer.Serialize(output, config);
 ```
 
-You can also deserialize from `byte[]` (UTF-8):
-
-```csharp
-byte[] utf8Bytes = File.ReadAllBytes("config.toml");
-var config = TomlSerializer.Deserialize<ServerConfig>(utf8Bytes)!;
-```
-
 ## Configure options
 
 [`TomlSerializerOptions`](xref:Tomlyn.TomlSerializerOptions) is an immutable `sealed record` - create it once and reuse it:

site/docs/low-level.md

@@ -125,16 +125,12 @@ var parser = TomlParser.Create(toml, parserOptions);
 
 ### Input sources
 
-[`TomlParser.Create(...)`](xref:Tomlyn.Parsing.TomlParser) accepts `string`, `TextReader`, and `byte[]` (UTF-8):
+[`TomlParser.Create(...)`](xref:Tomlyn.Parsing.TomlParser) accepts `string` and `TextReader`:
 
 ```csharp
 // From a file stream
 using var reader = new StreamReader("config.toml");
 var parser = TomlParser.Create(reader);
-
-// From UTF-8 bytes
-byte[] utf8 = File.ReadAllBytes("config.toml");
-var parser2 = TomlParser.Create(utf8);
 ```
 
 ## SyntaxParser (full-fidelity syntax tree)
@@ -168,7 +164,7 @@ Console.WriteLine(doc.ToString());
 | [`SyntaxParser.Parse(...)`](xref:Tomlyn.Parsing.SyntaxParser) | Tolerant - collects errors in `doc.Diagnostics`, always returns a tree. |
 | [`SyntaxParser.ParseStrict(...)`](xref:Tomlyn.Parsing.SyntaxParser) | Strict - throws [`TomlException`](xref:Tomlyn.TomlException) on the first error. |
 
-Both accept `string` and [`TomlLexer`](xref:Tomlyn.Parsing.TomlLexer) inputs. `Parse` also accepts `TextReader` and `byte[]`.
+Both accept `string` and [`TomlLexer`](xref:Tomlyn.Parsing.TomlLexer) inputs. `Parse` also accepts `TextReader`.
 
 ### Syntax tree structure
 

site/docs/performance.md

@@ -2,7 +2,7 @@
 title: Performance
 ---
 
-Tomlyn is designed for high throughput and low allocations.
+Tomlyn is designed for high throughput and low allocations for **configuration-style** TOML workloads.
 
 ## Architecture
 
@@ -13,20 +13,29 @@ Tomlyn is designed for high throughput and low allocations.
 | [`TomlSerializer`](xref:Tomlyn.TomlSerializer) | Reads directly from the parser stream when possible; source-generated metadata avoids reflection. |
 | [`SyntaxParser`](xref:Tomlyn.Parsing.SyntaxParser) | Allocates tree nodes - use only when full-fidelity round-tripping is needed. |
 
+## What “high-performance” means for Tomlyn
+
+Tomlyn is optimized around the common TOML use-case: **configuration files** that can be loaded in memory.
+Tomlyn is not a streaming parser and it does not aim to match the design goals of `System.Text.Json` for high-throughput web API scenarios.
+
+Internally, Tomlyn parses a **UTF-16 `ReadOnlyMemory<char>`** view of the TOML payload.
+This means that:
+
+- Passing TOML as `string` is the most direct path.
+- When you use `Stream` or `TextReader`, Tomlyn reads the entire payload into memory before parsing.
+
 ## Practical tips
 
 | Tip | Why |
 | --- | --- |
 | **Cache [`TomlSerializerOptions`](xref:Tomlyn.TomlSerializerOptions)** | It is an immutable `sealed record`. Creating it once avoids repeated metadata resolution. |
 | **Prefer source generation** | [`TomlSerializerContext`](xref:Tomlyn.Serialization.TomlSerializerContext) / [`TomlTypeInfo<T>`](xref:Tomlyn.TomlTypeInfo`1) avoids reflection, reduces startup, and is required for NativeAOT. |
-| **Use `Stream` overloads** | [`TomlSerializer.Deserialize<T>(Stream)`](xref:Tomlyn.TomlSerializer) reads UTF-8 directly, avoiding a full-string allocation. |
-| **Use `byte[]` overloads** | [`TomlSerializer.Deserialize<T>(byte[])`](xref:Tomlyn.TomlSerializer) avoids the `string` → `byte[]` conversion for pre-loaded data. |
+| **Use `Stream`/`TextReader` for convenience** | Tomlyn reads the entire content into memory before parsing; use these overloads to integrate with existing IO code. |
 | **Set `SourceName` once** | Attach it to cached options, not per-call - it doesn't affect parsing, only exception messages. |
 | **Avoid [`SyntaxParser`](xref:Tomlyn.Parsing.SyntaxParser) for mapping** | [`SyntaxParser`](xref:Tomlyn.Parsing.SyntaxParser) builds a full tree. If you only need .NET objects, use [`TomlSerializer`](xref:Tomlyn.TomlSerializer) directly. |
 
 > [!TIP]
-> For the best performance, combine source generation with `Stream` or `byte[]` overloads.
-> This gives you zero-reflection, zero-intermediate-string deserialization.
+> For the best performance, combine source generation with a cached `TomlSerializerContext` and cached `TomlSerializerOptions`.
 
 ## When to use each API
 

site/docs/serialization.md

@@ -19,17 +19,13 @@ var toml = TomlSerializer.Serialize(new Person("Ada", 37));
 var person = TomlSerializer.Deserialize<Person>(toml)!;
 ```
 
-[`TomlSerializer`](xref:Tomlyn.TomlSerializer) provides overloads for `string`, `byte[]` (UTF-8), `Stream`, and `TextReader`/`TextWriter`:
+[`TomlSerializer`](xref:Tomlyn.TomlSerializer) provides overloads for `string`, `Stream`, and `TextReader`/`TextWriter`:
 
 ```csharp
-// From/to a file stream (UTF-8)
+// From a file stream (UTF-8)
 using var stream = File.OpenRead("config.toml");
 var config = TomlSerializer.Deserialize<MyConfig>(stream);
 
-// From a UTF-8 byte array
-byte[] utf8 = File.ReadAllBytes("config.toml");
-var config2 = TomlSerializer.Deserialize<MyConfig>(utf8);
-
 // To a TextWriter
 using var writer = new StreamWriter("output.toml");
 TomlSerializer.Serialize(writer, config);
@@ -576,5 +572,5 @@ if (!TomlSerializer.TryDeserialize<MyConfig>(toml, out var config))
 }
 ```
 
-`TryDeserialize` is available for all input types (`string`, `byte[]`, `Stream`, `TextReader`)
+`TryDeserialize` is available for all input types (`string`, `Stream`, `TextReader`)
 and with all metadata styles (options, context, type info).

src/Tomlyn.Tests/NewApiExceptionLocationTests.cs

@@ -41,29 +41,6 @@ public void Parse_SyntaxError_IncludesLocation()
         Assert.That(ex.Message, Does.Contain("test.toml("));
     }
 
-    [Test]
-    public void Parse_InvalidUtf8_IncludesByteOffset()
-    {
-        // "a = " followed by an invalid UTF-8 sequence: 0xC3 0x28
-        var bytes = new byte[] { (byte)'a', (byte)' ', (byte)'=', (byte)' ', 0xC3, 0x28, (byte)'\n' };
-        var options = new TomlSerializerOptions { SourceName = "utf8.toml" };
-
-        var parser = TomlParser.Create(bytes, options);
-        var ex = Assert.Throws<TomlException>(() =>
-        {
-            while (parser.MoveNext())
-            {
-            }
-        });
-
-        Assert.That(ex, Is.Not.Null);
-        Assert.That(ex!.Span.HasValue, Is.True);
-        Assert.That(ex.Offset, Is.EqualTo(4));
-        Assert.That(ex.Line, Is.EqualTo(1));
-        Assert.That(ex.Column, Is.EqualTo(5));
-        Assert.That(ex.Message, Does.Contain("utf8.toml("));
-    }
-
     [Test]
     public void Parse_InvalidKeyHexEscape_IncludesLocation()
     {

src/Tomlyn.Tests/NewApiMetadataStreamOverloadsTests.cs

@@ -1,4 +1,5 @@
 using System.IO;
+using System.Text;
 using NUnit.Framework;
 using Tomlyn.Model;
 using Tomlyn.Serialization;
@@ -38,27 +39,21 @@ public void Serialize_Stream_ObjectTypeInfo_WritesToml()
     }
 
     [Test]
-    public void Deserialize_Utf8Bytes_ObjectTypeInfo_UsesByteOffsets()
+    public void Deserialize_Stream_ObjectTypeInfo_Works()
     {
         var context = new BuiltInContext();
         var typeInfo = context.GetTypeInfo(typeof(TomlTable), context.Options);
         Assert.NotNull(typeInfo);
 
-        var bytes = new byte[]
+        using var stream = new MemoryStream();
+        using (var writer = new StreamWriter(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false), bufferSize: 1024, leaveOpen: true))
         {
-            (byte)'a',
-            (byte)' ',
-            (byte)'=',
-            (byte)' ',
-            (byte)'"',
-            0xC3,
-            0x28,
-            (byte)'"',
-        };
+            writer.Write("a = 1\n");
+        }
 
-        var ex = Assert.Throws<TomlException>(() => TomlSerializer.Deserialize(bytes, typeInfo!));
-        Assert.NotNull(ex);
-        Assert.AreEqual(5, ex!.Offset);
+        stream.Position = 0;
+        var table = (TomlTable)TomlSerializer.Deserialize(stream, typeInfo!)!;
+        Assert.AreEqual(1L, (long)table["a"]);
     }
 
     [Test]

src/Tomlyn.Tests/NewApiSerializerOverloadTests.cs

@@ -37,11 +37,17 @@ public void Deserialize_String_Type_WithContext_Works()
     }
 
     [Test]
-    public void Deserialize_Utf8Bytes_WithContext_Works()
+    public void Deserialize_Stream_WithContext_Works()
     {
         var context = TestTomlSerializerContext.Default;
-        var bytes = Encoding.UTF8.GetBytes(SampleToml);
-        var person = TomlSerializer.Deserialize<GeneratedPerson>(bytes, context);
+        using var stream = new MemoryStream();
+        using (var writer = new StreamWriter(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false), bufferSize: 1024, leaveOpen: true))
+        {
+            writer.Write(SampleToml);
+        }
+
+        stream.Position = 0;
+        var person = TomlSerializer.Deserialize<GeneratedPerson>(stream, context);
 
         Assert.That(person, Is.Not.Null);
         Assert.That(person!.Name, Is.EqualTo("Ada"));
@@ -75,7 +81,13 @@ public void Deserialize_TextReader_Type_WithContext_Works()
     public void Deserialize_Stream_WithTypeInfo_Works()
     {
         var context = TestTomlSerializerContext.Default;
-        using var stream = new MemoryStream(Encoding.UTF8.GetBytes(SampleToml));
+        using var stream = new MemoryStream();
+        using (var writer = new StreamWriter(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false), bufferSize: 1024, leaveOpen: true))
+        {
+            writer.Write(SampleToml);
+        }
+
+        stream.Position = 0;
         var person = TomlSerializer.Deserialize(stream, context.GeneratedPerson);
 
         Assert.That(person, Is.Not.Null);
@@ -124,12 +136,17 @@ public void TryDeserialize_String_WithContext_ReturnsFalseOnFailure()
     }
 
     [Test]
-    public void TryDeserialize_Utf8Bytes_WithContext_ReturnsFalseOnFailure()
+    public void TryDeserialize_Stream_WithContext_ReturnsFalseOnFailure()
     {
         var context = TestTomlSerializerContext.Default;
-        var bytes = Encoding.UTF8.GetBytes("name = \"Ada\"\nage = \"not-a-number\"\n");
+        using var stream = new MemoryStream();
+        using (var writer = new StreamWriter(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false), bufferSize: 1024, leaveOpen: true))
+        {
+            writer.Write("name = \"Ada\"\nage = \"not-a-number\"\n");
+        }
 
-        var ok = TomlSerializer.TryDeserialize<GeneratedPerson>(bytes, context, out var value);
+        stream.Position = 0;
+        var ok = TomlSerializer.TryDeserialize<GeneratedPerson>(stream, context, out var value);
 
         Assert.That(ok, Is.False);
         Assert.That(value, Is.Null);
@@ -151,7 +168,13 @@ public void TryDeserialize_TextReader_Type_WithContext_ReturnsFalseOnFailure()
     public void TryDeserialize_Stream_Type_WithContext_ReturnsFalseOnFailure()
     {
         var context = TestTomlSerializerContext.Default;
-        using var stream = new MemoryStream(Encoding.UTF8.GetBytes("name = \"Ada\"\nage = \"not-a-number\"\n"));
+        using var stream = new MemoryStream();
+        using (var writer = new StreamWriter(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false), bufferSize: 1024, leaveOpen: true))
+        {
+            writer.Write("name = \"Ada\"\nage = \"not-a-number\"\n");
+        }
+
+        stream.Position = 0;
 
         var ok = TomlSerializer.TryDeserialize(stream, typeof(GeneratedPerson), context, out var value);
 

src/Tomlyn.Tests/NewApiUtf8StreamExceptionLocationTests.cs

@@ -1,3 +1,5 @@
+using System.IO;
+using System.Text;
 using NUnit.Framework;
 using Tomlyn.Model;
 
@@ -6,40 +8,34 @@ namespace Tomlyn.Tests;
 public sealed class NewApiUtf8StreamExceptionLocationTests
 {
     [Test]
-    public void Deserialize_Utf8Bytes_InvalidUtf8_ThrowsTomlExceptionWithByteOffset()
+    public void Deserialize_Stream_SyntaxError_ThrowsTomlExceptionWithLocation()
     {
-        // a = "<invalid utf-8>"
-        var bytes = new byte[]
+        using var stream = new MemoryStream();
+        using (var writer = new StreamWriter(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false), bufferSize: 1024, leaveOpen: true))
         {
-            (byte)'a',
-            (byte)' ',
-            (byte)'=',
-            (byte)' ',
-            (byte)'"',
-            0xC3, // invalid sequence start (expects continuation byte in 0x80..0xBF, but 0x28 is '(')
-            0x28,
-            (byte)'"',
-        };
+            writer.Write("a = [\n");
+        }
 
-        var ex = Assert.Throws<TomlException>(() => TomlSerializer.Deserialize<TomlTable>(bytes));
+        stream.Position = 0;
+        var ex = Assert.Throws<TomlException>(() => TomlSerializer.Deserialize<TomlTable>(stream));
         Assert.NotNull(ex);
         Assert.NotNull(ex!.Span);
 
-        Assert.AreEqual(5, ex.Offset, "Offset must be the character position of the invalid sequence.");
-        Assert.AreEqual(1, ex.Line);
-        Assert.AreEqual(6, ex.Column);
+        Assert.That(ex.Line, Is.EqualTo(2));
+        Assert.That(ex.Column, Is.GreaterThan(0));
     }
 
     [Test]
-    public void Deserialize_Utf8Bytes_WithBom_Succeeds()
+    public void Deserialize_Stream_WithBom_Succeeds()
     {
-        var bytes = new byte[]
+        using var stream = new MemoryStream();
+        using (var writer = new StreamWriter(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: true), bufferSize: 1024, leaveOpen: true))
         {
-            0xEF, 0xBB, 0xBF, // UTF-8 BOM
-            (byte)'a', (byte)' ', (byte)'=', (byte)' ', (byte)'1', (byte)'\n',
-        };
+            writer.Write("a = 1\n");
+        }
 
-        var table = TomlSerializer.Deserialize<TomlTable>(bytes);
+        stream.Position = 0;
+        var table = TomlSerializer.Deserialize<TomlTable>(stream);
 
         Assert.NotNull(table);
         Assert.AreEqual(1L, (long)table!["a"]);

src/Tomlyn.Tests/StandardTests.cs

@@ -105,14 +105,15 @@ private static void ValidateSpec(string type, string inputName, string toml, str
             }
 
             {
-                var docUtf8 = SyntaxParser.Parse(Encoding.UTF8.GetBytes(toml), inputName);
-                var roundtripUtf8 = docUtf8.ToString();
-                if (roundtrip != roundtripUtf8)
+                using var reader = new StringReader(toml);
+                var docFromReader = SyntaxParser.Parse(reader, inputName);
+                var roundtripFromReader = docFromReader.ToString();
+                if (roundtrip != roundtripFromReader)
                 {
                     Console.WriteLine($"Testing {inputName}");
                     Dump(toml, doc, roundtrip);
                 }
-                Assert.AreEqual(roundtrip, roundtripUtf8, "The UTF8 version doesn't match with the UTF16 version");
+                Assert.AreEqual(roundtrip, roundtripFromReader, "The TextReader version doesn't match with the string version");
             }
         }
 
@@ -211,6 +212,13 @@ public static IEnumerable ListTomlFiles(string type)
 
                 if (type == InvalidSpec)
                 {
+                    // The toml-test "invalid/encoding" suite validates raw UTF-8 byte-level correctness.
+                    // Tomlyn's test harness feeds TOML as text (string/TextReader), so these cases are not applicable here.
+                    if (normalizedFile.IndexOf("/invalid/encoding/", StringComparison.OrdinalIgnoreCase) >= 0)
+                    {
+                        goto next_file;
+                    }
+
                     for (var i = 0; i < Toml11ValidButTomlTestMarksInvalid.Length; i++)
                     {
                         if (normalizedFile.EndsWith(Toml11ValidButTomlTestMarksInvalid[i], StringComparison.OrdinalIgnoreCase))
@@ -222,7 +230,7 @@ public static IEnumerable ListTomlFiles(string type)
 
                 var functionName = Path.GetFileName(file);
 
-                var input = Encoding.UTF8.GetString(File.ReadAllBytes(file));
+                var input = File.ReadAllText(file);
 
                 string? json = null;
                 if (type == "valid")