Async Streaming

Async Streaming

TinyTokenizer supports async tokenization from streams and pipes for processing large files without loading them entirely into memory.

Stream Tokenization

TokenizeAsync

Tokenize an entire stream and return all tokens:

using TinyTokenizer;

await using var stream = File.OpenRead("source.txt");
var tokens = await stream.TokenizeAsync();

foreach (var token in tokens)
{
    Console.WriteLine(token);
}

With Options

var options = TokenizerOptions.Default
    .WithOperators(CommonOperators.JavaScript)
    .WithCommentStyles(CommentStyle.CStyleSingleLine);

await using var stream = File.OpenRead("source.js");
var tokens = await stream.TokenizeAsync(options);

Streaming with IAsyncEnumerable

Process tokens as they're parsed without waiting for the entire file:

await using var stream = File.OpenRead("large-file.txt");

await foreach (var token in stream.TokenizeStreamingAsync())
{
    Console.WriteLine(token);

    // Can break early — won't read the entire file
    if (token is ErrorToken)
        break;
}

With Options

var options = TokenizerOptions.Default
    .WithOperators(CommonOperators.CFamily);

await foreach (var token in stream.TokenizeStreamingAsync(options))
{
    // Process token
}

PipeReader Support

For high-performance scenarios, use System.IO.Pipelines:

using System.IO.Pipelines;

var pipe = PipeReader.Create(stream);

await foreach (var token in pipe.TokenizeStreamingAsync())
{
    Console.WriteLine(token);
}

Custom Encoding

Specify encoding for the pipe reader:

using System.Text;

var options = TokenizerOptions.Default;
var encoding = Encoding.UTF8;

await foreach (var token in pipe.TokenizeStreamingAsync(options, encoding))
{
    // Process token
}

DecodingPipeReader

For advanced scenarios, use DecodingPipeReader directly:

using TinyTokenizer;

var pipeReader = PipeReader.Create(stream);
var decoder = new DecodingPipeReader(pipeReader, Encoding.UTF8);

// Use with tokenizer
var tokenizer = new AsyncPipeTokenizer(decoder, TokenizerOptions.Default);

await foreach (var token in tokenizer.TokenizeAsync())
{
    Console.WriteLine(token);
}

Memory Efficiency

Streaming tokenization is memory-efficient for large files:

Approach	Memory Usage	Best For
string.TokenizeToTokens()	Entire file in memory	Small files
stream.TokenizeAsync()	Entire file + all tokens	Medium files
stream.TokenizeStreamingAsync()	Buffered chunks only	Large files
PipeReader.TokenizeStreamingAsync()	Minimal buffering	Very large files

Cancellation

Support cancellation for long-running operations:

var cts = new CancellationTokenSource();
cts.CancelAfter(TimeSpan.FromSeconds(30));

try
{
    await foreach (var token in stream.TokenizeStreamingAsync()
        .WithCancellation(cts.Token))
    {
        // Process token
    }
}
catch (OperationCanceledException)
{
    Console.WriteLine("Tokenization cancelled");
}

Practical Examples

Count Tokens in Large File

int count = 0;
await foreach (var token in stream.TokenizeStreamingAsync())
{
    if (token is not WhitespaceToken)
        count++;
}
Console.WriteLine($"Token count: {count}");

Find All Identifiers

var identifiers = new HashSet<string>();

await foreach (var token in stream.TokenizeStreamingAsync())
{
    if (token is IdentToken ident)
    {
        identifiers.Add(ident.Content.ToString());
    }
}

Early Termination

// Stop after finding first error
await foreach (var token in stream.TokenizeStreamingAsync())
{
    if (token is ErrorToken error)
    {
        Console.WriteLine($"Error at {error.Position}: {error.ErrorMessage}");
        break;  // Don't process rest of file
    }
}

Process Multiple Files

var files = Directory.GetFiles("src", "*.cs", SearchOption.AllDirectories);

foreach (var file in files)
{
    await using var stream = File.OpenRead(file);

    await foreach (var token in stream.TokenizeStreamingAsync())
    {
        // Process token from each file
    }
}

Limitations

• No TinyAst support — Streaming produces tokens, not syntax trees
• Forward-only — Cannot seek backwards in the stream
• No undo/redo — Streaming is read-only

For syntax tree features, load the file and use SyntaxTree.Parse():

// For small/medium files where you need TinyAst
var source = await File.ReadAllTextAsync("source.txt");
var tree = SyntaxTree.Parse(source, Schema.Default);

See Also

• Getting Started — Basic tokenization
• Configuration — TokenizerOptions
• Error Handling — Processing ErrorToken in streams