[API Proposal]: Add bson format to .net

[msschl]

Background and motivation

Currently, there is no built-in way to read/parse a bson file. There are also no libraries to convert bson files to json. Newtonsoft has a bson writer and reader but the reader is allocating. The Utf8JsonReader is a robust and good low-level api that can be adapted for bson file reading.

I would propose adapting the Utf8JsonReader to the bson file format.

API Proposal

This is a rough starting point. I have a working version written my self and am using it currently in production. Native support from the BCL however would be great.

namespace System.Formats.Bson;

public struct BsonReaderState
{
    internal bool inObject;

    internal int binaryLength;

    internal BsonElementType elementType;

    internal BsonElementType previousElementType;

    internal BitStack bitStack;
}

public ref partial struct Utf8BsonReader
{
    public Utf8BsonReader(ReadOnlySpan<byte> bsonData, bool isFinalBlock, BsonReaderState state)
    { }

    public Utf8BsonReader(ReadOnlySequence<byte> bsonData, bool isFinalBlock, BsonReaderState state)
    { }

    public bool HasKeySequence { get; private set; }

    public bool HasValueSequence { get; private set; }

    public readonly bool IsFinalBlock { get; }

    public ReadOnlySpan<byte> KeySpan { get; private set; }

    public ReadOnlySpan<byte> ValueSpan { get; private set; }

    public readonly long BytesConsumed { get; }

    public long ElementStartIndex { get; private set; }

    public readonly int CurrentDepth { get; }

    public readonly BsonElementType ElementType { get; }

    public readonly bool IsInArray { get; }

    public ReadOnlySequence<byte> KeySequence { get; private set; }

    public ReadOnlySequence<byte> ValueSequence { get; private set; }

    public readonly SequencePosition Position { get; }

    public readonly BsonReaderState CurrentState { get };

    private bool IsLastSpan { get; }

    public bool Read();
}

public enum BsonElementType : byte
{
    None,
    RootObject,
    StartObject,
    StartArray,
    EndOfObject,
    EndOfArray,
    Binary,
    BinaryData,
    Undefined,
    ObjectId,
    Regex,
    Reference,
    Code,
    CodeWScope,
    Symbol,
    MinKey,
    MaxKey,
    String,
    Number,
    Decimal,
    Boolean,
    Integer,
    Long,
    Date,
    TimeStamp,
    Null,
}

API Usage

using System.Buffers;
using System.IO.Pipelines;
using System.Text;
using System.Text.Json;

using ASGmbH.Lib.Formats.Bson;
using ASGmbH.Lib.Formats.Bson.TestConsole;

Console.WriteLine("Hello, World!");

using var outputFileStream = new FileStream("test-out.json", FileMode.Create, FileAccess.Write);;
using var jsonWriter = new Utf8JsonWriter(outputFileStream, new JsonWriterOptions()
{
    Indented = true,
    Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping
});

using var fileStream = File.OpenRead("test.bson");
var pipeReader = PipeReader.Create(fileStream);

var state = new BsonReaderState();

while (true)
{
    var result = await pipeReader.ReadAsync();
    var buffer = result.Buffer;

    if (result.IsCompleted && buffer.IsEmpty)
        break;

    var position = ParseSequence(ref buffer, result.IsCompleted, ref state, jsonWriter); //, cancellationToken);

    pipeReader.AdvanceTo(position, buffer.End);
}

await pipeReader.CompleteAsync();
await jsonWriter.FlushAsync();
await jsonWriter.DisposeAsync();



static SequencePosition ParseSequence(ref ReadOnlySequence<byte> buffer, bool isFinalBlock, ref BsonReaderState state, Utf8JsonWriter writer) //, CancellationToken cancellationToken)
{
    var reader = new Utf8BsonReader(buffer, isFinalBlock, state);

    while (reader.Read()) // && cancellationToken.IsCancellationRequested is false)
    {
#if DEBUG
        DebugWriteToken(ref reader);
#endif

        WriteJsonToken(ref reader, writer);
    }

    state = reader.CurrentState;

    return reader.Position;
}



#if DEBUG
static void DebugWriteToken(ref Utf8BsonReader reader)
{
    var key = ((reader.HasKeySequence) ?
            Encoding.UTF8.GetString(reader.KeySequence) :
            Encoding.UTF8.GetString(reader.KeySpan));
    var value = (reader.ElementType, reader.HasValueSequence) switch
    {
        (BsonElementType.String, true) => Encoding.UTF8.GetString(reader.ValueSequence),
        (BsonElementType.String, false) => Encoding.UTF8.GetString(reader.ValueSpan),
        (_, true) => ValueSequenceToHexString(reader.ValueSequence),
        (_, false) => Convert.ToHexString(reader.ValueSpan)
    };

    var str = "Element: " + reader.ElementType + ", Key: " + key + ", Value: " + value;
    Console.WriteLine(str);

    string ValueSequenceToHexString(ReadOnlySequence<byte> sequence)
    {
        var length = sequence.Length;
        Span<byte> valueHexBuffer = stackalloc byte[(int)length];

        sequence.CopyTo(valueHexBuffer);

        return Convert.ToHexString(valueHexBuffer);
    }
}
#endif

static void WriteJsonToken(ref Utf8BsonReader reader, Utf8JsonWriter writer)
{
    if (reader.ElementType == BsonElementType.None)
    {
        return;
    }

    if (reader.ElementType == BsonElementType.EndOfDocument)
    {
        if (reader.IsInArray)
            writer.WriteEndObject();
        else
            writer.WriteEndArray();
        return;
    }

    if (reader.ElementType == BsonElementType.RootObject)
    {
        writer.WriteStartObject();
        return;
    }

    if (reader.HasKeySequence && reader.ElementType != BsonElementType.BinaryData && (reader.IsInArray is false || reader.ElementType == BsonElementType.StartArray))
    {
        writer.WritePropertyName(reader.KeySequence);
    }

    if (reader.HasKeySequence is false && reader.ElementType != BsonElementType.BinaryData && (reader.IsInArray is false || reader.ElementType == BsonElementType.StartArray))
    {
        writer.WritePropertyName(reader.KeySpan);
    }

    if (reader.ElementType == BsonElementType.Binary)
    {
        return;
    }

    switch (reader.ElementType)
    {
        case BsonElementType.StartObject:
            writer.WriteStartObject();
            break;

        case BsonElementType.StartArray:
            writer.WriteStartArray();
            break;

        case BsonElementType.Integer when (reader.HasValueSequence):
        {
            // TODO(markus): Check length > MAX_BUFFER_SIZE
            var length = reader.ValueSequence.Length;
            Span<byte> valueStringBuffer = stackalloc byte[(int)length];

            reader.ValueSequence.CopyTo(valueStringBuffer);

            writer.WriteNumberValue(BitConverter.ToInt32(valueStringBuffer));
            break;
        }

        case BsonElementType.Integer:
        {
            writer.WriteNumberValue(BitConverter.ToInt32(reader.ValueSpan));
            break;
        }

        case BsonElementType.String when (reader.HasValueSequence):
        {
            // TODO(markus): Check length > MAX_BUFFER_SIZE
            var length = reader.ValueSequence.Length;
            Span<byte> valueStringBuffer = stackalloc byte[(int)length];

            reader.ValueSequence.CopyTo(valueStringBuffer);

            writer.WriteStringValue(valueStringBuffer);
            break;
        }

        case BsonElementType.String:
            writer.WriteStringValue(reader.ValueSpan);
            break;

        case BsonElementType.BinaryData when (reader.HasValueSequence):
        {
            var length = reader.ValueSequence.Length;
            Span<byte> valueStringBuffer = stackalloc byte[(int)length];

            reader.ValueSequence.CopyTo(valueStringBuffer);

            writer.WriteStringValue(Convert.ToBase64String(valueStringBuffer));
            break;
        }

        case BsonElementType.BinaryData:
        {
            writer.WriteStringValue(Convert.ToBase64String(reader.ValueSpan));
            break;
        }
    }
}

Alternative Designs

No response

Risks

No response

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

Currently, there is no built-in way to read/parse a bson file. There are also no libraries to convert bson files to json. Newtonsoft has a bson writer and reader but the reader is allocating. The Utf8JsonReader is a robust and good low-level api that can be adapted for bson file reading.

I would propose adapting the Utf8JsonReader to the bson file format.

API Proposal

This is a rough starting point. I have a working version written my self and am using it currently in production. Native support from the BCL however would be great.

namespace System.Formats.Bso;

public struct BsonReaderState
{
    internal bool inObject;

    internal int binaryLength;

    internal BsonElementType elementType;

    internal BsonElementType previousElementType;

    internal BitStack bitStack;
}

public ref partial struct Utf8BsonReader
{
    public Utf8BsonReader(ReadOnlySpan<byte> bsonData, bool isFinalBlock, BsonReaderState state)
    { }

    public Utf8BsonReader(ReadOnlySequence<byte> bsonData, bool isFinalBlock, BsonReaderState state)
    { }

    public bool HasKeySequence { get; private set; }

    public bool HasValueSequence { get; private set; }

    public readonly bool IsFinalBlock => this.isFinalBlock;

    public ReadOnlySpan<byte> KeySpan { get; private set; }

    public ReadOnlySpan<byte> ValueSpan { get; private set; }

    public readonly long BytesConsumed { get; }

    public long ElementStartIndex { get; private set; }

    public readonly int CurrentDepth { get; }

    public readonly BsonElementType ElementType { get; }

    public readonly bool IsInArray { get; }

    public ReadOnlySequence<byte> KeySequence { get; private set; }

    public ReadOnlySequence<byte> ValueSequence { get; private set; }

    public readonly SequencePosition Position { get; }

    public readonly BsonReaderState CurrentState { get };

    private bool IsLastSpan { get; }

    public bool Read();
}

public enum BsonElementType : byte
{
    None,

    RootObject,

    StartObject,

    StartArray,

    EndOfObject,
    EndOfArray,

    Binary,

    BinaryData,

    Undefined,

    ObjectId,

    Regex,

    Reference,

    Code,

    CodeWScope,

    Symbol,

    MinKey,

    MaxKey,

    String,

    Number,

    Decimal,

    Boolean,

    Integer,

    Long,

    Date,

    TimeStamp,

    Null,
}

API Usage

using System.Buffers;
using System.IO.Pipelines;
using System.Text;
using System.Text.Json;

using ASGmbH.Lib.Formats.Bson;
using ASGmbH.Lib.Formats.Bson.TestConsole;

Console.WriteLine("Hello, World!");

using var outputFileStream = new FileStream("test-out.json", FileMode.Create, FileAccess.Write);;
using var jsonWriter = new Utf8JsonWriter(outputFileStream, new JsonWriterOptions()
{
    Indented = true,
    Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping
});

using var fileStream = File.OpenRead("test.bson");
var pipeReader = PipeReader.Create(fileStream);

var state = new BsonReaderState();

while (true)
{
    var result = await pipeReader.ReadAsync();
    var buffer = result.Buffer;

    if (result.IsCompleted && buffer.IsEmpty)
        break;

    var position = ParseSequence(ref buffer, result.IsCompleted, ref state, jsonWriter); //, cancellationToken);

    pipeReader.AdvanceTo(position, buffer.End);
}

await pipeReader.CompleteAsync();
await jsonWriter.FlushAsync();
await jsonWriter.DisposeAsync();



static SequencePosition ParseSequence(ref ReadOnlySequence<byte> buffer, bool isFinalBlock, ref BsonReaderState state, Utf8JsonWriter writer) //, CancellationToken cancellationToken)
{
    var reader = new Utf8BsonReader(buffer, isFinalBlock, state);

    while (reader.Read()) // && cancellationToken.IsCancellationRequested is false)
    {
#if DEBUG
        DebugWriteToken(ref reader);
#endif

        WriteJsonToken(ref reader, writer);
    }

    state = reader.CurrentState;

    return reader.Position;
}



#if DEBUG
static void DebugWriteToken(ref Utf8BsonReader reader)
{
    var key = ((reader.HasKeySequence) ?
            Encoding.UTF8.GetString(reader.KeySequence) :
            Encoding.UTF8.GetString(reader.KeySpan));
    var value = (reader.ElementType, reader.HasValueSequence) switch
    {
        (BsonElementType.String, true) => Encoding.UTF8.GetString(reader.ValueSequence),
        (BsonElementType.String, false) => Encoding.UTF8.GetString(reader.ValueSpan),
        (_, true) => ValueSequenceToHexString(reader.ValueSequence),
        (_, false) => Convert.ToHexString(reader.ValueSpan)
    };

    var str = "Element: " + reader.ElementType + ", Key: " + key + ", Value: " + value;
    Console.WriteLine(str);

    string ValueSequenceToHexString(ReadOnlySequence<byte> sequence)
    {
        var length = sequence.Length;
        Span<byte> valueHexBuffer = stackalloc byte[(int)length];

        sequence.CopyTo(valueHexBuffer);

        return Convert.ToHexString(valueHexBuffer);
    }
}
#endif

static void WriteJsonToken(ref Utf8BsonReader reader, Utf8JsonWriter writer)
{
    if (reader.ElementType == BsonElementType.None)
    {
        return;
    }

    if (reader.ElementType == BsonElementType.EndOfDocument)
    {
        if (reader.IsInArray)
            writer.WriteEndObject();
        else
            writer.WriteEndArray();
        return;
    }

    if (reader.ElementType == BsonElementType.RootObject)
    {
        writer.WriteStartObject();
        return;
    }

    if (reader.HasKeySequence && reader.ElementType != BsonElementType.BinaryData && (reader.IsInArray is false || reader.ElementType == BsonElementType.StartArray))
    {
        writer.WritePropertyName(reader.KeySequence);
    }

    if (reader.HasKeySequence is false && reader.ElementType != BsonElementType.BinaryData && (reader.IsInArray is false || reader.ElementType == BsonElementType.StartArray))
    {
        writer.WritePropertyName(reader.KeySpan);
    }

    if (reader.ElementType == BsonElementType.Binary)
    {
        return;
    }

    switch (reader.ElementType)
    {
        case BsonElementType.StartObject:
            writer.WriteStartObject();
            break;

        case BsonElementType.StartArray:
            writer.WriteStartArray();
            break;

        case BsonElementType.Integer when (reader.HasValueSequence):
        {
            // TODO(markus): Check length > MAX_BUFFER_SIZE
            var length = reader.ValueSequence.Length;
            Span<byte> valueStringBuffer = stackalloc byte[(int)length];

            reader.ValueSequence.CopyTo(valueStringBuffer);

            writer.WriteNumberValue(BitConverter.ToInt32(valueStringBuffer));
            break;
        }

        case BsonElementType.Integer:
        {
            writer.WriteNumberValue(BitConverter.ToInt32(reader.ValueSpan));
            break;
        }

        case BsonElementType.String when (reader.HasValueSequence):
        {
            // TODO(markus): Check length > MAX_BUFFER_SIZE
            var length = reader.ValueSequence.Length;
            Span<byte> valueStringBuffer = stackalloc byte[(int)length];

            reader.ValueSequence.CopyTo(valueStringBuffer);

            writer.WriteStringValue(valueStringBuffer);
            break;
        }

        case BsonElementType.String:
            writer.WriteStringValue(reader.ValueSpan);
            break;

        case BsonElementType.BinaryData when (reader.HasValueSequence):
        {
            var length = reader.ValueSequence.Length;
            Span<byte> valueStringBuffer = stackalloc byte[(int)length];

            reader.ValueSequence.CopyTo(valueStringBuffer);

            writer.WriteStringValue(Convert.ToBase64String(valueStringBuffer));
            break;
        }

        case BsonElementType.BinaryData:
        {
            writer.WriteStringValue(Convert.ToBase64String(reader.ValueSpan));
            break;
        }
    }
}

Alternative Designs

No response

Risks

No response

Author:	msschl
Assignees:	-
Labels:	api-suggestion, area-System.Text.Json, untriaged
Milestone:	-

[eiriktsarpalis]

One issue with the System.Text.Json serializer is that unlike Json.NET it is not possible to abstract the underlying writer and reader types. So even if we were to build BSON writer and reader types, they would not be interoperable with any of the serialization abstractions available in the library. As such, it is unlikely we would add BSON support, at least within the System.Text.Json namespace.

[msschl]

That’s why I would propose adding it to the System.Formats.Bson namespace. It is also a binary format.

[eiriktsarpalis]

It's unlikely we'd add support for a new format under System.Formats unless we had a strong motivating use case. For instance System.Formats.Asn1 and System.Formats.Cbor were added to accommodate specific applications cc @terrajobst @bartonjs

[bartonjs]

I'm not sure if the comment is "that namespace is off limits", or "in the interest of sanity, we don't try to add just everything".

If it's the former, I'd disagree with it... we added low-level TAR support via System.Formats.Tar, and we had planned a similar feature as System.Formats.Zip (which was just cut due to resource constraints). If BSON were 100% unrelated to JSON then System.Formats.Bson would be the right place (modulo Bson/BSON/BinaySerializationObjectNotation/etc), since System.Formats is a container namespace for "file format stuff". Since it isn't 100% unrelated there might be an argument for keeping it closer to System.Text.Json... though "Text" does make that hard. (If we'd already "invented" System.Formats, I think it'd've been SFJ instead of STJ).

If it's the latter, then I'd agree. There's a lot of work involved in having file format (or anything, really) support in the BCL, even if it's "not being maintained"; and generally that means that to make it into the BCL it needs to be so ubiquitous that the BCL or SDK wants to depend on it. That generally is also true of System.* NuGet packages. Instead, we share our design guidelines in Framework Design Guidelines, and share our review process through our oft-erroneously named GitHub Quick Reviews stream archive, and hope that a vibrant NuGet community can fill in the gaps. (And, hopefully one day, we can get to "we've added something just because it's popular in the world, even though we don't have a direct need for it ourselves, yet"... but I don't think we're quite that close with file/network format stuff yet)

[eiriktsarpalis]

Since it isn't 100% unrelated there might be an argument for keeping it closer to System.Text.Json... though "Text" does make that hard.

I would say that the argument for keeping a BSON component out of the STJ namespace is a technical one rather than the namespace being appropriate per se. Adding it would almost necessarily give rise to the request of making Utf8BsonWriter interoperate with JsonConverter which we would not be able to accommodate without authoring a parallel BSON serialization layer. Json.NET does not have such constraints making the addition of a BSON component obvious and valuable.