Reading bits from arbitrary data types (especially byte[array])

[juliusfriedman]

Background and Motivation

Is very common, we have BitVector BitSet both of which are little endian only afaik.

Proposed API

namespace System.Buffers
{
//nuint instead of int ,long maybe better?
+public static long ReadBits(byte[] bytes, int bitOffset, int bitCount, bool reverse);
+public static void WriteBits(byte[] bytes, int bitOffset, int bitCount, long value, bool reverse)
     }

Usage Examples

Writing a LE 16 bit value on BE

var value = System.Buffers.WriteBits(someByteArray, 0, 16, false == BitConverter.IsLittleEndian)

Writing a BE 16 bit value on LE

var value = System.Buffers.WriteBits(someByteArray, 0, 16, ushort.MaxValue, BitConverter.IsLittleEndian)

Alternative Designs

Not sure, Prior art here

Risks

LOW, People get this wrong all the time.

cc @danmosemsft

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[mburbea]

Are you missing some parameters in your example?
What's wrong with the existing BinaryPrimitives?

BinaryPrimitives.TryWriteInt64LittleEndian(bytes.AsSpan(offset), 8675309u);
BinaryPrimitives.TryWriteInt64BigEndian(bytes.AsSpan(offset), 8675309u);

[juliusfriedman]

Are you missing some parameters in your example?
What's wrong with the existing BinaryPrimitives?

BinaryPrimitives.TryWriteInt64LittleEndian(bytes.AsSpan(offset), 8675309u);
BinaryPrimitives.TryWriteInt64BigEndian(bytes.AsSpan(offset), 8675309u);

Nothing but I just want the higher level class that can do both as the same time, I think my proposed API is more user friendly.

Additionally what if you wanted 3 bits?

Skip 2 bits and then read 1 bit?

Yes a mask is probably better but the API / JIT can maybe even optimize that depending on certain things.

This API allows reading from any position and any number of bits up to 64 but technically not limited to such.

Tagging subscribers to this area: @tannergooding, @pgovind
Notify danmosemsft if you want to be subscribed.

[mburbea]

Your api lists a method for write with 5 parameters, yet you're calling it with 4.

[FiniteReality]

I don't think users get this wrong enough to warrant APIs like this. BinaryPrimitives works good enough for >8 bits, and just getting the byte out of the array works for <=8 bits.

[juliusfriedman]

People need to read nibbles and bits. Sometimes 1 likely more, mostly not aligned.

https://tools.ietf.org/html/rfc3550#section-5.1

[danmoseley]

@juliusfriedman some examples of evidence of need we see in API proposals are - existing .NET libraries that are widely used to solve this problem; popular examples in other major ecosystems; several customers or community members sharing examples of how they would use it; examples of code in this repo or other repos (grep.app) that would be improved if this API exists; etc.

Every API has a cost - as well as design/implementation there is also maintenance, docs, and cognitive load on users. That API will exist likely for 10-20 years or more. So we have to have fairly solid evidence it will be used and valued - it has to be more than some apparent value.

[juliusfriedman]

I understand you. Thank you for your input!

I would also like to see Endian agnostic support but myself was only able to cope with Big, Little and the middles, who knows what I missed and I did not propose anything hard IMHO or that would not provide value. I did not propose the Endian converter or other such types like BitOrder/ByteOrder etc which I have implemented and do actually work.

With that typed, how would you suggest users handling something like a MPEG TS bit stream? Most of the fields are bit packed there and also mostly not aligned.

Yes they can read in bytes, mask and shift but this is work which users will get wrong where as these API's cannot.

I will find the examples but SNIPacket I believe as well as TDSStream are going to use these as well as in any other place where you want to read more than 1 bit from a byte at an unaligned location.

I know R2R also has methods for reading bits separated from the runtime but no writing bit capabilities. Those methods are optimized and do not use BinaryPrimitives.

runtime/src/coreclr/src/tools/aot/ILCompiler.Reflection.ReadyToRun/Amd64/GcInfo.cs

Line 113 in 6072e4d

ReturnKind = (ReturnKinds)NativeReader.ReadBits(image, returnKindBits, ref bitOffset);

https://github.com/dotnet/runtime/search?q=ReadBits&unscoped_q=ReadBits

Thank you for the link

(AsnReader,KerberosAPRequestAuthenticationToken, etc)
https://grep.app/search?q=ReadBits&filter[path.pattern][0]=dotnet

I also also fairly certain that in HTTP2, Compression and various other places this API will be useful and performance oriented vs the user trying to mask and shift themselves.

[john-h-k]

why not just MemoryMarshal.Read<T> and shift the outputted bits (as that is probably the most efficient way anyway)?

[FiniteReality]

With that typed, how would you suggest users handling something like a MPEG TS bit stream? Most of the fields are bit packed there and also mostly not aligned.

Most of the values in MPEG streams are actually aligned to some useful position: each set of related values is packed into a single byte, so you can easily use masking and shifting to read them. Looking at the documentation on wikipedia, I can do this:

// Expects a packet aligned to the first sync byte
bool TryReadMpegTsPacket(ReadOnlySpan<byte> packet)
{
    var syncByte = stream[0];
    if (syncByte != 0x74) return false; // invalid sync byte

    if (!BinaryPrimitives.TryReadUInt16BigEndian(packet.Slice(1), out ushort packetMetadata))
        return false; // not enough bytes

    var packetFields = stream[3];

    var errorIndicator = (packetMetadata & 0x8000) >> 15;
    var startIndicator = (packetMetadata & 0x4000) >> 14;
    var priority = (packetMetadata & 0x2000) >> 13;
    var packetId = packetMetadata & 0x1FFF);
    var scramblingControl = (packetFields & 0xC0) >> 6;
    var fieldControl = (packetFields & 0x30) >> 4;
    var continuityCounter = (packetFields & 0x0F);

    // TODO: validate, likely assign out parameters, etc.
    return true;
}

[juliusfriedman]

why not just MemoryMarshal.Read<T> and shift the outputted bits (as that is probably the most efficient way anyway)?

Endian and BitOrder are seperate things and not all bits are contiguous...