Skip to content

Commit

Permalink
Merge pull request #37 from houseofcat/master
Browse files Browse the repository at this point in the history
Release v1.2.2
  • Loading branch information
houseofcat committed Nov 15, 2021
2 parents 30b3493 + 2981b5e commit 9039355
Show file tree
Hide file tree
Showing 11 changed files with 253 additions and 68 deletions.
82 changes: 42 additions & 40 deletions guides/csharp/CompressionNumeroTres.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ your situation. You must figure that out for yourself based on your systems and
while reading their guide on RecyclableMemoryStream *sucked shit* in terms of allocations and performance - over just default - when
benchmarking. It's possible that its just improved over time.

I have included my default setup, but this one primarily only worked well for a segment of message sizes I was dealing with.
I have included my default setup, but this one only worked well for a segment of message sizes I was dealing with.
It's a specific use case and I am not convinced about even keeping this.

#### Important!
Expand Down Expand Up @@ -245,7 +245,7 @@ out the `Stream` that you can then dispose of afterwards.
If you are using the Recyclable classes from my library, you will want an implementation that looks like this. I have included one
under Houseofcat.Data.Recyclable called the `RecyclableTransformer`.

This a snippet of it under the hood.
This is a snippet of it under the hood.

```csharp
using HouseofCat.Compression;
Expand Down Expand Up @@ -321,8 +321,9 @@ namespace HouseofCat.Data.Recyclable
}
```

My HouseofCat libraries really are meant to streamline taking an object `<TIn>` and outputting serialized, compressed, and encrypted
bytes and then the ability to `Restore` that back to the original object!
A couple of my HouseofCat libraries really are meant to streamline taking an object `<TIn>` and outputting serialized, compressed,
and encrypted bytes and then the ability to `Restore` that back to the original object! You should check them out if you are in
the market for streamlining portions of system.

### Benchmarks
There is always room for improvement, but this seems pretty damn good as an out-of-the-box improvement!
Expand All @@ -342,43 +343,44 @@ AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores

Job=.NET 5.0 Runtime=.NET 5.0

| Method | x | Mean | Error | StdDev | Median | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated | Decrease |
|------------------------------------- |----- |------------:|----------:|------------:|------------:|------:|--------:|---------:|--------:|------:|----------:|---------:|
| BasicCompressDecompress_5KBytes | 10 | 213.3 us | 4.25 us | 9.14 us | 215.8 us | 1.00 | 0.00 | 6.5918 | - | - | 109 KB | - % |
| GzipProvider_5KBytes | 10 | 203.5 us | 4.06 us | 10.98 us | 210.1 us | 0.94 | 0.05 | 3.4180 | - | - | 59 KB | 45.872 % |
| GzipProviderStream_5KBytes | 10 | 206.5 us | 4.11 us | 10.97 us | 210.9 us | 0.96 | 0.06 | 3.4180 | - | - | 58 KB | 46.789 % |
| RecyclableGzipProvider_5KBytes | 10 | 207.5 us | 4.10 us | 9.08 us | 211.3 us | 0.97 | 0.06 | 0.7324 | 0.2441 | - | 13 KB | 88.073 % |
| RecyclableGzipProviderStream_5KBytes | 10 | 203.6 us | 2.16 us | 1.91 us | 203.8 us | 0.97 | 0.06 | 0.7324 | - | - | 13 KB | 88.073 % |
| | | | | | | | | | | | | |
| BasicCompressDecompress_5KBytes | 100 | 2,188.8 us | 42.41 us | 47.14 us | 2,195.4 us | 1.00 | 0.00 | 66.4063 | - | - | 1,088 KB | - % |
| GzipProvider_5KBytes | 100 | 1,909.6 us | 29.74 us | 27.82 us | 1,911.0 us | 0.87 | 0.02 | 35.1563 | - | - | 588 KB | 45.956 % |
| GzipProviderStream_5KBytes | 100 | 1,897.6 us | 22.44 us | 18.73 us | 1,892.8 us | 0.86 | 0.02 | 35.1563 | - | - | 583 KB | 46.415 % |
| RecyclableGzipProvider_5KBytes | 100 | 2,342.6 us | 45.13 us | 48.29 us | 2,335.8 us | 1.07 | 0.03 | 3.9063 | - | - | 121 KB | 88.879 % |
| RecyclableGzipProviderStream_5KBytes | 100 | 2,182.7 us | 79.15 us | 223.26 us | 2,290.1 us | 0.95 | 0.12 | 5.8594 | - | - | 126 KB | 88.420 % |
| | | | | | | | | | | | | |
| BasicCompressDecompress_5KBytes | 500 | 11,190.1 us | 213.12 us | 218.86 us | 11,148.9 us | 1.00 | 0.00 | 328.1250 | - | - | 5,438 KB | - % |
| GzipProvider_5KBytes | 500 | 10,144.5 us | 214.08 us | 631.21 us | 10,449.1 us | 0.84 | 0.06 | 171.8750 | - | - | 2,938 KB | 45.973 % |
| GzipProviderStream_5KBytes | 500 | 11,105.7 us | 143.25 us | 126.99 us | 11,110.2 us | 0.99 | 0.03 | 171.8750 | - | - | 2,914 KB | 46.414 % |
| RecyclableGzipProvider_5KBytes | 500 | 11,653.7 us | 182.54 us | 179.28 us | 11,656.0 us | 1.04 | 0.03 | 31.2500 | 15.6250 | - | 635 KB | 88.323 % |
| RecyclableGzipProviderStream_5KBytes | 500 | 11,433.7 us | 171.46 us | 160.39 us | 11,420.8 us | 1.02 | 0.02 | 31.2500 | - | - | 629 KB | 88.433 % |
| | | | | | | | | | | | | |
| BasicCompressDecompress_5KBytes | 1000 | 20,582.4 us | 422.06 us | 1,237.82 us | 21,169.3 us | 1.00 | 0.00 | 656.2500 | - | - | 10,875 KB | - % |
| GzipProvider_5KBytes | 1000 | 20,492.8 us | 481.45 us | 1,419.57 us | 21,292.6 us | 1.00 | 0.07 | 343.7500 | - | - | 5,875 KB | 45.977 % |
| GzipProviderStream_5KBytes | 1000 | 20,491.4 us | 440.79 us | 1,299.68 us | 20,990.3 us | 1.00 | 0.05 | 343.7500 | - | - | 5,828 KB | 46.641 % |
| RecyclableGzipProvider_5KBytes | 1000 | 18,568.1 us | 42.56 us | 33.23 us | 18,570.2 us | 0.94 | 0.05 | 62.5000 | 31.2500 | - | 1,271 KB | 88.313 % |
| RecyclableGzipProviderStream_5KBytes | 1000 | 18,192.9 us | 120.91 us | 134.40 us | 18,188.5 us | 0.96 | 0.07 | 62.5000 | - | - | 1,258 KB | 88.432 % |
| Method | x | Mean | Error | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated | Decrease |
|------------------------------------- |----- |------------:|----------:|------:|--------:|---------:|--------:|------:|----------:|---------:|
| BasicCompressDecompress_5KBytes | 10 | 213.3 us | 4.25 us | 1.00 | 0.00 | 6.5918 | - | - | 109 KB | - % |
| GzipProvider_5KBytes | 10 | 203.5 us | 4.06 us | 0.94 | 0.05 | 3.4180 | - | - | 59 KB | 45.872 % |
| GzipProviderStream_5KBytes | 10 | 206.5 us | 4.11 us | 0.96 | 0.06 | 3.4180 | - | - | 58 KB | 46.789 % |
| RecyclableGzipProvider_5KBytes | 10 | 207.5 us | 4.10 us | 0.97 | 0.06 | 0.7324 | 0.2441 | - | 13 KB | 88.073 % |
| RecyclableGzipProviderStream_5KBytes | 10 | 203.6 us | 2.16 us | 0.97 | 0.06 | 0.7324 | - | - | 13 KB | 88.073 % |
| | | | | | | | | | | |
| BasicCompressDecompress_5KBytes | 100 | 2,188.8 us | 42.41 us | 1.00 | 0.00 | 66.4063 | - | - | 1,088 KB | - % |
| GzipProvider_5KBytes | 100 | 1,909.6 us | 29.74 us | 0.87 | 0.02 | 35.1563 | - | - | 588 KB | 45.956 % |
| GzipProviderStream_5KBytes | 100 | 1,897.6 us | 22.44 us | 0.86 | 0.02 | 35.1563 | - | - | 583 KB | 46.415 % |
| RecyclableGzipProvider_5KBytes | 100 | 2,342.6 us | 45.13 us | 1.07 | 0.03 | 3.9063 | - | - | 121 KB | 88.879 % |
| RecyclableGzipProviderStream_5KBytes | 100 | 2,182.7 us | 79.15 us | 0.95 | 0.12 | 5.8594 | - | - | 126 KB | 88.420 % |
| | | | | | | | | | | |
| BasicCompressDecompress_5KBytes | 500 | 11,190.1 us | 213.12 us | 1.00 | 0.00 | 328.1250 | - | - | 5,438 KB | - % |
| GzipProvider_5KBytes | 500 | 10,144.5 us | 214.08 us | 0.84 | 0.06 | 171.8750 | - | - | 2,938 KB | 45.973 % |
| GzipProviderStream_5KBytes | 500 | 11,105.7 us | 143.25 us | 0.99 | 0.03 | 171.8750 | - | - | 2,914 KB | 46.414 % |
| RecyclableGzipProvider_5KBytes | 500 | 11,653.7 us | 182.54 us | 1.04 | 0.03 | 31.2500 | 15.6250 | - | 635 KB | 88.323 % |
| RecyclableGzipProviderStream_5KBytes | 500 | 11,433.7 us | 171.46 us | 1.02 | 0.02 | 31.2500 | - | - | 629 KB | 88.433 % |
| | | | | | | | | | | |
| BasicCompressDecompress_5KBytes | 1000 | 20,582.4 us | 422.06 us | 1.00 | 0.00 | 656.2500 | - | - | 10,875 KB | - % |
| GzipProvider_5KBytes | 1000 | 20,492.8 us | 481.45 us | 1.00 | 0.07 | 343.7500 | - | - | 5,875 KB | 45.977 % |
| GzipProviderStream_5KBytes | 1000 | 20,491.4 us | 440.79 us | 1.00 | 0.05 | 343.7500 | - | - | 5,828 KB | 46.641 % |
| RecyclableGzipProvider_5KBytes | 1000 | 18,568.1 us | 42.56 us | 0.94 | 0.05 | 62.5000 | 31.2500 | - | 1,271 KB | 88.313 % |
| RecyclableGzipProviderStream_5KBytes | 1000 | 18,192.9 us | 120.91 us | 0.96 | 0.07 | 62.5000 | - | - | 1,258 KB | 88.432 % |
```

### Conclusion
Memory allocation optimizations can be a huge boon to plain-ole throughput. When you start reaching the scale of thousands of requests,
maybe tens of thousands of requests a second, you start finding these issues fast. Speaking in generality, most devs/managers/product
owners start talking about scaling up immediately. I know, I know, Cloud hardware solves are often cheaper than devs, but costs can
be just as prohibitive and back in the day you couldn't easily throw hardware at your problems. This is basically just leaving free
performance on the table.

There is nothing special about these three guides or code snippets. The only potential secret sauce was taking something mundane like
Compression, something you may have see on StackOverflow answered a thousand times and just not taking the first popularly upvoted
answer. Instead, challenging oneself with some interesting constraints.

In this particular use case, I was working on my library and I wanted it to be as lean and clean as humanly possible so devs
can get more free/out-of-the-box performance. I hope you found it useful expedition regardless!
maybe tens of thousands of requests a second, you start finding these issues quickly. Speaking in generality, most devs/managers/product
owners start talking about scaling up immediately. I know, I know, Cloud hardware solves are often cheaper than devs, but those costs can
be just as prohibitive. Back in the day you couldn't easily throw hardware at your problems. Considering how easy it is to implement, basically
just leaving free performance on the table.

There isn't anything special about these three Compression guides or code snippets. The only potential secret sauce was taking
something mundane like Compression, something you may have see on StackOverflow answered a thousand times and just not taking the first popularly upvoted
answer. Instead, I challenged myself with some interesting constraints.

In my actual use case, I was working on my library and I wanted it to be as lean and clean as humanly possible so devs
can get more free/out-of-the-box performance. I hope you found it useful expedition regardless and helped exposed you to some of the
newer tricks coming out of C#/NetCore land in recent years!
14 changes: 6 additions & 8 deletions src/HouseofCat.Dapper.LegacySqlServer/SqlServerGridReader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -19,20 +19,18 @@ public SqlServerGridReader(SqlConnection connection, GridReader reader)
_reader = reader;
}

public async Task<T> GetAsync<T>() where T : class, new()
public Task<T> GetAsync<T>() where T : class, new()
{
if (!IsReaderConsumed)
{ return await _reader.ReadSingleAsync<T>().ConfigureAwait(false); }
if (IsReaderConsumed) throw new InvalidOperationException();

throw new InvalidOperationException();
return _reader.ReadSingleAsync<T>();
}

public async Task<IEnumerable<T>> GetManyAsync<T>(bool buffered = true) where T : class, new()
public Task<IEnumerable<T>> GetManyAsync<T>(bool buffered = true) where T : class, new()
{
if (!IsReaderConsumed)
{ return await _reader.ReadAsync<T>(buffered).ConfigureAwait(false); }
if (IsReaderConsumed) throw new InvalidOperationException();

throw new InvalidOperationException();
return _reader.ReadAsync<T>(buffered);
}

private bool _disposedValue;
Expand Down
14 changes: 6 additions & 8 deletions src/HouseofCat.Dapper.SqlServer/SqlServerGridReader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -19,20 +19,18 @@ public SqlServerGridReader(SqlConnection connection, GridReader reader)
_reader = reader;
}

public async Task<T> GetAsync<T>() where T : class, new()
public Task<T> GetAsync<T>() where T : class, new()
{
if (!IsReaderConsumed)
{ return await _reader.ReadSingleAsync<T>().ConfigureAwait(false); }
if (IsReaderConsumed) throw new InvalidOperationException();

throw new InvalidOperationException();
return _reader.ReadSingleAsync<T>();
}

public async Task<IEnumerable<T>> GetManyAsync<T>(bool buffered = true) where T : class, new()
public Task<IEnumerable<T>> GetManyAsync<T>(bool buffered = true) where T : class, new()
{
if (!IsReaderConsumed)
{ return await _reader.ReadAsync<T>(buffered).ConfigureAwait(false); }
if (IsReaderConsumed) throw new InvalidOperationException();

throw new InvalidOperationException();
return _reader.ReadAsync<T>(buffered);
}

private bool _disposedValue;
Expand Down
10 changes: 5 additions & 5 deletions src/HouseofCat.Data/DataTransformer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -34,22 +34,22 @@ public DataTransformer(
/// <typeparam name="TOut"></typeparam>
/// <param name="data"></param>
/// <returns></returns>
public async Task<TOut> DeserializeAsync<TOut>(ReadOnlyMemory<byte> data)
public Task<TOut> DeserializeAsync<TOut>(ReadOnlyMemory<byte> data)
{
if (_encryptionProvider != null && _compressionProvider != null)
{
return await DecryptDecompressDeserializeAsync<TOut>(data);
return DecryptDecompressDeserializeAsync<TOut>(data);
}
else if (_encryptionProvider != null)
{
return await DecryptDeserializeAsync<TOut>(data);
return DecryptDeserializeAsync<TOut>(data);
}
else if (_compressionProvider != null)
{
return await DecompressDeserializeAsync<TOut>(data);
return DecompressDeserializeAsync<TOut>(data);
}

return await _serializationProvider.DeserializeAsync<TOut>(data.AsStream());
return _serializationProvider.DeserializeAsync<TOut>(data.AsStream());
}

/// <summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ public async Task<MemoryStream> EncryptAsync(MemoryStream unencryptedStream, boo

if (returnTobuffer)
{ _pool.Return(buffer.Array); }

_pool.Return(encryptedBytes);
_pool.Return(tag);
_pool.Return(nonce);
Expand Down
141 changes: 141 additions & 0 deletions src/HouseofCat.Encryption/AesStreamEncryptionProvider.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
using HouseofCat.Utilities.Errors;
using System;
using System.Buffers;
using System.IO;
using System.Security.Cryptography;

namespace HouseofCat.Encryption
{
public class AesStreamEncryptionProvider : IStreamEncryptionProvider
{
public string Type { get; }

private readonly ReadOnlyMemory<byte> _key;
private readonly ArrayPool<byte> _pool = ArrayPool<byte>.Shared;

private readonly CipherMode _cipherMode;
private readonly PaddingMode _paddingMode;

public AesStreamEncryptionProvider(
ReadOnlyMemory<byte> key,
string hashType,
CipherMode cipherMode = CipherMode.CBC,
PaddingMode paddingMode = PaddingMode.PKCS7)
{
Guard.AgainstEmpty(key, nameof(key));

if (!Constants.Aes.ValidKeySizes.Contains(key.Length)) throw new ArgumentException("Keysize is an invalid length.");
_key = key;

_cipherMode = cipherMode;
_paddingMode = paddingMode;

switch (_key.Length)
{
case 16: Type = $"AES{_cipherMode}_128"; break;
case 24: Type = $"AES{_cipherMode}_192"; break;
case 32: Type = $"AES{_cipherMode}_256"; break;
}

if (!string.IsNullOrWhiteSpace(hashType)) { Type = $"HOC_{hashType}-{Type}"; }
}

/// <summary>
/// Creates a CryptoStream from the input Stream, but with Nonce/IV size and nonce pre-appended to the Stream
/// before creating the CryptoStream. Primarily designed for FileStreams.
/// <para>
/// Byte Structure
/// </para>
/// <para>
/// IV Nonce Length (NL)
/// [ 0 - 3 ]
/// </para>
/// <para>
/// IV Nonce
/// [ 3 - (NL) ]
/// </para>
/// <para>
/// CipherText
/// [ (NL) - n ]
/// </para>
/// </summary>
/// <param name="stream"></param>
/// <returns></returns>
public CryptoStream GetEncryptStream(Stream stream, bool leaveOpen = false)
{
stream.Seek(0, SeekOrigin.Begin);

using AesManaged aes = new AesManaged
{
Key = _key.ToArray(),
Mode = _cipherMode,
Padding = _paddingMode,
};

var nonceSize = aes.BlockSize / 8;
stream.Write(BitConverter.GetBytes(nonceSize));
stream.Write(aes.IV, 0, aes.IV.Length);

return new CryptoStream(
stream,
aes.CreateEncryptor(aes.Key, aes.IV),
CryptoStreamMode.Write,
leaveOpen);
}

/// <summary>
/// Creates a CryptoStream from the input Stream, but with Nonce/IV size and nonce pre-read to the Stream
/// before creating the CryptoStream. Primarily designed for FileStreams.
/// <para>
/// Byte Structure
/// </para>
/// <para>
/// IV Nonce Length (NL)
/// [ 0 - 3 ]
/// </para>
/// <para>
/// IV Nonce
/// [ 3 - (NL) ]
/// </para>
/// <para>
/// CipherText
/// [ (NL) - n ]
/// </para>
/// </summary>
/// <param name="stream"></param>
/// <returns></returns>
public CryptoStream GetDecryptStream(Stream stream, bool leaveOpen = false)
{
stream.Seek(0, SeekOrigin.Begin);

using AesManaged aes = new AesManaged
{
Key = _key.ToArray(),
Mode = _cipherMode,
Padding = _paddingMode,
};

var nonceSizeBytes = _pool.Rent(4);
stream.Read(nonceSizeBytes, 0, 4);

var nonceSize = BitConverter.ToInt32(nonceSizeBytes);
var nonce = _pool.Rent(nonceSize);

stream.Read(nonce, 0, nonceSize);

aes.IV = nonce
.AsSpan()
.Slice(0, nonceSize)
.ToArray();

_pool.Return(nonceSizeBytes);
_pool.Return(nonce);

return new CryptoStream(
stream,
aes.CreateDecryptor(aes.Key, aes.IV),
CryptoStreamMode.Read,
leaveOpen);
}
}
}
Loading

0 comments on commit 9039355

Please sign in to comment.