ResponseTime is almost doubled with RecyclableMemoryStream.

[anilgupta29Dev]

I am using RecyclableMemoryStream for deserializing a 1.5 GB json file. Even though I was able to reduce the memory usage by almost 40% but the method response time was almost doubled from ~40s to ~80s. Is this expected?
I am using the configuration below.

private static RecyclableMemoryStreamManager GetRecyclableMemoryStreamManager()
{
int blockSize = 1024;
int largeBufferMultiple = 1024 * 1024;
int maxBufferSize = 16 * largeBufferMultiple;
int maximumFreeLargePoolBytes = 3 * maxBufferSize;
int maximumFreeSmallPoolBytes = 100 * blockSize;
RecyclableMemoryStreamManager manager = new RecyclableMemoryStreamManager(
blockSize: blockSize,
largeBufferMultiple: largeBufferMultiple,
maximumBufferSize: maxBufferSize,
maximumSmallPoolFreeBytes: maximumFreeSmallPoolBytes,
maximumLargePoolFreeBytes: maximumFreeLargePoolBytes);

        manager.AggressiveBufferReturn = true;
        return manager;
    }

[benmwatson]

Without more information, it's hard to say what the problem is.

RMS is made for repeated use. Are you deserializing that file once or repeatedly? Are you creating a single manager object and re-using it?

Your block size is very, very small--you would be using so many chained buffers, your efficiency could go down from that alone. I'd probably use something like 128KB or bigger, depending on your typical data sizes.

[anilgupta29Dev]

The file is deserialized multiple times. There are many files in the range of 500MB to 2GB. At a time, 5-6 files will be deserialized.
I am using a static memory stream that provided me with a ~50% gain in memory usage. I will try with a bigger blocker.
I wanted to make sure if response time degradation is expected especially in case of large files?

Here is the code.
using (MemoryStream ms = RecyclableMemoryStream.GetMemoryStream())
{
if (await Func(assetLocation, ms, cancellationToken).ConfigureAwait(false))
{
ms.Position = 0;
using (StreamReader sr = new StreamReader(ms))
{
using (JsonTextReader jr = new JsonTextReader(sr))
{
result = jsonSerializer.Deserialize(jr);
}
}
}
}

public static class RecyclableMemoryStream
{
///

/// The object used for lock.
///

private static readonly object LockObject = new object();

    /// <summary>
    /// The recyclable memory stream manager.
    /// It has to be static to take the benefit of the library.
    /// </summary>
    private static RecyclableMemoryStreamManager recyclableMemoryStreamManager;

    /// <summary>
    /// Gets or sets a value indicating whether to use recyclable memory stream manager.
    /// </summary>
    public static bool UseRecyclableMemoryStreamManager { get; set; }

    /// <summary>
    /// Gets the recyclable memory stream from recyclable memory stream manager if <see cref="UseRecyclableMemoryStreamManager"/> is true, otherwise <see cref="MemoryStream" />.
    /// </summary>
    /// <returns>The Memory stream.</returns>
    public static MemoryStream GetMemoryStream()
    {
        if (UseRecyclableMemoryStreamManager && recyclableMemoryStreamManager == null)
        {
            lock (LockObject)
            {
                recyclableMemoryStreamManager ??= GetRecyclableMemoryStreamManager();
            }
        }

        return recyclableMemoryStreamManager == null ? new MemoryStream() : recyclableMemoryStreamManager.GetStream();
    }

    /// <summary>
    /// Returns the RecyclableMemoryStreamManager instance with default values.
    /// </summary>
    /// <returns><see cref="RecyclableMemoryStreamManager"/>.</returns>
    private static RecyclableMemoryStreamManager GetRecyclableMemoryStreamManager()
    {
        int blockSize = 1024;
        int largeBufferMultiple = 1024 * 1024;
        int maxBufferSize = 16 * largeBufferMultiple;
        int maximumFreeLargePoolBytes = 3 * maxBufferSize;
        int maximumFreeSmallPoolBytes = 100 * blockSize;
        RecyclableMemoryStreamManager manager = new RecyclableMemoryStreamManager(
            blockSize: blockSize,
            largeBufferMultiple: largeBufferMultiple,
            maximumBufferSize: maxBufferSize,
            maximumSmallPoolFreeBytes: maximumFreeSmallPoolBytes,
            maximumLargePoolFreeBytes: maximumFreeLargePoolBytes);

        manager.AggressiveBufferReturn = true;
        return manager;
    }
}

[benmwatson]

There should be no inherent reason why using RMS is slower than using MS.

If using a larger block size doesn't work, you will need to do some profiling to see where the slowdown is coming.

[benmwatson]

@anilgupta29Dev Do you have any updates on your end to this issue?

[anilgupta29Dev]

We can close the issue. I was able to get similar response time with changed settings.