'Unable to expand length of this stream beyond its capacity.' when using DefaultLambdaJsonSerializer #692
Description
Description
On migrating from Amazon.Lambda.Serialization.Json.JsonSerializer to Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer there is a significant increase in the size of the serialised response.
I'm generating JSON and returning it in the body of APIGatewayProxyResponse. Previously the quotes in the JSON where encoded as \" ( 2 characters), now they are encoded as \u0022 ( 6 characters). I could previously return more than 5MB of JSON before hitting the Lambda Response Payload limitation of 6MB, now I can only return around 3.5MB of JSON
The error I get is:
Error converting the response object of type Amazon.Lambda.APIGatewayEvents.APIGatewayProxyResponse from the Lambda function to JSON: Unable to expand length of this stream beyond its capacity.
I'm raising this as a bug, as it's not clear that there is a significant impact from moving to the new serialiser and there doesn't seem to be an easy way to override the behaviour to make it produce JSON in the way it used to be produced.
Reproduction Steps
Sample code showing the difference:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Mime;
using System.Text.Json;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using Amazon.Lambda.Serialization.SystemTextJson;
using Microsoft.Net.Http.Headers;
namespace testserialiser
{
class Program
{
static void Main(string[] args)
{
var o = new
{
a = "test",
b = 27,
c = new
{
a = "foobar"
}
};
string body = System.Text.Json.JsonSerializer.Serialize(o);
int qc = body.Count(x => x == '\"');
Console.WriteLine($"Source:{body} Length:{body.Length}, QuoteCount:{qc}");
//Source:{"a":"test","b":27,"c":{"a":"foobar"}} Length:38, QuoteCount:12
var response = new APIGatewayProxyResponse
{
StatusCode = (int)HttpStatusCode.OK,
Headers = new Dictionary<string, string> { { HeaderNames.ContentType, MediaTypeNames.Application.Json } },
Body = body
};
//Use DefaultLambdaJsonSerializer
var ms = new MemoryStream(new byte[6 * 1000 * 1000]);
ILambdaSerializer ser = new DefaultLambdaJsonSerializer();
ser.Serialize(response, ms);
LogMS(ms);
//Actual:{"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\u0022a\u0022:\u0022test\u0022,\u0022b\u0022:27,\u0022c\u0022:{\u0022a\u0022:\u0022foobar\u0022}}","isBase64Encoded":false} Length:196
//Use DefaultLambdaJsonSerializer, attempt to override JsonEscaping
ms.Position = 0;
JsonSerializerOptions options = null;
ser = new DefaultLambdaJsonSerializer(o =>
{
options = o;
o.Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping;
o.WriteIndented = false;
});
ser.Serialize(response, ms);
LogMS(ms);
//Actual:{"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\u0022a\u0022:\u0022test\u0022,\u0022b\u0022:27,\u0022c\u0022:{\u0022a\u0022:\u0022foobar\u0022}}","isBase64Encoded":false} Length:196
//Plain Serialize using same options as DefaultLambdaJsonSerializer
var js = JsonSerializer.Serialize(response, options);
Console.WriteLine($"Expected:{js} Length:{js.Length}");
//Expected:{"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\"a\":\"test\",\"b\":27,\"c\":{\"a\":\"foobar\"}}","isBase64Encoded":false} Length:148
//Serialize using same options as DefaultLambdaJsonSerializer and Utf8writer
ms.Position = 0;
using (var writer = new Utf8JsonWriter(ms))
{
JsonSerializer.Serialize(writer, response, options);
}
LogMS(ms);
//Actual:{"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\u0022a\u0022:\u0022test\u0022,\u0022b\u0022:27,\u0022c\u0022:{\u0022a\u0022:\u0022foobar\u0022}}","isBase64Encoded":false} Length:196
//Serialize using old JsonSerializer
ms.Position = 0;
ser = new Amazon.Lambda.Serialization.Json.JsonSerializer();
ser.Serialize(response, ms);
LogMS(ms);
//Actual:{"statusCode":200,"headers":{"Content-Type":"application/json"},"multiValueHeaders":null,"body":"{\"a\":\"test\",\"b\":27,\"c\":{\"a\":\"foobar\"}}","isBase64Encoded":false} Length:173
}
private static void LogMS(MemoryStream ms)
{
long len = ms.Position;
ms.SetLength(len);
ms.Position = 0;
var sr = new StreamReader(ms);
var j = sr.ReadToEnd();
Console.WriteLine($"Actual:{j} Length:{j.Length}");
ms.Position = 0;
}
}
}Logs
Error converting the response object of type Amazon.Lambda.APIGatewayEvents.APIGatewayProxyResponse from the Lambda function to JSON: Unable to expand length of this stream beyond its capacity.: JsonSerializerException
at Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer.Serialize[T](T response, Stream responseStream)at System.IO.UnmanagedMemoryStream.WriteCore(ReadOnlySpan
1 buffer) at System.IO.UnmanagedMemoryStream.Write(ReadOnlySpan1 buffer)
at System.Text.Json.Utf8JsonWriter.Flush()
at System.Text.Json.JsonSerializer.WriteCore(Utf8JsonWriter writer, Object value, Type type, JsonSerializerOptions options)
at System.Text.Json.JsonSerializer.WriteValueCore(Utf8JsonWriter writer, Object value, Type type, JsonSerializerOptions options)
at System.Text.Json.JsonSerializer.Serialize[TValue](Utf8JsonWriter writer, TValue value, JsonSerializerOptions options)
at Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer.Serialize[T](T response, Stream responseStream)
Environment
- Build Version: 2.0.1
- OS Info: Ubuntu
- Build Environment: VSCode
- Targeted .NET Platform:dotnetcore3.1
Resolution
This is a 🐛 bug-report