Adjusted frame length exceeds 1300000000

[dnickless]

We encounter errors in production using Akka.NET v1.3.15 and DotNetty v0.6.0 running on Windows.

> 2020-02-20 23:12:46.5251		Error	Calculation	[ERROR][2/20/2020 7:12:46 PM][Thread 0030][Akka.Remote.Transport.DotNetty.TcpServerHandler] Error caught channel [[::ffff:10.163.456.123]:56060->[::ffff:10.163.456.11]:55612](Id=56684b4f)2020-02-20 23:12:46.5251		Error	Calculation	[ERROR][2/20/2020 7:12:46 PM][Thread 0030][Akka.Remote.Transport.DotNetty.TcpServerHandler] Error caught channel [[::ffff:10.163.456.123]:56060->[::ffff:10.163.456.11]:55612](Id=56684b4f)
> Cause: DotNetty.Codecs.TooLongFrameException: Adjusted frame length exceeds 1300000000: 1347175756 - discarded
>    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Fail(Int64 frameLength)
>    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.FailIfNecessary(Boolean firstDetectionOfTooLongFrame)
>    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input)
>    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input, List`1 output)
>    at DotNetty.Codecs.ByteToMessageDecoder.CallDecode(IChannelHandlerContext context, IByteBuffer input, List`1 output)
>    at DotNetty.Codecs.ByteToMessageDecoder.ChannelRead(IChannelHandlerContext context, Object message)
>    at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelRead(Object msg)

I was hoping those issues would be gone after #3327 but it would appear like they are not. Any ideas?

[Aaronontheweb]

@dnickless so quick question - are you messages genuinely pretty large? If you enable akka.remote.log-frame-size-exceeding = on do you get warning messages? Because if that's the case, then this isn't a bug - your messages are huge.

[dnickless]

Hi @Aaronontheweb , this is Daniel - we met in Zurich in December. You should know our system pretty well. ;) Messages are not really large (not small but not insane)... Problems generally appear to mainly happen during some kind of stress situation (e.g. machine low on RAM --> swapping --> perf. degradation) from what I can tell.

[Aaronontheweb]

Well well! Hi Daniel! The other possibility is that you could be getting nicked by this bug: #3879

That's caused by a problem with how we use DotNetty reference counting and I'm going to fix it by simply not pooling bytes in that particular area, since it seems to be rather error-prone (we can't tell what DotNetty is doing with referencing counting internally all that easily.)

But I would recommend turning that setting on and see if you get warnings about large message sizes. I'll add another comment to here once #4212 gets merged on how to add compression to outbound and inbound messages just in case you're not already doing that.

[Aaronontheweb]

1300000000 bytes does seem a bit absurd (I'm confident Akka.Remote would time out trying to receive and deserialize an object that large) so I'm leaning towards the DotNetty headers being framed incorrectly.

[tometchy]

@Aaronontheweb I've read your answer:

dnickless so quick question - are you messages genuinely pretty large? If you enable akka.remote.log-frame-size-exceeding = on do you get warning messages? Because if that's the case, then this isn't a bug - your messages are huge.

I set akka.remote.log-frame-size-exceeding = on (Akka version 1.4.8) but got exception at system creation:

Error while creating actor instance of type Akka.Remote.EndpointWriter with 9 args: (<null>,akka.tcp://Q@localhost:54192,akka.tcp://Q@127.0.0.1:4053,<null>,Akka.Remote.Transport.AkkaProtocolTransport,Akka.Remote.RemoteSettings,Akka.Remote.Transport.AkkaPduProtobuffCodec,System.Collections.Concurrent.ConcurrentDictionary`2[Akka.Remote.EndpointManager+Link,Akka.Remote.EndpointManager+ResendState],[akka://Q/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FQ%40127.0.0.1%3A4053-1#1175502508])] | EXCEPTION OCCURRED:Akka.Actor.ActorInitializationException Exception during creation Void Create(System.Exception)    at Akka.Actor.ActorCell.Create(Exception failure)
   at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)
TypeLoadException Error while creating actor instance of type Akka.Remote.EndpointWriter with 9 args: (<null>,akka.tcp://Q@localhost:54192,akka.tcp://Q@127.0.0.1:4053,<null>,Akka.Remote.Transport.AkkaProtocolTransport,Akka.Remote.RemoteSettings,Akka.Remote.Transport.AkkaPduProtobuffCodec,System.Collections.Concurrent.ConcurrentDictionary`2[Akka.Remote.EndpointManager+Link,Akka.Remote.EndpointManager+ResendState],[akka://Q/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FQ%40127.0.0.1%3A4053-1#1175502508]) Akka.Actor.ActorBase NewActor()    at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass117_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.Create(Exception failure)
TargetInvocationException Exception has been thrown by the target of an invocation. System.Object InvokeMethod(System.Object, System.Object[], System.Signature, Boolean, Boolean)    at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, Object[] args)
   at Akka.Actor.Props.ActivatorProducer.Produce()
   at Akka.Actor.Props.NewActor()
FormatException Input string was not in a correct format. Void StringToNumber(System.ReadOnlySpan`1[System.Char], System.Globalization.NumberStyles, NumberBuffer ByRef, System.Globalization.NumberFormatInfo, Boolean)    at System.Number.StringToNumber(ReadOnlySpan`1 str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt64(ReadOnlySpan`1 value, NumberStyles options, NumberFormatInfo numfmt)
   at System.Int64.Parse(String s)
   at Akka.Configuration.Hocon.HoconValue.GetByteSize()
   at Akka.Configuration.Config.GetByteSize(String path, Nullable`1 def)
   at Akka.Remote.RemoteMetricsOn..ctor(ExtendedActorSystem system)
   at Akka.Remote.RemoteMetricsExtension.CreateExtension(ExtendedActorSystem system)
   at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.CreateExtension(ExtendedActorSystem system)
   at Akka.Actor.Internal.ActorSystemImpl.<>c__DisplayClass56_0.<RegisterExtension>b__1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension(Type extensionType, Object& extension)
   at Akka.Actor.Internal.ActorSystemImpl.GetExtension(IExtensionId extensionId)
   at Akka.Actor.ExtensionIdProvider`1.Get(ActorSystem system)
   at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.Get(ActorSystem system)
   at Akka.Actor.Internal.ActorSystemImpl.RegisterExtension(IExtensionId extension)
   at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
   at Akka.Remote.RemoteMetricsExtension.Create(ExtendedActorSystem system)
   at Akka.Remote.EndpointWriter..ctor(AkkaProtocolHandle handleOrActive, Address localAddress, Address remoteAddress, Nullable`1 refuseUid, AkkaProtocolTransport transport, RemoteSettings settings, AkkaPduCodec codec, ConcurrentDictionary`2 receiveBuffers, IActorRef reliableDeliverySupervisor)

I see that the reason is in incorrect config value: FormatException Input string was not in a correct format. Void StringToNumber and because this was the only change I made, I know that it is guilty.

So I checked the docs (https://getakka.net/articles/configuration/akka.remote.html) and it tells that:

By default this feature is turned off. Activate it by setting the property to a value in bytes, such as 1000b

So in my view it looks like your suggestion to set it to on value is incorrect and your answer should be edited, for others not to waste time trying to set it and figuring out why it doesn't work.

If I'm wrong please correct me what I did wrong :)