Exhausted heap space: Out of Memory error with remote-terminated long-running rpc call #458
Comments
|
Debugger is not notified on OOM exceptions because it might OOM itself. Try running your dart server with |
|
The error is happening on a dart client, sorry should have specified. Server is using a go library. let me try make a demo you which can reproduce the issue |
|
You can use the same flag on client as well, sorry I could have been more
specific. It is a flag to dart binary itself.
--
*// If you get an email from me outside of the 9-5 it is not because I'm
always// on or expect an immediate response from you; it is because of work
flexibility
<https://www.inc.com/john-boitnott/how-flexible-hours-can-create-a-better-work-life-balance.html>.//
Evening
and weekend emails are a sign I allocated some regular working hours// for
other things (such as family, gym, friends,...). And I encourage you to//
feel free to do the same.*
|
|
Ah right sorry, I misunderstood. Will try that and report back. Thank you |
|
I've got something, but not sure what I'm looking at.
|
|
This looks like we are receiving some garbage from the server which it then tries to misinterpret as a gRPC message. You could try to patch the package, by adding the following code before if (dataLength == 1752460652) {
print(utf8.decode(chunkData));
}This would print the content that server send back. I guess when server is shutting down it takes grpc handler down before taking down HTTP server and then you receive a normal 404 back or something like that. I can actually reproduce this behavior if I make a normal HTTP/2 server (not gRPC capable) and then try to use gRPC client to talk to it. There is no checking in the I guess we could put something like this in: diff --git a/lib/src/shared/streams.dart b/lib/src/shared/streams.dart
index cfd5869..aa6aff1 100644
--- a/lib/src/shared/streams.dart
+++ b/lib/src/shared/streams.dart
@@ -117,7 +117,10 @@ class _GrpcMessageConversionSink extends ChunkedConversionSink<StreamMessage> {
// TODO(jakobr): Handle duplicate header names correctly.
headers[ascii.decode(header.name)] = ascii.decode(header.value);
}
- // TODO(jakobr): Check :status, go to error mode if not 2xx.
+ final contentType = headers['content-type'];
+ if (contentType == null || !contentType.startsWith('application/grpc')) {
+ throw GrpcError.unimplemented('Received non-gRPC message');
+ }
_out.add(GrpcMetadata(headers));
} |
|
Thanks, that's vert helpful! 1752460652 bytes is ~1.7gb. I'm not sure what a sane default would be though |
Hello,
I've isolated an issue with this library, although I am unable to debug it and find the precise cause. Would appreciate help with this.
The issue is with a long running RPC which, if terminated by the remote (f.e. server killed) causes the following error message
Library: grpc 2.9.0 on flutter mobile.
Repro steps
Set up a Bidirectional or server stream (long-running).
Stream was terminated by peerthrown, otherwise straight to 6.Out of memorythrownExpected result: no out of heap space complaints
Actual result: As above in the intro
Details
I've tried looking through the dart debugger into memory when the heap space exhaustion happens but I cannot find anything relevant.
Tracing the thrown error leads to
client/call.dartfunction_terminate(but line 0 ??). Dart cannot specify the line number and further frames up are not shown, probably due to the call trace size limit or due to dart being unable to locate the frame because of Futures.What is the best way to debug this and find what's causing this?
Would love to help via PR, but need to find the issue first
The text was updated successfully, but these errors were encountered: