dart server failed in interop tests

[markdroth]

https://source.cloud.google.com/results/invocations/3224a370-b637-4688-af9d-f04d238b2534/log

2019-04-01 15:22:20,741 FAILED: Some tests failed
2019-04-01 15:22:20,817 Server listening on port 8080...
Socket error: HandshakeException: Connection terminated during handshake
Socket error: HandshakeException: Connection terminated during handshake
Socket error: HandshakeException: Connection terminated during handshake
Socket error: HandshakeException: Connection terminated during handshake
Socket error: HandshakeException: Connection terminated during handshake
Socket error: HandshakeException: Connection terminated during handshake
Unhandled exception:
NoSuchMethodError: The method 'remove' was called on null.
Receiver: null
Tried calling: remove(":status")
#0      Object.noSuchMethod (dart:core/runtime/libobject_patch.dart:48:5)
#1      ServerHandler.sendTrailers (package:grpc/src/server/handler.dart:288:23)
#2      ServerHandler._sendError (package:grpc/src/server/handler.dart:352:5)
#3      ServerHandler._onResponse (package:grpc/src/server/handler.dart:241:7)
#4      _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10)
#5      _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11)
#6      _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7)
#7      _SyncStreamController._sendData (dart:async/stream_controller.dart:763:19)
#8      _StreamController._add (dart:async/stream_controller.dart:639:7)
#9      new Stream.fromFuture.<anonymous closure> (dart:async/stream.dart:107:18)
#10     _RootZone.runUnary (dart:async/zone.dart:1379:54)
#11     _FutureListener.handleValue (dart:async/future_impl.dart:129:18)
#12     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:642:45)
#13     Future._propagateToListeners (dart:async/future_impl.dart:671:32)
#14     Future._complete (dart:async/future_impl.dart:476:7)
#15     _SyncCompleter.complete (dart:async/future_impl.dart:51:12)
#16     _AsyncAwaitCompleter.complete (dart:async/runtime/libasync_patch.dart:28:18)
#17     _completeOnAsyncReturn (dart:async/runtime/libasync_patch.dart:295:13)
#18     TestService.streamingInputCall (file:///var/local/git/grpc-dart/interop/bin/server.dart)
<asynchronous suspension>
#19     ServiceMethod.handle (package:grpc/src/server/service.dart:61:20)
#20     ServerHandler._startStreamingRequest (package:grpc/src/server/handler.dart:155:30)
#21     ServerHandler._onDataIdle (package:grpc/src/server/handler.dart:132:5)
<asynchronous suspension>
#22     _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10)
#23     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11)
#24     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7)
#25     _SinkTransformerStreamSubscription._add (dart:async/stream_transformers.dart:68:11)
#26     _EventSinkWrapper.add (dart:async/stream_transformers.dart:15:11)
#27     grpcDecompressor.<anonymous closure> (package:grpc/src/shared/streams.dart:54:12)
#28     _HandlerEventSink.add (dart:async/stream_transformers.dart:236:7)
#29     _SinkTransformerStreamSubscription._handleData (dart:async/stream_transformers.dart:120:24)
#30     _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10)
#31     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11)
#32     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7)
#33     _SinkTransformerStreamSubscription._add (dart:async/stream_transformers.dart:68:11)
#34     _EventSinkWrapper.add (dart:async/stream_transformers.dart:15:11)
#35     _GrpcMessageConversionSink._addHeaders (package:grpc/src/shared/streams.dart:163:10)
#36     _GrpcMessageConversionSink.add (package:grpc/src/shared/streams.dart:171:7)
#37     _ConverterStreamEventSink.add (dart:convert/chunked_conversion.dart:86:18)
#38     _SinkTransformerStreamSubscription._handleData (dart:async/stream_transformers.dart:120:24)
#39     _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10)
#40     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11)
#41     _DelayedData.perform (dart:async/stream_impl.dart:591:14)
#42     _StreamImplEvents.handleNext (dart:async/stream_impl.dart:707:11)
#43     _PendingEvents.schedule.<anonymous closure> (dart:async/stream_impl.dart:667:7)
#44     _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
#45     _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
#46     _Timer._runTimers (dart:isolate/runtime/libtimer_impl.dart:391:30)
#47     _Timer._handleMessage (dart:isolate/runtime/libtimer_impl.dart:416:5)
#48     _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:171:12)
2019-04-01 15:22:20,818 FAILED: interop_server_dart [ret=255, pid=1868, time=389.4sec]
Server "dart" has exited prematurely.

[jtattermusch]

@jakobr-google @mit-mit @sigurdm can you take a look?

[sigurdm]

That looks like a bug (invariant violation).
What code is being executed?
(How) can I reproduce this locally?

[sigurdm]

Ping!
I don't have much insight into the interop test setup. How can I run the same code?

[jtattermusch]

@sigurdm you can use the run_interop_tests.py script that will help you run the interop scenarios.

https://github.com/grpc/grpc/tree/master/tools/run_tests#interop-tests-run_interop_testspy

E.g. you can try the docker-based scenario (which is the most similar to the test our CI actually runs)

# run java client against docker server, using docker containers
$ tools/run_tests/run_interop_tests.py -l java -s dart --use_docker

Alternatively, you can just run the dart interop server manually and run the java interop client manually against it. The cmdline args should conform to https://github.com/grpc/grpc/blob/master/doc/interop-test-descriptions.md

[jtattermusch]

It looks like the first interop scenario that failed was actually dart client against dart server:
FAILED: cloud_to_cloud:dart:dart_server:unimplemented_service:tls [ret=255, pid=16102, time=6.4sec] (but that might be just a coincidence)

[markdroth]

Seeing dart server failures again:

https://source.cloud.google.com/results/invocations/984e155f-3f88-4d40-87df-6d801690fd08/targets/github%2Fgrpc%2Finterop_test/tests;group=tests;test=cloud_to_cloud:java:dart_server:client_streaming:tls;row=1

[jtattermusch]

dupe of #21755 (I'll try to provide more info there).