`gRPCServiceCallException: 0` when trying to connect with Python grpc server

[manuelbb-upb]

I am trying to make a request to a server that is started using the grpcio Python package.
The server starts an insecure channel at "localhost:8888".
I can successfully make requests using client code in Python.

When I run gRPCClient.generate on my .proto file and try to do the same in Julia I get the following exception:

gRPCServiceCallException: 0, HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

I guess it happens around here

Reproduce

1. Obtain the python example code from [this repo].(https://github.com/grpc/grpc/tree/v1.38.0/examples/python/route_guide)
2. In downloaded folder grpc/examples/protos call gRPCClient.generate on route_guide.proto.
3. Using Python (with grpcio installed) start grpc/examples/python/route_guide/route_guide_server.py
4. In the folder with the outputs from step 2 do

using .RouteguideClients: Point, GetFeature

p = Point(;longitude = 1, latitude = 1)
client = RouteGuideBlockingClient("localhost:50051")

feature, status = GetFeature( client, p )
gRPCCheck(status)

This gives me

error: loaderror: grpcservicecallexception: 0, http/2 stream 0 was not closed cleanly: internal_error (err 2)
stacktrace:
  [1] grpccheck(status::grpcstatus; throw_error::bool)
    @ grpcclient ~/.julia/packages/grpcclient/v6sar/src/grpc.jl:61
  [2] grpccheck(status_future::task; throw_error::bool)
    @ grpcclient ~/.julia/packages/grpcclient/v6sar/src/grpc.jl:57
  [3] grpccheck
    @ ~/.julia/packages/grpcclient/v6sar/src/grpc.jl:57 [inlined]
  [4] call_method(channel::grpcchannel, service::protobuf.servicedescriptor, method::protobuf.methoddescriptor, controller::grpccontroller, input::channel{point}, #unused#::type{main.routeguideclients.routeguide.feature})
    @ grpcclient ~/.julia/packages/grpcclient/v6sar/src/grpc.jl:186
  [5] call_method(channel::grpcchannel, service::protobuf.servicedescriptor, method::protobuf.methoddescriptor, controller::grpccontroller, input::channel{point})
    @ grpcclient ~/.julia/packages/grpcclient/v6sar/src/grpc.jl:177
  [6] call_method
    @ ~/.julia/packages/grpcclient/v6sar/src/grpc.jl:175 [inlined]
  [7] call_method(stub::protobuf.protoserviceblockingstub, meth::protobuf.methoddescriptor, controller::grpccontroller, request::point)
    @ protobuf ~/.julia/packages/protobuf/tyedo/src/svc.jl:100
  [8] getfeature
    @ ~/project_dir/grpc/examples/julia/route_guide_pb.jl:236 [inlined]
  [9] getfeature(client::routeguideblockingclient, inp::point)
    @ main.routeguideclients ~/project_dir/grpc/examples/julia/routeguideclients.jl:49
 [10] top-level scope
    @ ~/project_dir/grpc/examples/julia/test_client.jl:11
in expression starting at ~/project_dir/grpc/examples/julia/test_client.jl:11

caused by: InvalidStateException("Channel is closed.", :closed)
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:710
  [2] wait
    @ ./task.jl:769 [inlined]
  [3] wait(c::Base.GenericCondition{ReentrantLock})
    @ Base ./condition.jl:106
  [4] take_unbuffered(c::Channel{Main.RouteguideClients.routeguide.Feature})
    @ Base ./channels.jl:405
  [5] take!
    @ ./channels.jl:383 [inlined]
  [6] call_method(channel::gRPCChannel, service::ProtoBuf.ServiceDescriptor, method::ProtoBuf.MethodDescriptor, controller::gRPCController, input::Channel{Point}, #unused#::Type{Main.RouteguideClients.routeguide.Feature})
    @ gRPCClient ~/.julia/packages/gRPCClient/V6sAr/src/grpc.jl:184
  [7] call_method(channel::gRPCChannel, service::ProtoBuf.ServiceDescriptor, method::ProtoBuf.MethodDescriptor, controller::gRPCController, input::Channel{Point})
    @ gRPCClient ~/.julia/packages/gRPCClient/V6sAr/src/grpc.jl:177
  [8] call_method
    @ ~/.julia/packages/gRPCClient/V6sAr/src/grpc.jl:175 [inlined]
  [9] call_method(stub::ProtoBuf.ProtoServiceBlockingStub, meth::ProtoBuf.MethodDescriptor, controller::gRPCController, request::Point)
    @ ProtoBuf ~/.julia/packages/ProtoBuf/TYEdo/src/svc.jl:100
 [10] GetFeature
    @ ~/project_dir/grpc/examples/julia/route_guide_pb.jl:236 [inlined]
 [11] GetFeature(client::RouteGuideBlockingClient, inp::Point)
    @ Main.RouteguideClients ~/project_dir/grpc/examples/julia/RouteguideClients.jl:49
 [12] top-level scope
    @ ~/project_dir/grpc/examples/julia/test_client.jl:11

Output of ]st is:

Status `~/.julia/environments/v1.6/Project.toml`
  [6e4b80f9] BenchmarkTools v1.1.0
  [d96e819e] Parameters v0.12.2
  [3349acd9] ProtoBuf v0.11.2
  [295af30f] Revise v3.1.17
  [aaca4a50] gRPCClient v0.1.1

[tanmaykm]

Does it make a difference if you specify the protocol explicitly while creating the client? E.g. RouteGuideBlockingClient("http://localhost:50051") or RouteGuideBlockingClient("https://localhost:50051")?

And if the python server is serving with a self signed certificate, Julia client will also need the server certificate and CA certificate configured appropriately.

[tanmaykm]

Actually debugging a bit further, it does not look like a connection setup issue. Rather something to do with the protocol? I see this in the debug messages of the python server. Any clues?

I0802 21:36:58.586854766  793765 parsing.cc:665]             parsing initial_metadata
I0802 21:36:58.586873963  793765 parsing.cc:414]             HTTP:1:HDR:SVR: :method: 50 4f 53 54 'POST'
I0802 21:36:58.586904098  793765 parsing.cc:414]             HTTP:1:HDR:SVR: :path: 2f 72 6f 75 74 65 67 75 69 64 65 2e 52 6f 75 74 65 47 75 69 64 65 2f 47 65 74 46 65 61 74 75 72 65 '/routeguide.RouteGuide/GetFeature'
I0802 21:36:58.586922520  793765 parsing.cc:414]             HTTP:1:HDR:SVR: :scheme: 68 74 74 70 'http'
I0802 21:36:58.586944239  793765 parsing.cc:414]             HTTP:1:HDR:SVR: :authority: 6c 6f 63 61 6c 68 6f 73 74 3a 35 30 30 35 31 'localhost:50051'
I0802 21:36:58.586963611  793765 parsing.cc:414]             HTTP:1:HDR:SVR: accept: 2a 2f 2a '*/*'
I0802 21:36:58.586987227  793765 parsing.cc:414]             HTTP:1:HDR:SVR: user-agent: 63 75 72 6c 2f 37 2e 37 30 2e 30 20 6a 75 6c 69 61 2f 31 2e 35 'curl/7.70.0 julia/1.5'
I0802 21:36:58.587011170  793765 parsing.cc:414]             HTTP:1:HDR:SVR: content-type: 61 70 70 6c 69 63 61 74 69 6f 6e 2f 67 72 70 63 2b 70 72 6f 74 6f 'application/grpc+proto'
I0802 21:36:58.587032350  793765 chttp2_transport.cc:812]    W:0x7fa9ec008090 SERVER [ipv4:127.0.0.1:51726] state IDLE -> WRITING [BDP_PING]
I0802 21:36:58.587052249  793765 tcp_posix.cc:544]           TCP:0x7fa9ec0076e0 notify_on_read
I0802 21:36:58.587074530  793765 chttp2_transport.cc:1646]   perform_stream_op[s=0x7fa9ec0142a0]:  RECV_INITIAL_METADATA
I0802 21:36:58.587098574  793765 chttp2_transport.cc:1386]   perform_stream_op_locked:  RECV_INITIAL_METADATA; on_complete = (nil)
I0802 21:36:58.587190551  793765 chttp2_transport.cc:1646]   perform_stream_op[s=0x7fa9ec0142a0]:  CANCEL:{"created":"@1627920418.587124954","description":"Missing :authority or :path","file":"src/core/lib/surface/server.cc","file_line":1420,"referenced_errors":[{"created":"@1627920418.587120958","description":"Failed processing incoming headers","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":123,"referenced_errors":[{"created":"@1627920418.587117762","description":"Missing header","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":204,"key":"te"}]}]}
I0802 21:36:58.587224639  793765 chttp2_transport.cc:1386]   perform_stream_op_locked:  CANCEL:{"created":"@1627920418.587124954","description":"Missing :authority or :path","file":"src/core/lib/surface/server.cc","file_line":1420,"referenced_errors":[{"created":"@1627920418.587120958","description":"Failed processing incoming headers","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":123,"referenced_errors":[{"created":"@1627920418.587117762","description":"Missing header","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":204,"key":"te"}]}]}; on_complete = 0x7fa9ec00cdf0

[tanmaykm]

Actually, looks like the python gRPC server insists on being presented with a te header.

Sending out a PR in a bit.

[tanmaykm]

With the changes in #17, the protocol exchange with a Python server looks like:

julia> include("RouteguideClients/RouteguideClients.jl")
Main.RouteguideClients

julia> using .RouteguideClients: Point, GetFeature

julia> p = Point(;latitude=409146138, longitude=-746188906)
Point(ProtoMeta for Point
ProtoBuf.ProtoMetaAttribs[ProtoBuf.ProtoMetaAttribs(1, :latitude, :int32, Int32, 0, false, Int32[0], nothing), ProtoBuf.ProtoMetaAttribs(2, :longitude, :int32, Int32, 0, false, Int32[0], nothing)]
, Dict{Symbol,Any}(:longitude => -746188906,:latitude => 409146138), Set{Symbol}())

julia> client = RouteguideClients.RouteGuideBlockingClient("http://127.0.0.1:50051"; verbose=true)
RouteGuideBlockingClient(http://127.0.0.1:50051)

julia> res = GetFeature(client, p)
* Couldn't find host 127.0.0.1 in the .netrc file; using defaults
*   Trying 127.0.0.1:50051...
* Connected to 127.0.0.1 (127.0.0.1) port 50051 (#0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x288a610)
> POST /routeguide.RouteGuide/GetFeature HTTP/2
Host: 127.0.0.1:50051
accept: */*
user-agent: curl/7.70.0 julia/1.5
content-type: application/grpc+proto
te: trailers

* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200 
< content-type: application/grpc
< grpc-accept-encoding: identity,deflate,gzip
< accept-encoding: identity,gzip
< 
* Connection #0 to host 127.0.0.1 left intact
(Main.RouteguideClients.routeguide.Feature(ProtoMeta for Main.RouteguideClients.routeguide.Feature
ProtoBuf.ProtoMetaAttribs[ProtoBuf.ProtoMetaAttribs(1, :name, :string, AbstractString, 0, false, [""], nothing), ProtoBuf.ProtoMetaAttribs(2, :location, :obj, Point, 0, false, Any[], ProtoMeta for Point
ProtoBuf.ProtoMetaAttribs[ProtoBuf.ProtoMetaAttribs(1, :latitude, :int32, Int32, 0, false, Int32[0], nothing), ProtoBuf.ProtoMetaAttribs(2, :longitude, :int32, Int32, 0, false, Int32[0], nothing)]
)]
, Dict{Symbol,Any}(:location => Point(ProtoMeta for Point
ProtoBuf.ProtoMetaAttribs[ProtoBuf.ProtoMetaAttribs(1, :latitude, :int32, Int32, 0, false, Int32[0], nothing), ProtoBuf.ProtoMetaAttribs(2, :longitude, :int32, Int32, 0, false, Int32[0], nothing)]
, Dict{Symbol,Any}(:longitude => -746188906,:latitude => 409146138), Set{Symbol}()),:name => "Berkshire Valley Management Area Trail, Jefferson, NJ, USA"), Set{Symbol}()), Task (done) @0x00007fea66a76e60)

[manuelbb-upb]

Thank you very much for addressing the issue! I am currently on vacation, but this will make my life so much easier once I am back to work!