bug: rpc error: code = Internal desc = server closed the stream without sending trailers

[gitcome]

Current Behavior

apisix: apache/apisix:2.15.0-alpine
plugin-runner: apache/apisix-go-plugin-runner v0.4.0

run apisix-go-plugin-runner ,in func RequestFilter, WriteHeader then return, grpc client will report an error(code = Internal desc = server closed the stream without sending trailers)

I studied for a day and didn't know how to solve the problem. can anybody help solve the problem? thanks a lot!

Expected Behavior

grpc client not report error,and can get trailers

Error Logs

rpc error: code = Internal desc = server closed the stream without sending trailers

Steps to Reproduce

apisix: apache/apisix:2.15.0-alpine
plugin-runner: apache/apisix-go-plugin-runner v0.4.0

1. modify code in apisix-go-plugin-runner-master\cmd\go-runner\plugins\say.go, as follows:

func (p *Say) RequestFilter(conf interface{}, w http.ResponseWriter, r pkgHTTP.Request) {
	w.Header().Set("Content-Type", r.Header().Get("Content-Type"))
	num := rand.Int()
	if num%2 == 1 {
		w.WriteHeader(http.StatusOK)
		w.Header().Set("mykey", "test")
		w.Header().Set("Grpc-Status", "0")
		w.Header().Set("Grpc-Message", "ok")
		w.Header().Set("Grpc-Status-Details-Bin", "")
		return
	}
}

run a grpc server in 192.168.56.19:8301
2. then run apisix, config upstream in APISIX Dashboard，as follows:

{
  "nodes": [
    {
      "host": "192.168.56.19",
      "port": 8301,
      "weight": 1
    }
  ],
  "timeout": {
    "connect": 6,
    "send": 6,
    "read": 6
  },
  "type": "roundrobin",
  "scheme": "grpc",
  "pass_host": "pass",
  "name": "testa",
  "keepalive_pool": {
    "idle_timeout": 60,
    "requests": 1000,
    "size": 320
  }
}

3. then add a route config, as follows:

{
  "uri": "/routeguide.RouteGuide/GetFeature",
  "name": "testa1",
  "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
  ],
  "plugins": {
    "ext-plugin-post-req": {
      "conf": [
        {
          "name": "say",
          "value": "{\"body\":\"123\"}"
        }
      ],
      "disable": false
    }
  },
  "upstream_id": "425744995673178819",
  "status": 1
}

4. then use client grpc call apisix, but have an error (rpc error: code = Internal desc = server closed the stream without sending trailers)，as follows:

2022/09/21 19:45:47.860 [I] [main.go:59]  header: map[content-type:[application/grpc] date:[Wed, 21 Sep 2022 11:45:47 GMT] mykey:[test] server:[APISIX/2.15.0]]
2022/09/21 19:45:47.860 [I] [main.go:60]  trailer: map[]
2022/09/21 19:45:47.861 [E] [main.go:62]  rpc error: code = Internal desc = server closed the stream without sending trailers

Environment

os: win10
apisix: apache/apisix:2.15.0-alpine (use apisix-docker\example\docker-compose.yml docker-compose up)
plugin-runner: apache/apisix-go-plugin-runner v0.4.0

[soulbird]

If you want to use the grpc client to access APISIX, you need to enable the grpc-transcode plugin.

"scheme": "grpc" in upstream configures the protocol for accessing upstream to be grpc.

[gitcome]

If you want to use the grpc client to access APISIX, you need to enable the grpc-transcode plugin.

"scheme": "grpc" in upstream configures the protocol for accessing upstream to be grpc.

I've read this grpc-transcode.md, but it doesn't seem to be relevant to this problem.
Can you contact me via WeChat 'program_no_error'? thanks a lot!

[gitcome]

If you want to use the grpc client to access APISIX, you need to enable the grpc-transcode plugin.

"scheme": "grpc" in upstream configures the protocol for accessing upstream to be grpc.

my call chain like this:

                                                           1.[RequestFilter return] --?--> GrpcClient (report error)
GrpcClient --grpc--> Apisix ----> Go-plugin-runner(ext-plugin-post-req)  
                                                           2.[RequestFilter continue] ----> GrpcServer(upstream) ----> Apisix ----> GrpcClient(ok)

in Go-plugin-runner, func RequestFilter like this:

func (p *Say) RequestFilter(conf interface{}, w http.ResponseWriter, r pkgHTTP.Request) {
	w.Header().Set("Content-Type", r.Header().Get("Content-Type"))
	num := rand.Int()
	if num%2 == 1 {
		w.WriteHeader(http.StatusOK)
		w.Header().Set("mykey", "test")
		w.Header().Set("Grpc-Status", "0")
		w.Header().Set("Grpc-Message", "ok")
		w.Header().Set("Grpc-Status-Details-Bin", "")
		return   // here retrun to GrpcClient, see above chain 1, client report "server closed the stream without sending trailers"
	}
        return // here continue to GrpcServer(upstream), then return to GrpcClient, see above chain 2, ok
}

above not use grpc-transcode，grpc-transcode not suitable for our scene。Please have a look and help， thanks a lot!

[soulbird]

Regarding this step GrpcClient --grpc--> Apisix, how did you do it? What tool or command was used ?

[gitcome]

Regarding this step GrpcClient --grpc--> Apisix, how did you do it? What tool or command was used ?

client use grpc protocol request Apisix. code as follows:

	conn, err := grpc.Dial("192.168.0.1:9080", grpc.WithTransportCredentials(insecure.NewCredentials()))
	if nil != err {
		return
	}
	defer func() {
		conn.Close()
	}()
	client := routeguide.NewRouteGuideClient(conn)
	md := metadata.MD{}
	md.Set("header1", "test")
	ctx = metadata.NewOutgoingContext(ctx, md)
	var rspheader metadata.MD
	var rsptrailer metadata.MD
	req := routeguide.Point{Latitude: 99, Longitude: 88}
	rsp, err := cli.GetFeature(ctx , &req, grpc.Header(&rspheader), grpc.Trailer(&rsptrailer))
	if err != nil {
		logs.Error(err)      //here report error "server closed the stream without sending trailers"
		return
	}
	logs.Info("rsp:", rsp.String())
        logs.Info("rspheader:", rspheader)
        logs.Info("rsptrailer:", rsptrailer)

[soulbird]

It seems that it is the same issue as apache/apisix#7973.
Please do not submit the same issue in the two projects, it will be inconvenient to track, and information will be lost.

[gitcome]

ok, sorry, i close this