help request: proxy grpc sometimes occurs 502

[jujiale]

Description

hello:
we have two apisix env(such as A and B), when we use apisix to proxy grpc service, A always OK, but in B, when you first time send a request. It returns 502, but the following request all returns 200. and I also found that when we send a request in B, after a while，send request again, then the first request always return 502, the following return 200. I am confused of such thing. because A and B env apisix version and depends are same.

when 502 occurs. the following error occurs:
`

[error] 20620#20620: *327238 no connection data found for keepalive http2 connection while sending request to upstream

`

I want to get some idea from you. what causes it and how to solve it.
hope yours. thanks.

Environment

• APISIX version (run apisix version): v2.12.0
• Operating system (run uname -a): the problem env is :Linux 3.10.0-1160.36.2.el7.x86_64. the good env is: Linux 3.10.0-957.21.3.el7.x86_64
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.21.4.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.0
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[tzssangglass]

1. what are routes and upstreams config info?
2. whether the route and the bound upstream which occurs 502 still proxy HTTP1.1/ requests？

[jujiale]

1. what are routes and upstreams config info?
2. whether the route and the bound upstream which occurs 502 still proxy HTTP1.1/ requests？

@tzssangglass
1.
the following is our config info.
route config:
`

  {
  "uri": "/*",
  "name": "xxx-xxx-xxx.grpc",
  "priority": 100,
  "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
  ],
  "host": "xxx-xxx-xxx.grpc",
  "upstream_id": "431317757666001848",
  "status": 1
}

`

upstream config:
`

   {
  "nodes": [
    {
      "host": "172.xx.xxx.xxx",
      "port": 20000,
      "weight": 1
    }
  ],
  "timeout": {
    "connect": 6,
    "send": 6,
    "read": 6
  },
  "type": "roundrobin",
  "scheme": "grpc",
  "pass_host": "pass",
  "name": "xxx-xxx-xxx.grpc",
  "keepalive_pool": {
    "idle_timeout": 60,
    "requests": 1000,
    "size": 320
  }
}

`
and I think the config is not have problem. as I mentioned above, the same config in differenet env has different result. so I want to get some ideas from you. if you have some good suggestions. if some os config colud influence it

the route and the bound upstream not occurs 502 while proxy HTTP1.1/ requests

[tzssangglass]

the route and the bound upstream not occurs 502 while proxy HTTP1.1/ requests

"uri":"/*" would result in a mix of grpc (HTTP/2) and HTTP/1.1 requests. I think you can create two routes, one of which hits only grpc requests and the other handles HTTP/1.1 requests.

Verify that it appears with this distinction 502.

[jujiale]

the route and the bound upstream not occurs 502 while proxy HTTP1.1/ requests

"uri":"/*" would result in a mix of grpc (HTTP/2) and HTTP/1.1 requests. I think you can create two routes, one of which hits only grpc requests and the other handles HTTP/1.1 requests.

Verify that it appears with this distinction 502.

I do not think so. because we add host in mate rule. also apisix config.yaml open http2 port. we send grpc request with http2 port in apisix.

[tzssangglass]

or disable keepalive_pool in upstream by set idle_timeout to 0.

[jujiale]

or disable keepalive_pool in upstream by set idle_timeout to 0.

I have tried this. the problem also exists.

I will continue to solve this problem. if I find the reason which caused the problem. I will comment it in here.

[tzssangglass]

I thought this would be helpful to you: argoproj/argo-cd#3896 and kubernetes/ingress-nginx#4095

Is your upstream service handling both grpc requests and http/1.1 requests on the same port?

[jujiale]

I thought this would be helpful to you: argoproj/argo-cd#3896 and kubernetes/ingress-nginx#4095

Is your upstream service handling both grpc requests and http/1.1 requests on the same port?

I will have a look. thank you very much.

[jujiale]

I thought this would be helpful to you: argoproj/argo-cd#3896 and kubernetes/ingress-nginx#4095

Is your upstream service handling both grpc requests and http/1.1 requests on the same port?

Yes. our upstream service handling both grpc requests and http/1.1 requests on the same port.

[jujiale]

I find as in this issue said. keepalive might be the reason which caused the problem.
https://github.com/kubernetes/ingress-nginx/issues/4836#issuecomment-871342543

[tokers]

Maybe APISIX needs to add the upstream scheme as a component of the keepalive pool name.

[tzssangglass]

Maybe APISIX needs to add the upstream scheme as a component of the keepalive pool name.

+1

[Fijileijun]

Maybe APISIX needs to add the upstream scheme as a component of the keepalive pool name.

+1

+1, full of expectation. So what's the plannning that APISIX will support this issue?

[soulbird]

Maybe APISIX needs to add the upstream scheme as a component of the keepalive pool name.

+1

+1, full of expectation. So what's the plannning that APISIX will support this issue?

I'll fix it soon.