Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Http2Rpc配置两个Http2Rpc，路由转发失败 #926

Closed
java4candy opened this issue Apr 25, 2024 · 8 comments · Fixed by #935
Closed

Http2Rpc配置两个Http2Rpc，路由转发失败 #926

java4candy opened this issue Apr 25, 2024 · 8 comments · Fixed by #935
Assignees
@hanxiantao
Labels
area/dubbo sig/controller type/bug Something isn't working

Comments

@java4candy
Copy link

java4candy commented Apr 25, 2024 •
edited

Ⅰ. Issue Description

Http2Rpc配置两个Http2Rpc，路由转发失败

Ⅱ. Describe what happened

通过Higress，实现HTTP转同一个Dubbo下的两个Service时，配置了两个Ingress后，对应配置任意一个Http2Rpc，都可以正常访问。
同时配置两个Http2Rpc，两个API接口都返回失败：
NOT_FOUND: Could not resolve /account/name to a method
NOT_FOUND: Could not resolve /echo/echo to a method

Ingress && Http2Rpc yaml

curl --location --request GET 'http://x.x.x.x:xxx/echo/echo?p=123'
curl --location --request GET 'http://x.x.x.x:xxx/account/name?p=123'

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/destination: providers:com.alibaba.edas.boot.EchoService:1.0.0:DUBBO.zookeeper
    higress.io/ignore-path-case: 'false'
    higress.io/rpc-destination-name: echo
  labels:
    higress.io/resource-definer: higress
  name: echo-higress
  namespace: higress-system
spec:
  ingressClassName: higress
  rules:
    - http:
        paths:
          - backend:
              resource:
                apiGroup: networking.higress.io
                kind: McpBridge
                name: default
            path: /echo
            pathType: Prefix
status:
  loadBalancer: {}

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/destination: providers:com.alibaba.edas.boot.AccountService:1.0.0:DUBBO.zookeeper
    higress.io/ignore-path-case: 'false'
    higress.io/rpc-destination-name: account
  labels:
    higress.io/resource-definer: higress
  name: account-higress
  namespace: higress-system
spec:
  ingressClassName: higress
  rules:
    - http:
        paths:
          - backend:
              resource:
                apiGroup: networking.higress.io
                kind: McpBridge
                name: default
            path: /account
            pathType: Prefix
status:
  loadBalancer: {}



apiVersion: networking.higress.io/v1
kind: Http2Rpc
metadata:
  name: echo
  namespace: higress-system
spec:
  dubbo:
    group: DUBBO
    methods:
      - httpMethods:
          - GET
        httpPath: /echo/echo
        params:
          - paramKey: p
            paramSource: QUERY
            paramType: java.lang.String
        serviceMethod: echo
    service: com.alibaba.edas.boot.EchoService
    version: 1.0.0


apiVersion: networking.higress.io/v1
kind: Http2Rpc
metadata:
  name: account
  namespace: higress-system
spec:
  dubbo:
    group: DUBBO
    methods:
      - httpMethods:
          - GET
        httpPath: /account/name
        params:
          - paramKey: p
            paramSource: QUERY
            paramType: java.lang.String
        serviceMethod: name
    service: com.alibaba.edas.boot.AccountService
    version: 1.0.0

Environment:

  • Higress version: 1.3.3/1.3.6
  • OS : K8S v1.26.7
@johnlanni
Copy link
Collaborator

@java4candy 麻烦配置后，执行下hgctl gateway-config listener，发一下对应的配置

@java4candy
Copy link
Author

@java4candy 麻烦配置后，执行下hgctl gateway-config listener，发一下对应的配置

配置如下：

{
  "@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump",
  "dynamic_listeners": [
    {
      "active_state": {
        "last_updated": "2024-04-26T09:42:56.292Z",
        "listener": {
          "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
          "access_log": [
            {
              "filter": {
                "response_flag_filter": {
                  "flags": [
                    "NR"
                  ]
                }
              },
              "name": "envoy.access_loggers.file",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
                "log_format": {
                  "text_format_source": {
                    "inline_string": "{\"authority\":\"%REQ(:AUTHORITY)%\",\"bytes_received\":\"%BYTES_RECEIVED%\",\"bytes_sent\":\"%BYTES_SENT%\",\"downstream_local_address\":\"%DOWNSTREAM_LOCAL_ADDRESS%\",\"downstream_remote_address\":\"%DOWNSTREAM_REMOTE_ADDRESS%\",\"duration\":\"%DURATION%\",\"istio_policy_status\":\"%DYNAMIC_METADATA(istio.mixer:status)%\",\"method\":\"%REQ(:METHOD)%\",\"path\":\"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\"protocol\":\"%PROTOCOL%\",\"request_id\":\"%REQ(X-REQUEST-ID)%\",\"requested_server_name\":\"%REQUESTED_SERVER_NAME%\",\"response_code\":\"%RESPONSE_CODE%\",\"response_flags\":\"%RESPONSE_FLAGS%\",\"route_name\":\"%ROUTE_NAME%\",\"start_time\":\"%START_TIME%\",\"trace_id\":\"%REQ(X-B3-TRACEID)%\",\"upstream_cluster\":\"%UPSTREAM_CLUSTER%\",\"upstream_host\":\"%UPSTREAM_HOST%\",\"upstream_local_address\":\"%UPSTREAM_LOCAL_ADDRESS%\",\"upstream_service_time\":\"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\"upstream_transport_failure_reason\":\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\"user_agent\":\"%REQ(USER-AGENT)%\",\"x_forwarded_for\":\"%REQ(X-FORWARDED-FOR)%\"}\n"
                  }
                },
                "path": "/dev/stdout"
              }
            }
          ],
          "address": {
            "socket_address": {
              "address": "0.0.0.0",
              "port_value": 80
            }
          },
          "filter_chains": [
            {
              "filters": [
                {
                  "name": "envoy.filters.network.http_connection_manager",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                    "access_log": [
                      {
                        "filter": {
                          "not_health_check_filter": {}
                        },
                        "name": "envoy.access_loggers.file",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
                          "log_format": {
                            "text_format_source": {
                              "inline_string": "{\"authority\":\"%REQ(:AUTHORITY)%\",\"bytes_received\":\"%BYTES_RECEIVED%\",\"bytes_sent\":\"%BYTES_SENT%\",\"downstream_local_address\":\"%DOWNSTREAM_LOCAL_ADDRESS%\",\"downstream_remote_address\":\"%DOWNSTREAM_REMOTE_ADDRESS%\",\"duration\":\"%DURATION%\",\"istio_policy_status\":\"%DYNAMIC_METADATA(istio.mixer:status)%\",\"method\":\"%REQ(:METHOD)%\",\"path\":\"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\"protocol\":\"%PROTOCOL%\",\"request_id\":\"%REQ(X-REQUEST-ID)%\",\"requested_server_name\":\"%REQUESTED_SERVER_NAME%\",\"response_code\":\"%RESPONSE_CODE%\",\"response_flags\":\"%RESPONSE_FLAGS%\",\"route_name\":\"%ROUTE_NAME%\",\"start_time\":\"%START_TIME%\",\"trace_id\":\"%REQ(X-B3-TRACEID)%\",\"upstream_cluster\":\"%UPSTREAM_CLUSTER%\",\"upstream_host\":\"%UPSTREAM_HOST%\",\"upstream_local_address\":\"%UPSTREAM_LOCAL_ADDRESS%\",\"upstream_service_time\":\"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\"upstream_transport_failure_reason\":\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\"user_agent\":\"%REQ(USER-AGENT)%\",\"x_forwarded_for\":\"%REQ(X-FORWARDED-FOR)%\"}\n"
                            }
                          },
                          "path": "/dev/stdout"
                        }
                      }
                    ],
                    "common_http_protocol_options": {
                      "idle_timeout": "180s"
                    },
                    "forward_client_cert_details": "SANITIZE_SET",
                    "http2_protocol_options": {
                      "initial_connection_window_size": 1048576,
                      "initial_stream_window_size": 65535,
                      "max_concurrent_streams": 100
                    },
                    "http_filters": [
                      {
                        "name": "envoy.filters.http.cors",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
                        }
                      },
                      {
                        "name": "envoy.filters.http.rbac",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC"
                        }
                      },
                      {
                        "name": "envoy.filters.http.local_ratelimit",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit",
                          "stat_prefix": "http_local_rate_limiter"
                        }
                      },
                      {
                        "name": "envoy.filters.http.fault",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
                        }
                      },
                      {
                        "name": "envoy.filters.http.http_dubbo_transcoder",
                        "typed_config": {
                          "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                          "type_url": "type.googleapis.com/envoy.extensions.filters.http.http_dubbo_transcoder.v3.HttpDubboTranscoder"
                        }
                      },
                      {
                        "name": "envoy.filters.http.http_dubbo_transcoder",
                        "typed_config": {
                          "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                          "type_url": "type.googleapis.com/envoy.extensions.filters.http.http_dubbo_transcoder.v3.HttpDubboTranscoder"
                        }
                      },
                      {
                        "name": "envoy.filters.http.router",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                        }
                      }
                    ],
                    "http_protocol_options": {
                      "accept_http_10": true
                    },
                    "max_request_headers_kb": 60,
                    "normalize_path": true,
                    "path_with_escaped_slashes_action": "KEEP_UNCHANGED",
                    "rds": {
                      "config_source": {
                        "ads": {},
                        "initial_fetch_timeout": "0s",
                        "resource_api_version": "V3"
                      },
                      "route_config_name": "http.80"
                    },
                    "server_name": "istio-envoy",
                    "set_current_client_cert_details": {
                      "cert": true,
                      "dns": true,
                      "subject": true,
                      "uri": true
                    },
                    "stat_prefix": "outbound_0.0.0.0_80",
                    "stream_idle_timeout": "180s",
                    "tracing": {
                      "client_sampling": {
                        "value": 100
                      },
                      "custom_tags": [
                        {
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_allow_shadow_effective_policy_id"
                                }
                              ]
                            }
                          },
                          "tag": "istio.authorization.dry_run.allow_policy.name"
                        },
                        {
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_allow_shadow_engine_result"
                                }
                              ]
                            }
                          },
                          "tag": "istio.authorization.dry_run.allow_policy.result"
                        },
                        {
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_deny_shadow_effective_policy_id"
                                }
                              ]
                            }
                          },
                          "tag": "istio.authorization.dry_run.deny_policy.name"
                        },
                        {
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_deny_shadow_engine_result"
                                }
                              ]
                            }
                          },
                          "tag": "istio.authorization.dry_run.deny_policy.result"
                        },
                        {
                          "literal": {
                            "value": "latest"
                          },
                          "tag": "istio.canonical_revision"
                        },
                        {
                          "literal": {
                            "value": "unknown"
                          },
                          "tag": "istio.canonical_service"
                        },
                        {
                          "literal": {
                            "value": "unknown"
                          },
                          "tag": "istio.mesh_id"
                        },
                        {
                          "literal": {
                            "value": "higress-system"
                          },
                          "tag": "istio.namespace"
                        }
                      ],
                      "overall_sampling": {
                        "value": 100
                      },
                      "random_sampling": {
                        "value": 1
                      }
                    },
                    "upgrade_configs": [
                      {
                        "upgrade_type": "websocket"
                      }
                    ],
                    "use_remote_address": true
                  }
                }
              ]
            }
          ],
          "name": "0.0.0.0_80",
          "per_connection_buffer_limit_bytes": 32768,
          "traffic_direction": "OUTBOUND"
        },
        "version_info": "2024-04-26T09:13:52Z/21"
      },
      "name": "0.0.0.0_80"
    }
  ],
  "static_listeners": [
    {
      "last_updated": "2024-04-26T08:59:49.183Z",
      "listener": {
        "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
        "address": {
          "socket_address": {
            "address": "0.0.0.0",
            "port_value": 15090
          }
        },
        "filter_chains": [
          {
            "filters": [
              {
                "name": "envoy.filters.network.http_connection_manager",
                "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                  "http_filters": [
                    {
                      "name": "envoy.filters.http.router",
                      "typed_config": {
                        "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                      }
                    }
                  ],
                  "route_config": {
                    "virtual_hosts": [
                      {
                        "domains": [
                          "*"
                        ],
                        "name": "backend",
                        "routes": [
                          {
                            "match": {
                              "prefix": "/stats/prometheus"
                            },
                            "route": {
                              "cluster": "prometheus_stats"
                            }
                          }
                        ]
                      }
                    ]
                  },
                  "stat_prefix": "stats"
                }
              }
            ]
          }
        ]
      }
    },
    {
      "last_updated": "2024-04-26T08:59:49.197Z",
      "listener": {
        "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
        "address": {
          "socket_address": {
            "address": "0.0.0.0",
            "port_value": 15021
          }
        },
        "filter_chains": [
          {
            "filters": [
              {
                "name": "envoy.filters.network.http_connection_manager",
                "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                  "http_filters": [
                    {
                      "name": "envoy.filters.http.router",
                      "typed_config": {
                        "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                      }
                    }
                  ],
                  "route_config": {
                    "virtual_hosts": [
                      {
                        "domains": [
                          "*"
                        ],
                        "name": "backend",
                        "routes": [
                          {
                            "match": {
                              "prefix": "/healthz/ready"
                            },
                            "route": {
                              "cluster": "agent"
                            }
                          }
                        ]
                      }
                    ]
                  },
                  "stat_prefix": "agent"
                }
              }
            ]
          }
        ]
      }
    }
  ],
  "version_info": "2024-04-26T09:42:56Z/30"
}

@johnlanni
Copy link
Collaborator

envoy.filters.http.http_dubbo_transcoder

这个filter在全局被插入了两次，多个http2rpc的cr，应该复用同一个filter
cc @Hinsteny

@johnlanni johnlanni added type/bug Something isn't working help wanted Extra attention is needed area/dubbo sig/controller labels Apr 30, 2024
@hanxiantao
Copy link
Contributor

我这边来修复下，请帮忙分配下，感谢 @johnlanni

@CH3CHO CH3CHO removed the help wanted Extra attention is needed label May 7, 2024
@hanxiantao
Copy link
Contributor

hanxiantao commented May 10, 2024 •
edited

https://github.com/alibaba/higress/blob/2031c659c2061293ef71f393ef2bf98a3881ae11/pkg/ingress/config/ingress_config.go#L512C1-L532C5
修复方案：在convertEnvoyFilter()调用constructHttp2RpcEnvoyFilter()时添加一个标志，只有第一次调用constructHttp2RpcEnvoyFilter()时才会添加envoy.filters.http.http_dubbo_transcoder的部分，后续转换的EnvoyFilter不再添加envoy.filters.http.http_dubbo_transcoder的部分
我这边会在本地做下测试后，再提交PR

@johnlanni
Copy link
Collaborator

@hanxiantao 可以的

hanxiantao added a commit to hanxiantao/higress that referenced this issue May 12, 2024
@hanxiantao
fix: fix when multiple http2Rpc config constructHttp2RpcEnvoyFilter (a…
0fa9167 
…libaba#926)
@hanxiantao hanxiantao mentioned this issue May 12, 2024
Merged
hanxiantao added a commit to hanxiantao/higress that referenced this issue May 12, 2024
@hanxiantao
fix: when multiple http2Rpc config constructHttp2RpcEnvoyFilter (alib…
7b3fee3 
…aba#926)
@hanxiantao
Copy link
Contributor

hanxiantao commented May 14, 2024 •
edited

@hanxiantao 可以的

@johnlanni pr已提交，有时间帮忙review下 #935

@johnlanni
Copy link
Collaborator

fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hanxiantao hanxiantao

Labels
area/dubbo sig/controller type/bug Something isn't working
Projects
Higress
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: when multiple http2Rpc config constructHttp2RpcEnvoyFilter (… hanxiantao/higress
4 participants
@CH3CHO @johnlanni @java4candy @hanxiantao