envoyfilter on istiogateway conflict with default multi-clusterdefault NETWORK envoyfilter

[johnzheng1975]

Istio 1.4.6

Bug description
envoyfilter on istiogateway conflict with default multi-cluster default NETWORK envoyfilter
Step 1: I installed istio multi-cluster successfully
https://archive.istio.io/v1.4/docs/setup/install/multicluster/gateways/
https://istio.io/latest/docs/setup/install/multicluster/gateways/
Step 2: I found a default NETWORK filter installed in remote cluster by default

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  creationTimestamp: "2020-06-02T11:31:29Z"
  generation: 1
  labels:
    app: gateways
    chart: gateways
    heritage: Tiller
    release: istio
  name: istio-multicluster-ingressgateway
  namespace: istio-system
  resourceVersion: "2703610"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/envoyfilters/istio-multicluster-ingressgateway
  uid: f9fd8377-face-4249-b4c8-fc6a972fa141
spec:
  filters:
  - filterConfig:
      cluster_pattern: \.global$
      cluster_replacement: .svc.cluster.local
    filterName: envoy.filters.network.tcp_cluster_rewrite
    filterType: NETWORK
    insertPosition:
      index: AFTER
      relativeTo: envoy.filters.network.sni_cluster
    listenerMatch:
      listenerType: GATEWAY
      portNumber: 15443
  workloadLabels:
    app: istio-ingressgateway
    istio: ingressgateway

Step 3: I installed my own envoyfilter

kind: EnvoyFilter
metadata:
  name: istio-multicluster-ingressgateway-tenant-id
  namespace: istio-system
spec:
  workloadLabels:
    app: istio-ingressgateway
    istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.http_connection_manager
            subFilter:
              name: envoy.router
        portNumber: 80
    patch:
      operation: INSERT_BEFORE
      value:
        config:
          inlineCode: |
            function envoy_on_request(request_handle)
              request_handle:logInfo("*************** enter envoy_on_request istio-multicluster-ingressgateway-tenant-id")  
              local path = headers:get(":path")
              local method = headers:get(":method")
              local host = headers:get(":authority")
              request_handle:logInfo(path)
              request_handle:logInfo(method)
              request_handle:logInfo(host)
              request_handle:headers():replace("tenant_id","A_Great_Tenant")
            end
        name: envoy.lua

Expect result:
Both envoyfilter can take effective.

Actual result:
If I access from internet, My own envoyfilter works.
If I access from another cluster, ONLY istio default envoyfilter istio-multicluster-ingressgateway takes effective, my own envoyfilter not works. The http request does not go into my enovyfilter at all.

Question: Is there anyway to make both envoyfilter works? Thanks.

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[x] Developer Infrastructure

Affected features (please put an X in all that apply)

[x] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
istio 1.4.6
k8s 1.15

How was Istio installed?
Istio multi-cluster installed per
https://archive.istio.io/v1.4/docs/setup/install/multicluster/gateways/
https://istio.io/latest/docs/setup/install/multicluster/gateways/

Environment where bug was observed (cloud vendor, OS, etc)
AWS, EKS

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2020-06-29. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.