listener filter timeout when connecting mysql after upgrade from v0.5.0 to 4b9e393c

[addozhang]

Bug description:

It works in v0.5.0 with egress enabled.

[2020-12-09 03:48:01.621][18][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
[2020-12-09 03:48:01.621][18][debug][filter] [source/extensions/filters/listener/http_inspector/http_inspector.cc:38] http inspector: new connection accepted
[2020-12-09 03:48:16.622][18][debug][conn_handler] [source/server/connection_handler_impl.cc:273] listener filter times out after 15000 ms

Affected area (please mark with X where applicable):

• Install [ ]
• SMI Traffic Access Policy [ ]
• SMI Traffic Specs Policy [ ]
• SMI Traffic Split Policy [ ]
• Permissive Traffic Policy [ ]
• Ingress [ ]
• Egress [ ]
• Envoy Control Plane [ ]
• CLI Tool [ ]
• Metrics [ ]
• Certificate Management [ ]
• Sidecar Injection [ ]
• Logging [ ]
• Debugging [ ]
• Tests [ ]
• CI System [ ]

Expected behavior:

Connect mysql successfully.

Steps to reproduce the bug (as precisely as possible):

How was OSM installed?:

Anything else we need to know?:

Environment:

• OSM version (use osm version): 4b9e393
• Kubernetes version (use kubectl version):
• Size of cluster (number of worker nodes in the cluster):
• Others:

[addozhang]

Checked source code, it is issue with the new added envoy.filters.listener.http_inspector in outbound listener.

When connecting to mysql, the inspector will keep waiting data from socket, but there is no http flag.

@shashankram I found your description in #2101, it only support HTTP protocol. How to handle the non-http traffic?

There is an issue in envoy envoyproxy/envoy#7195

[shashankram]

Checked source code, it is issue with the new added envoy.filters.listener.http_inspector in outbound listener.

When connecting to mysql, the inspector will keep waiting data from socket, but there is no http flag.

@shashankram I found your description in #2101, it only support HTTP protocol. How to handle the non-http traffic?

There is an issue in envoy envoyproxy/envoy#7195

Current OSM only supports HTTP traffic. If you require support for non HTTP traffic, please open an issue for the same.
We are working on adding TCP support tracked in #1521.

We need the http_inspector listener filter to detect outbound HTTP traffic for in-mesh destination traffic.

It seems like this issue will be resolved once #2096 is implemented.

[shashankram]

@addozhang this should be resolved by PR #2299. The http_inspector filter no longer exists on the outbound listener. Could you test and see if this works on latest main branch.

[shashankram]

@addozhang this should be resolved by PR #2299. The http_inspector filter no longer exists on the outbound listener. Could you test and see if this works on latest main branch.

I verified that external mySQL traffic works via the egress functionality that can be enabled by setting enable_egress=true in osm-system/osm-config ConfigMap.

For testing, I used an external mySQL server and a client that is a part of the service mesh.

mysql-client

mysql-client-7fc98fc-trms5   2/2     Running            2          20s

The client will not connect to the mySQL server if connection is broken and the prompt seen below will not be available otherwise.

$ kubectl run -n test -it --rm --image=mysql:5.6 --restart=Always mysql-client -- mysql -h mysql.mysql -ppassword

If you don't see a command prompt, try pressing enter.

mysql> 
mysql> help