constant http request errors to external service domain even after modifying DNS record

[fatedier]

Bug Description

There are already K8s services with port 80 and protocol HTTP.

Start a new test service written in golang and it will keep sending HTTP requests to xxx.com resolved to an error IP with no response.

Test service always gets 503 connection failure error because of 10s of connection timeout. Then resolve xxx.com to correct IP, I expect the test service can get 200 responses but not. It can't recover anymore.

However, it can be recovered if xxx.com is changed to xxx.com:8080 since the go client will get network errors like connection reset by peer instead of HTTP 503 code response. It can reconnect to a new correct IP.

istioctl proxy-config listeners shows there are 80 listeners and not for 8080.

0.0.0.0        80    Trans: raw_buffer; App: HTTP                                                                    Route: 80
0.0.0.0        80    ALL                                                                                             PassthroughCluster

For my scenario, there are many services that constantly call external services on port 80 with different hosts. Assume that one external service is down and DNS record is resolved to another IP, internal services will always get 503 HTTP response and can't recover anymore until restart itself(client using keepalive HTTP transport won't disconnect and create a new connection).

Without istio, internal service can recover after the DNS record is resolved to correct IP.

Version

Istio
client version: 1.12.1
control plane version: 1.12.1
data plane version: 1.12.1 (3 proxies)

K8s version 1.16.15

Additional Information

No response

[howardjohn]

Is your application re-resolving DNS? Or it holds open connection to envoy, getting all 503s, so never re-resolves?

[fatedier]

Is your application re-resolving DNS? Or it holds open connection to envoy, getting all 503s, so never re-resolves?

It's a simple go HTTP client and will keep open connection to envoy because there are always HTTP requests on this connection getting 503s.

[howardjohn]

I opened envoyproxy/envoy#19458, this cannot be done without Envoy changes. Or explicitly make a https://istio.io/docs/reference/config/networking/service-entry/ for each domain

[fatedier]

Thanks for your quick response.

I have read about ServiceEntry, but we can't use it simply because it's difficult to list all domains used by all services in production environment.

Hope envoy changes can fix this issue.

[fatedier]

Not stale

[zirain]

Not stale