Ingress gateway request to headless service 503

[MiguelRexAlu]

Bug description
I'm configuring ISTIO using helm here you can find my istio-config.yaml

global:
  proxy:
    accessLogFile: "/dev/stdout"
    resources:
      requests:
        cpu: 10m
        memory: 40Mi

  disablePolicyChecks: false

sidecarInjectorWebhook:
  enabled: true
  rewriteAppHTTPProbe: false

pilot:
  autoscaleEnabled: false
  traceSampling: 100.0
  resources:
    requests:
      cpu: 10m
      memory: 100Mi

mixer:
  policy:
    enabled: true
    autoscaleEnabled: false
    resources:
      requests:
        cpu: 10m
        memory: 100Mi

  telemetry:
    enabled: true
    autoscaleEnabled: false
    resources:
      requests:
        cpu: 50m
        memory: 100Mi

  adapters:
    stdio:
      enabled: true

grafana:
  enabled: true

tracing:
  enabled: true

kiali:
  enabled: true
  createDemoSecret: true

gateways:
  istio-ingressgateway:
    autoscaleEnabled: false
    resources:
      requests:
        cpu: 10m
        memory: 40Mi

  istio-egressgateway:
    enabled: true
    autoscaleEnabled: false
    resources:
      requests:
        cpu: 10m
        memory: 40Mi

global:
  controlPlaneSecurityEnabled: false

  mtls:
    enabled: false

Then I deployed a bunch of microservices using istioctl, all of them are simple REST call using http. They can communicate each other with out any issue. If I exposed them with Node Ports I can reach and communicate with them correctly.

Here my svc:

kubectl get svc --all-namespaces
NAMESPACE              NAME                             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                                                                      AGE
default                activemq                         ClusterIP      None             <none>        61616/TCP                                                                                                                                    3h17m
default                activemq-np                      NodePort       10.110.76.147    <none>        8161:30061/TCP                                                                                                                               3h17m
default                api-exchange                     ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h16m
default                api-response                     ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h16m
default                authorization-server             ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h17m
default                de-communication                 ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h16m
default                gateway                          ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h17m
default                gateway-np                       NodePort       10.96.123.57     <none>        8080:30080/TCP                                                                                                                               3h17m
default                identity                         ClusterIP      None             <none>        88/TCP,8080/TCP                                                                                                                              3h18m
default                kubernetes                       ClusterIP      10.96.0.1        <none>        443/TCP                                                                                                                                      3h19m
default                matchengine                      ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h16m
default                monitor-redis                    ClusterIP      None             <none>        8081/TCP                                                                                                                                     3h17m
default                monitor-redis-np                 NodePort       10.106.178.13    <none>        8081:30082/TCP                                                                                                                               3h17m
default                postgres                         ClusterIP      None             <none>        5432/TCP                                                                                                                                     3h18m
default                postgres-np                      NodePort       10.106.223.216   <none>        5432:30032/TCP                                                                                                                               3h18m
default                redis                            ClusterIP      None             <none>        6379/TCP                                                                                                                                     3h18m
default                redis-np                         NodePort       10.101.167.194   <none>        6379:30079/TCP                                                                                                                               3h18m
default                synchronization                  ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h15m
default                tx-flow                          ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h15m
default                tx-manager                       ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h15m
default                tx-scheduler                     ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h15m
default                ubc-config                       ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h16m
default                ubc-services-config              ClusterIP      None             <none>        8888/TCP                                                                                                                                     3h18m
default                ubc-services-config-np           NodePort       10.110.11.213    <none>        8888:30088/TCP                                                                                                                               3h18m
default                user-admin                       ClusterIP      None             <none>        8080/TCP                                                                                                                                     3h17m
default                web-exchange-np                  NodePort       10.105.244.194   <none>        80:30081/TCP                                                                                                                                 3h15m
istio-system           grafana                          ClusterIP      10.97.134.230    <none>        3000/TCP                                                                                                                                     3h22m
istio-system           istio-citadel                    ClusterIP      10.99.159.56     <none>        8060/TCP,15014/TCP                                                                                                                           3h22m
istio-system           istio-egressgateway              ClusterIP      10.97.71.204     <none>        80/TCP,443/TCP,15443/TCP                                                                                                                     3h22m
istio-system           istio-galley                     ClusterIP      10.98.111.27     <none>        443/TCP,15014/TCP,9901/TCP                                                                                                                   3h22m
istio-system           istio-ingressgateway             LoadBalancer   10.96.182.202    <pending>     15020:30936/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31913/TCP,15030:30606/TCP,15031:32127/TCP,15032:30362/TCP,15443:31416/TCP   3h22m
istio-system           istio-pilot                      ClusterIP      10.101.117.169   <none>        15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                                       3h22m
istio-system           istio-policy                     ClusterIP      10.97.247.54     <none>        9091/TCP,15004/TCP,15014/TCP                                                                                                                 3h22m
istio-system           istio-sidecar-injector           ClusterIP      10.101.219.141   <none>        443/TCP                                                                                                                                      3h22m
istio-system           istio-telemetry                  ClusterIP      10.109.108.78    <none>        9091/TCP,15004/TCP,15014/TCP,42422/TCP                                                                                                       3h22m
istio-system           jaeger-agent                     ClusterIP      None             <none>        5775/UDP,6831/UDP,6832/UDP                                                                                                                   3h22m
istio-system           jaeger-collector                 ClusterIP      10.97.255.231    <none>        14267/TCP,14268/TCP                                                                                                                          3h22m
istio-system           jaeger-query                     ClusterIP      10.104.80.162    <none>        16686/TCP                                                                                                                                    3h22m
istio-system           kiali                            ClusterIP      10.104.41.71     <none>        20001/TCP                                                                                                                                    3h22m
istio-system           kiali-np                         NodePort       10.100.99.141    <none>        20001:30085/TCP                                                                                                                              29h
istio-system           prometheus                       ClusterIP      10.110.46.60     <none>        9090/TCP                                                                                                                                     3h22m
istio-system           tracing                          ClusterIP      10.111.173.205   <none>        80/TCP                                                                                                                                       3h22m
istio-system           zipkin                           ClusterIP      10.101.144.199   <none>        9411/TCP                                                                                                                                     3h22m
kube-system            kube-dns                         ClusterIP      10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP                                                                                                                       54d
kube-system            tiller-deploy                    ClusterIP      10.105.162.195   <none>        44134/TCP                                                                                                                                    24d

I created an ingress gateway and one VirtualServices to route call from outside the cluster. Here gateway and Virtual Services configuration:

Gateway:

kubectl describe gateway iris-gateway
Name:         iris-gateway
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"iris-gateway","namespace":"default"},"s...
API Version:  networking.istio.io/v1alpha3
Kind:         Gateway
Metadata:
  Creation Timestamp:  2019-08-23T17:25:20Z
  Generation:          1
  Resource Version:    7093263
  Self Link:           /apis/networking.istio.io/v1alpha3/namespaces/default/gateways/iris-gateway
  UID:                 4c4fac7d-a698-4c9c-97e6-ebc7416c96a8
Spec:
  Selector:
    Istio:  ingressgateway
  Servers:
    Hosts:
      *
    Port:
      Name:      http
      Number:    80
      Protocol:  HTTP
Events:          <none>

Virtual Services:

kubectl describe virtualservice apiexg
Name:         apiexg
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"apiexg","namespace":"default"},"...
API Version:  networking.istio.io/v1alpha3
Kind:         VirtualService
Metadata:
  Creation Timestamp:  2019-08-23T19:26:16Z
  Generation:          1
  Resource Version:    7107510
  Self Link:           /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/apiexg
  UID:                 861bca0d-be98-4bfb-bf92-b2bd2f1b703f
Spec:
  Gateways:
    iris-gateway
  Hosts:
    *
  Http:
    Match:
      Uri:
        Prefix:  /api-exchange
    Route:
      Destination:
        Host:  api-exchange.default.svc.cluster.local
        Port:
          Number:  8080
Events:            <none>

When I make a call to the service I always got a 503 Service Unavailable:

curl -X POST http://172.30.7.129:31380/api-exchange/ -vvv
* About to connect() to 172.30.7.129 port 31380 (#0)
*   Trying 172.30.7.129...
* Connected to 172.30.7.129 (172.30.7.129) port 31380 (#0)
> POST /api-exchange/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.30.7.129:31380
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< content-length: 19
< content-type: text/plain
< date: Fri, 23 Aug 2019 21:49:33 GMT
< server: istio-envoy
< 
* Connection #0 to host 172.30.7.129 left intact
no healthy upstream

Here log output for istio-ingressgateway pod:

[2019-08-23 21:49:34.185][38][warning][upstream] [external/envoy/source/common/upstream/original_dst_cluster.cc:110] original_dst_load_balancer: No downstream connection or no original_dst.

Affected product area (please put an X in all that apply)

[X] Configuration Infrastructure
[ ] Docs
[X] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior
Communication from outside using ingress gateway Node Port

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version)

istioctl version --remote
client version: 1.2.4
citadel version: 1.2.4
egressgateway version: 94746ccd404a8e056483dd02e4e478097b950da6-dirty
galley version: 1.2.4
ingressgateway version: 94746ccd404a8e056483dd02e4e478097b950da6-dirty
pilot version: 1.2.4
policy version: 1.2.4
sidecar-injector version: 1.2.4
telemetry version: 1.2.4

Kubernetes

kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?

helm install /opt/istio-1.2.4/install/kubernetes/helm/istio-init --name istio-init --namespace istio-system
helm install /opt/istio-1.2.4/install/kubernetes/helm/istio --name istio --namespace istio-system --values istio-config/istio-config.yaml

Environment where bug was observed (cloud vendor, OS, etc)
Virtual Server with RHEL 7 and Physical Server with RHEL 7

Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

[cmanzi]

@MiguelRexAlu I was running into the same issue. Upgraded to 1.2.5 and it's working now.

[MiguelRexAlu]

@cmanzi Thanks for the suggestion but I just found the issue and I solved it.

All my services was created with clusteIP: None and that's make my configuration to fail. I removed that tag and now is working aftwer three weeks of changing configuration. I think that should be part of the documentation in virtual services section.

[rshriram]

cc @frankbu ^^^

[frankbu]

@cmanzi Can you show us your api-exchange service definition? I'm wondering if the port and targetPort are different, which could be exposing a bug with headless (that is, clusterIP: None) services.

[frankbu]

^^^ Sorry, my above question was for @MiguelRexAlu (not @cmanzi).

[MiguelRexAlu]

@frankbu Hello, I think @cmanzi have a differente problem.

Here you can find my api-exchange definition befor I removed ClusterIP:None tag:

apiVersion: v1
kind: Service
metadata:
  name: api-exchange
  labels:
    app: api-exchange
spec:
  clusterIP: None
  ports:
  - port: 8080
    name: api-exchange
  selector:
    app: api-exchange
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-exchange
spec:
  selector:
    matchLabels:
      app: api-exchange
  replicas: 2
  template:
    metadata:
      labels:
        app: api-exchange
    spec:
      containers:
      - name: api-exchange
        image: api-exchange:SNAPSHOT-3.4.1_39
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "500m"

This is my new definition:

apiVersion: v1
kind: Service
metadata:
  name: api-exchange
  labels:
    app: api-exchange
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: api-exchange
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-exchange
spec:
  selector:
    matchLabels:
      app: api-exchange
  replicas: 2
  template:
    metadata:
      labels:
        app: api-exchange
        version: SNAPSHOT-3.4.1_50
    spec:
      containers:
      - name: api-exchange
        image: api-exchange:SNAPSHOT-3.4.1_50
        ports:
        - containerPort: 8080
          name: http-exchange
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "500m"

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2019-09-18. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.