Request Headers Route Rule with composite services does not work

[ayj]

@my3sons commented on Sun Oct 15 2017

Is this a BUG or FEATURE REQUEST?:
BUG

Did you review existing epics or issues to identify if this already being worked on? (please try to add the correct labels and epics):
Bug:
Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

**istioctl version** 
Version: 0.2.7
GitRevision: 6b145c189aad8306b13af1725123bebfbc7eefd4
GitBranch: master
User: root@f1eeb85f62ab
GolangVersion: go1.8.3

**kubectl version**
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not ?
NO AUTH

What happened:
I have a composite microservice (foobar) that is calling 2 nested atomic services (foo and bar) and all 3 services are part of my istio service mesh. I set up a Route Rule (see below) with a destination of bar and a match configuration based on source and request header. I am expecting that if I send a curl request to my ingress controller such as the following:

curl -s http://10.111.233.201:80/foobar --header "clientid:esp"

I should see that the http request from foobar to bar will be directed to v2 of bar. However, I am unable to ever get that match rule to work. I have tried making curl requests directly to ingress controller, as well as by getting a shell into one of the other pods (e.g. foo or bar) and submitting request from there, but I get the same outcome.

If I remove the request headers rule and just go with the match on source, the route rule works as expected. I am assuming that the request headers are not being passed from foobar down to bar via Envoy?

The actual implementation of my services is Java and foobar is making http requests to foo and bar using Spring RestTemplate. The urls that I have configured for those http calls are http://foo:8080/foo and http://bar:8080/bar respectively (see foobar.yaml below for my service configs).

As I have mentioned above, all the route rules I have set up previously based on source and/or version from foobar to the nested services have all worked as expected. It is just the request header based rules that seem to not be working.

It should be noted that if I set up a similar rule (using request headers) on one of the atomic services, everything works as expected.

My Route Rule

apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
  name: foobar-client1
spec:
  destination:
    name: bar
  precedence: 4
  match:
    source: 
        name: foobar
    request:
        headers:
            clientid: 
                exact: esp
  route:
  - labels:
      version: v2

foobar.yaml

# Copyright 2017 Istio Authors
#
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

##################################################################################################
# Foo service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: foo
  labels:
    app: foo
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: foo
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: foo-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: foo
        version: v1
    spec:
      containers:
      - name: foo
        image: boldec/isty-foo:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
##################################################################################################
# Bar service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: bar
  labels:
    app: bar
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: bar
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bar-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: bar
        version: v1
    spec:
      containers:
      - name: bar
        image: boldec/isty-bar:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bar-v2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: bar
        version: v2
    spec:
      containers:
      - name: bar
        image: boldec/isty-bar:v2
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bar-v3
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: bar
        version: v3
    spec:
      containers:
      - name: bar
        image: boldec/isty-bar:v3
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---

##################################################################################################
# FooBar service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: foobar
  labels:
    app: foobar
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: foobar
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: foobar-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: foobar
        version: v1
    spec:
      containers:
      - name: foobar
        image: boldec/isty-poobah:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---

######################
# Ingress resource (gateway)
##########################################################################
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gateway
  annotations:
    kubernetes.io/ingress.class: "istio"
spec:
  rules:
  - http:
      paths:
      - path: /foo
        backend:
          serviceName: foo
          servicePort: 8080
      - path: /bar
        backend:
          serviceName: bar
          servicePort: 8080
      - path: /foobar
        backend:
          serviceName: foobar
          servicePort: 8080
 
---

[kyessenov]

Issue moved to istio/istio #1380 via ZenHub