Request Headers Route Rule with composite services does not work

[kyessenov]

@ayj commented on Mon Oct 16 2017

@my3sons commented on Sun Oct 15 2017

Is this a BUG or FEATURE REQUEST?:
BUG

Did you review existing epics or issues to identify if this already being worked on? (please try to add the correct labels and epics):
Bug:
Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

**istioctl version** 
Version: 0.2.7
GitRevision: 6b145c189aad8306b13af1725123bebfbc7eefd4
GitBranch: master
User: root@f1eeb85f62ab
GolangVersion: go1.8.3

**kubectl version**
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not ?
NO AUTH

What happened:
I have a composite microservice (foobar) that is calling 2 nested atomic services (foo and bar) and all 3 services are part of my istio service mesh. I set up a Route Rule (see below) with a destination of bar and a match configuration based on source and request header. I am expecting that if I send a curl request to my ingress controller such as the following:

curl -s http://10.111.233.201:80/foobar --header "clientid:esp"

I should see that the http request from foobar to bar will be directed to v2 of bar. However, I am unable to ever get that match rule to work. I have tried making curl requests directly to ingress controller, as well as by getting a shell into one of the other pods (e.g. foo or bar) and submitting request from there, but I get the same outcome.

If I remove the request headers rule and just go with the match on source, the route rule works as expected. I am assuming that the request headers are not being passed from foobar down to bar via Envoy?

The actual implementation of my services is Java and foobar is making http requests to foo and bar using Spring RestTemplate. The urls that I have configured for those http calls are http://foo:8080/foo and http://bar:8080/bar respectively (see foobar.yaml below for my service configs).

As I have mentioned above, all the route rules I have set up previously based on source and/or version from foobar to the nested services have all worked as expected. It is just the request header based rules that seem to not be working.

It should be noted that if I set up a similar rule (using request headers) on one of the atomic services, everything works as expected.

My Route Rule

apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
  name: foobar-client1
spec:
  destination:
    name: bar
  precedence: 4
  match:
    source: 
        name: foobar
    request:
        headers:
            clientid: 
                exact: esp
  route:
  - labels:
      version: v2

foobar.yaml

# Copyright 2017 Istio Authors
#
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

##################################################################################################
# Foo service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: foo
  labels:
    app: foo
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: foo
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: foo-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: foo
        version: v1
    spec:
      containers:
      - name: foo
        image: boldec/isty-foo:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
##################################################################################################
# Bar service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: bar
  labels:
    app: bar
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: bar
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bar-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: bar
        version: v1
    spec:
      containers:
      - name: bar
        image: boldec/isty-bar:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bar-v2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: bar
        version: v2
    spec:
      containers:
      - name: bar
        image: boldec/isty-bar:v2
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bar-v3
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: bar
        version: v3
    spec:
      containers:
      - name: bar
        image: boldec/isty-bar:v3
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---

##################################################################################################
# FooBar service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: foobar
  labels:
    app: foobar
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: foobar
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: foobar-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: foobar
        version: v1
    spec:
      containers:
      - name: foobar
        image: boldec/isty-poobah:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---

######################
# Ingress resource (gateway)
##########################################################################
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gateway
  annotations:
    kubernetes.io/ingress.class: "istio"
spec:
  rules:
  - http:
      paths:
      - path: /foo
        backend:
          serviceName: foo
          servicePort: 8080
      - path: /bar
        backend:
          serviceName: bar
          servicePort: 8080
      - path: /foobar
        backend:
          serviceName: foobar
          servicePort: 8080
 
---

[rshriram]

@my3sons please try 0.8 and let us knwo if this issue persists.

[my3sons]

Hello Shriram @rshriram ,

I did retest this scenario with 0.8 and unfortunately, the issue still seems to persist. Below are the VirtualService configs for both my Composite (foo) and atomic (bar). When I send a request with header client-id:esp, the request gets to the composite fine, but the call to the atomic returns 404. I did check the istio-proxy config for my composite and I see the header match rule in there for bar and it all looks good. I am guessing that the header is still not being passed. Is there any easy way to debug this from the composite's proxy?

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: foo-vs
  namespace: esp
spec:
  hosts:
  - "*"
  gateways:
  - esp-gateway
  http:
  - match:
    - headers:
        client-id:
          exact: esp
    route:
    - destination:
        host: foo
		
		
		
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bar-vs
  namespace: esp
spec:
  hosts:
    - bar
  http:
  - match:
    - headers:
        client-id:
          exact: esp
    route:
    - destination:
        host: bar
        subset: v2

[rshriram]

@ymesika

[ymesika]

@my3sons It's quite difficult to track your issue as your foo service always return 404 even when calling it from within the mesh and the resources you shared doesn't match. Can you upload the full yamls for the app and the routing that you are trying to use?

Also, when you access the bar-vs from outside your cluster you need to specify a -HHost flag as described in the ingress tutorial.

[my3sons]

@ymesika Hi Yossi, I am not sure what you mean by the resources I shared don't match, but anyways I have uploaded all my yamls, they should be pretty self-explanatory, but please reach out if you have any questions. With respect to service bar, in my example, bar would never be called from outside the cluster, it would only be called by foo. FWIW, if I simply alter bar-vs.yaml to just route to destination bar (without client-id header), then when I call foo, I get back a successful response from bar.
Thanks for any help you can offer!
foo-bar.zip

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.