Unable to get basic sample app to work with OPA envoy plugin for Istio

[kyle-burnette]

Hello,
I was just trying to get my feet wet with OPA plugin for Istio and did the following:

Am doing this in Azure AKS. I first installed Istio in my cluster which went fine. I then installed Istio's sample bookinfo application in the cluster with Istio's envoy/sidecar injection enabled in default namespace. Confirmed the application worked as expected. Exposed it externally and was able to access it without any issues.

Removed the sample all completely before installing OPA envoy plugin.
I then proceeded to install & deploy opa's envoy plugin for Istio. Used the guide at https://github.com/open-policy-agent/opa-envoy-plugin/tree/master/examples/istio. Followed the steps in Quick Start. There were no errors in applying the quick_start.yaml as listed in the guide. Confirmed it created all resources that it should have like the ext-authz EnvoyFilter and admission controller under opa-istio namespace. Enabled opa-istion injection using "kubectl label namespace default opa-istio-injection="enabled""

Proceeded to deploy the sample bookinfo application and the accompanying bookinfo-gateway. Both were deployed without any errors. Saw the app was deploy and all pods/services were created and started correctly.

Expected Behavior

After exposing the app/service externally, attempted accessing it using
curl --user alice:password -i http://$GATEWAY_URL/productpage that I expected to work.

Actual Behavior

But get this error instead:

HTTP/1.1 503 Service Unavailable
content-length: 19
content-type: text/plain
date: Tue, 27 Oct 2020 17:54:13 GMT
server: istio-envoy

no healthy upstream

Even when attempting to access the productpage from inside the cluster using the following cmd get the same error:
kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -s productpage:9080/productpage.

If I remove opa-istio plugin completely including the ext-authz filter and all resources in opa-istio namespace, then the sample app becomes accessible without any errors.

Steps to Reproduce the Problem

In AKS cluster, install latest version of Istio 1.7.3 using default profile. Enable envoy injection using " kubectl label namespace default istio-injection=enabled". Follow the instructions on https://github.com/open-policy-agent/opa-envoy-plugin/tree/master/examples/istio.
Attempt accessing the productpage as indicated.

Am I missing something?

• OPA version - 0.24?

Pl see above. thanks
Kyle

[ashutosh-narkar]

It would be helpful to get logs from the product page, envoy and OPA containers from the product page pod to debug this further. Also make sure OPA is deployed inside every app pod.

[kyle-burnette]

I am in process of creating a new AKS cluster to start everything afresh (but waiting for our CloudOps person to do since I don't have permission to create one).
Just wanted to give heads up on this delay but will get back as soon as possible.

[kyle-burnette]

I repeated on a new AKS cluster. I see the error in envoy container but don't know the cause and how to fix it. Attached pl see all logs.
thanks,

istio-proxy.log
opa-istio.log
productpage.log

[ashutosh-narkar]

Apart from warnings, I don't see an error in the Envoy logs. OPA seems to be started as well. The logs don't show an actual request destined for the product page app being handled by OPA. Do you have that ?

[kyle-burnette]

I attempted accessing productpage both from within the cluster and externally but it does not seem like the request even makes it. There is no record of it anywhere in envoy, opa or productpage. Am attaching them but not sure how useful they would be.
istio-proxy.log
opa-istio.log
productpage.log

I then attempted the following:

1. Spun up a new cluster.
2. Install bookinfo app w/o anything else. Confirmed it was working and could be accessed fine.
3. Deleted the sample app and install Istio 1.7.3.
4. Create a "istio-only-test" ns and deployed the sample bookinfo app inside it. Enabled istio-injection in this namespace.
5. Confirmed the bookinfo in "istio-only-test" ns worked correctly and could be accessed.
6. Installed opa-istio plugin per https://github.com/open-policy-agent/opa-envoy-plugin/tree/master/examples/istio. Confirmed it created all resources under istio-system and opa-istio.
7. Created a new ns "opa-istio-test", enabled both opa-istio-injection and istio-injection in this ns, and deployed the sample bookinfo app in this ns.
8. But now even after waiting more than 30 mins after the deployment, the app's pods won't start. It does not even say CrashLoopBackoff on the pods status as it would typically do when there are issues. I cannot explain this as the last time around - at least the pods would come up.
9. Also after this the sample app in "istio-only-test" ns stopped working. New attempts to access the app do not even get recorded in either envoy or productpage containers - the logs show the same old entries from access attempts during step Add parser test #5 above.

[ashutosh-narkar]

The opa-envoy plugin seems to not be working with Istio v1.7.0+. With the 1.7 release, the OPA heath checks are passed through istio-proxy and then sent to OPA resulting in OPA returning a 403. So the health checks fail and the pods keep restarting.

[kyle-burnette]

Ouch! Is this something Istio needs to do something about that I can follow up with them? Or can OPA do something to resolve it (in future)?
I guess I can try with Istio v < 1.7. OR may be I can just use envoy w/o Istio?

thanks

[ashutosh-narkar]

The opa-envoy plugin will work with v1.6.x. For Istio v1.7.x, we could add a Rego policy to allow health checks as a workaround but it would be better if there was a way to configure the behavior in the EnvoyFilter config. There seems to open issue related to this.

[kyle-burnette]

I tested with Istio v1.6.13 and everything in OPA envoy plugin for Istio's quick start worked as expected. Thanks!

I want to update the simple policies to recognize JWT tokens and do some authorizations using certain claims in it. Time to read up on Rego syntax :-) Will see how it goes.

[shashanksrivastava10]

Can someone please help me out with opa rego policy to allow health checks to run it with 1.7.0 as mentioned above by @ashutosh-narkar , apologies but im am relatively new to opa and could not find an example online.

Regards.

[ashutosh-narkar]

@shashanksrivastava10 you would need to add an allow rule that allows the health check path. You can find the exact path from the OPA logs. There is an open issue in Istio to address this fwiw.

[shashsrivastava]

Hi @ashutosh-narkar, so i recently made changes to the rego policy that is available as a configmap in my target application namespace and it looks like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: opa-policy
namespace: liveness
data:
policy.rego: |
package istio.authz

import input.attributes.request.http as http_request

default allow = false

allow {
    input.method == "GET"
    input.path == "/health?plugins"
}

allow {
    input.method == "GET"
    input.path == "/healthz/ready"
}

allow {
    roles_for_user[r]
    required_roles[r]
}

roles_for_user[r] {
    r := user_roles[user_name][_]
}

required_roles[r] {
    perm := role_perms[r][_]
    perm.method = http_request.method
    perm.path = http_request.path
}

user_name = parsed {
    [_, encoded] := split(http_request.headers.authorization, " ")
    [parsed, _] := split(base64url.decode(encoded), ":")
}

user_roles = {
    "alice": ["guest"],
    "bob": ["admin"]
}

role_perms = {
    "guest": [
        {"method": "GET",  "path": "/productpage"},
    ],
    "admin": [
        {"method": "GET",  "path": "/productpage"},
        {"method": "GET",  "path": "/api/v1/products"},
    ],
}

But still im facing the same issue when i describe the pods :
Warning Unhealthy 10s (x2 over 20s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 403
Warning Unhealthy 6s (x2 over 16s) kubelet Liveness probe failed: HTTP probe failed with statuscode: 403

Any suggestions from you would be highly appreciated, rest of the configurations remain the same as mentioned in istio-opa quickstart guide.

[ashutosh-narkar]

Check the OPA logs to get the actual input OPA gets. Then take your policy and the OPA input and write some unit tests. This will help in faster debugging.

[srenatus]

👉 examples/istio/quick_start.yaml is updated for Istio 1.7.0 in open-policy-agent/opa-envoy-plugin#217

[srenatus]

🤔 I think this can be closed. If I'm mistaken, please re-open 😃