Seldon-Core Quickstart is not working

[elia-secchi]

/kind bug

What steps did you take and what happened:
After Installing Kubeflow in GKE, everything works fine part from Seldon.
I am following the quickstart example in Kubeflow Docs so I am:

• adding the new namespace
• adding a new istio gateway
• deploying the dummy model
• port-forwarding

when I send my prediction with:

curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}'    -X POST http://localhost:8004/seldon/testseldon/seldon-model/api/v1.0/predictions    -H "Content-Type: application/json"

I get:
Origin authentication failed
Inspecting the Logs of istio ingress-gateway I can see that the returned error code is 401 - Unauthorized so I think the problem is at istio ingress-gateway level.

I've tried several times with different Kubeflow versions but I am still getting the same error.

Environment:

• Kubeflow version: 1.0.1
• kfctl version: kfctl v1.0.1-0-gf3edb9b
• Kubernetes platform: GKE
• Kubernetes version: 1.14.10

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/bug	0.95

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[dilzeem]

Are you using any kind of authentication service?

I posted a similar issue just now for AWS.

It says here for port forwarding that authentication can be an issue:
https://www.kubeflow.org/docs/components/central-dash/overview/#using-kubectl-and-port-forwarding

[elia-secchi]

Yes I am using Google Identity Aware proxy when interacting with the endpoint.

[elia-secchi]

I've found a solution for the problem.
The request needs to be sent with IAP authentication (JWT token).
For that is possible to use a Python script similar to what suggested in the tensorflow serving page:
https://www.kubeflow.org/docs/components/serving/tfserving_new/#sending-prediction-request-through-ingress-and-iap

Something similar to the page I linked above can be done also for the Seldon Quickstart.

[jtfogarty]

/area seldon
/priority p1

[jlewi]

/cc @cliveseldon

[ukclivecox]

I can update the docs but looks like a lot of that script is generic IAP processing. I could try to generalize it and add a Seldon specific addition?

[elia-secchi]

The only thing I've changed in the script is adding
Content: application/json to the headers of the request.

[vladislavkoz]

I'm getting the same issue even with the Python script. I've created a separate service account with access to all that is related to IAP. Could you please provide more details on how did you fix that?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/inference	0.53

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[vladislavkoz]

It's because of the wrong service account. You need to add more permissions. Try the following: IAP-secured Web App User,
ML Engine Admin