End-to-end Kubeflow on AWS: Authenticating to KFServing via Cookie

[karlschriek]

I am following this guide and am attempting to authenticate KFServing using the AWSELBAuthSessionCookie-0 cookie. This is on a KF 1.0.2 rollout using KFServing 0.3.0.

https://www.kubeflow.org/docs/aws/aws-e2e/

I am rolling out the example iris model in the karl-schriek namespace, resulting in:

NAME              URL                                                                                              READY   DEFAULT TRAFFIC   CANARY TRAFFIC   
sklearn-iris      http://sklearn-iris.karl-schriek.dev-kubeflow-102.learn-mlops.com/v1/models/sklearn-iris         True    100                

I then proceed to use python's requests package to perform the POST as describe in the documentation:

host="sklearn-iris.karl-schriek.dev-kubeflow-102.learn-mlops.com"
endpoint=f"https://{host}/v1/models/sklearn-iris"
inference_url=f"{endpoint}:predict"
cookies = {"AWSELBAuthSessionCookie-0": "hjF4O6cvQHSp/...wjZACCyWe"}

data = json.dumps({
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
})

headers = {"Content-Type": "application/json", "Host": host}
response = requests.post(
    inference_url, 
    data=data, 
    headers=headers, 
    cookies=cookies
)
response.raise_for_status()

However, this results in the following error:

HTTPError: 403 Client Error: Forbidden for url: https://auth.dev-kubeflow-102.learn-mlops.com/oauth2/authorize?client_id=16008qhlkvbt0nnvoflb4e229l&redirect_uri=https%3A%2F%2Fsklearn-iris.karl-schriek.dev-kubeflow-102.learn-mlops.com%2Foauth2%2Fidpresponse&response_type=code&scope=openid&state=pb6uL%2Fcs5WwoUqrzoPWj5Qc6nhwom%2BwTxFLA0Ngu5I47PmWD0jTH1fZXxVw%2FZTDMQpUhjHk6jHMKmLRfIF2b%2FMAr%2BwlQ2uTXJyWLThJ409FBrlxMzfmFUPHFsHAM3AwkGMbzLwYHbEUhJgQuacJrr3BM6VBXfrUCAGru%2BFvl1OqtJF9%2BLWb6J1yT8IKilZMf%2F1%2BEhyYJqVfMp7cMjl2qHub5mb2moNc07wzwvfnfzEboQ%2FXk6fg%3D

My setup looks as follows:

• EKS 1.15, KF 1.0.4, KFServing 0.3.0
• In AWS Certifcate Manager, my certificate has the entry *.karl-schriek.dev-kubeflow-102.learn-mlops.com (in the guide "default" is used instead of "karl-schriek")
• In Route53, the hosted zone learn-mlops.com hast the following entries:

1. www.dev-kubeflow-102.learn-mlops.com (A-Record) pointing to the ELB (in the guide, "kubeflow" is used instead of "www")
2. *.karl-schriek.dev-kubeflow-102.learn-mlops.com (A-Record) pointing to the ELB
3. auth.dev-kubeflow-102.learn-mlops.com pointing to the CloudFront Distribution for the Cognito App

• In Cognito, the App and domain have been set up with:

1. Callback url is set to https://www.dev-kubeflow-102.learn-mlops.com/oauth2/idpresponse
2. Domain name: auth.dev-kubeflow-102.learn-mlops.com
3. Certificate points to one in us-east-1

I cannot find anything here that does not adhere to what is written in the guide. Am I missing something or is there something more that needs to be done that is not covered in the guide?

UPDATE

For clarity, the result is the same when using host="sklearn-iris-predictor-default.karl-schriek.dev-kubeflow-102.learn-mlops.com" or inference_url="https://www.dev-kubeflow-102.learn-mlops.com/v1/models/sklearn-iris:predict"

[theofpa]

You also need to add https://sklearn-iris.karl-schriek.dev-kubeflow-102.learn-mlops.com/oauth2/idpresponse in the callback URLs of the app client settings in cognito.

[karlschriek]

I have now set all of the following callback URLs:

https://sklearn-iris-predictor-default.karl-schriek.dev-kubeflow-102.learn-mlops.com/oauth2/idpresponse
https://sklearn-iris.karl-schriek.dev-kubeflow-102.learn-mlops.com/oauth2/idpresponse
https://www.dev-kubeflow-102.learn-mlops.com/oauth2/idpresponse

No change in the error message. Also, if this is needed, was it simply an oversight it was not noted in the guide?

[seizadi]

I opened an issue for updating the documentation for AWS Cognito use case, here: https://github.com/kubeflow/kfserving/issues/1233

[8bitmp3]

/assign @Jeffwan @PatrickXYS

(AWS)

[karlschriek]

Ok, so after going through various steps to finally upgrade to KF 1.2 (and to KFServing 0.4.1 with Istio 1.3.1) I am finally at the point where I make further progress here.

Aside from upgrading to KF 1.2 / KFServing 0.4.1/ Istio 1.3.1, I am no also including both cookies in the request. I am not sure if that is what is mainly responsible for the change, but it could be. That was also the case for authenticating in KFP and having both cookies solved it there.

Anyway, I am now using

cookies = {
    "AWSELBAuthSessionCookie-0": "hjF4O6cvQHSp/...wjZACCyWe", 
    "AWSELBAuthSessionCookie-1": "yccvsfG<sfv/...rvasdf" 
}

That gets me one step further. Auth is working, but I then get the error :

Exception has occurred: HTTPError
404 Client Error: Not Found for url: https://sklearn-iris.karl-schriek.dev-kubeflow-120.learn-mlops.com/v1/models/sklearn-iris:predict

Which I think means I am now at the point where @seizadi was when he opened this issue:(kserve/kserve#1154).

I am closing this and will see if I can get the approach in https://github.com/kubeflow/kfserving/issues/1233 working for me. Happy also to contribute documentation if I can get it to work.