Error running pipeline: cannot create tfjobs.kubeflow.org 403

[lluunn]

Steps:

1. followed wiki to deploy pipeline.
2. I took the code from @amygdala's blog, ran it in jupyter notebook to build the tarball
3. Upload the tarball, and graph looks correct:

4. Running the pipeline, got error:

INFO:root:Getting credentials for GKE cluster kfp1.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for kfp1.
INFO:root:Generating training template.
INFO:root:Start training.
ERROR:root:Exception when calling DefaultApi->apis_fqdn_v1_namespaces_namespace_resource_post: tfjobs.kubeflow.org is forbidden: User "system:serviceaccount:kubeflow:pipeline-runner" cannot create tfjobs.kubeflow.org in the namespace "kubeflow"
Traceback (most recent call last):
  File "/ml/train.py", line 230, in <module>
    main()
  File "/ml/train.py", line 188, in main
    create_response = tf_job_client.create_tf_job(api_client, content_yaml, version=kf_version)
  File "/tf-operator/py/tf_job_client.py", line 56, in create_tf_job
    raise e
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 15 Nov 2018 22:32:05 GMT', 'Audit-Id': '316cc166-62f2-43aa-926e-8014f66f4b2e', 'Content-Length': '318', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"tfjobs.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:pipeline-runner\" cannot create tfjobs.kubeflow.org in the namespace \"kubeflow\"","reason":"Forbidden","details":{"group":"kubeflow.org","kind":"tfjobs"},"code":403

[amygdala]

I think the issue is that you need to run this command or similar before running any pipelines:
kubectl create clusterrolebinding sa-admin --clusterrole=cluster-admin --serviceaccount=kubeflow:pipeline-runner
(there is another issue filed related to this... let me find it).

[amygdala]

Here's the related issue: #220

[lluunn]

Thanks!
I created the clusterrolebinding.
Now the error is:
py.util.JobTimeoutError: Timeout waiting for job trainer-s46w4 in namespace kubeflow to enter one of the conditions [{u'status': u'True', u'lastUpdateTime': u'2018-11-16T04:11:13Z', u'lastTransitionTime': u'2018-11-16T04:11:13Z', u'reason': u'TFJobCreated', u'message': u'TFJob trainer-s46w4 is created.', u'type': u'Created'}, {u'status': u'True', u'lastUpdateTime': u'2018-11-16T04:11:59Z', u'lastTransitionTime': u'2018-11-16T04:11:13Z', u'reason': u'TFJobRunning', u'message': u'TFJob trainer-s46w4 is running.', u'type': u'Running'}]

[amygdala]

Hmm, I have not seen this (and can't repro). :(
Do the logs of the tf-job-operator* pod have any clues?

Jeremy, I have a vague memory of your saying there was some recent issue with tf-job -- could this be related? @jlewi

[jlewi]

@amygdala Only issues I'm aware of are RBAC and IAM.

@IronPan might know more.

[jlewi]

@IronPan @vicaire Any update on this?

[vicaire]

@qimingj @gaoning777, could you please have a look?

[jlewi]

@qimingj @gaoning777 any update on this?

[IronPan]

The RBAC issue should be resolved with the latest version.
https://github.com/kubeflow/kubeflow/blob/master/kubeflow/pipeline/pipeline-apiserver.libsonnet#L322

@lluunn Could you give another try and if issue still persist, could you share me the pipeline definition and the kubeflow version?

[jlewi]

@IronPan Is there a pipelines test that covers firing off TFJob from pipelines? If not can we open up an issue to add such at test and use that to verify the fix is working?

[qimingj]

We currently don't have any samples covering tf-job except for @amygdala's sample. The sample's trainer container (gcr.io/google-samples/ml-pipeline-kubeflow-tf-taxi) was contributed by Amy and is out of pipeline's repo either.

@amygdala, does your sample still work in latest kubeflow deployment?

I am up for covering tf-job since this is the key component in Kubeflow.

[amygdala]

Barbara reported that there is a credentials issue (see below) with running my tf-job step in her setup, which was created using the launcher (in contrast to my original instructions, in which I created the GKE cluster nodes with cloud-platform scope, then ran the bootstrapper.yaml).
I'm about to try to repro.
She reports that the issue is NOT fixed by adding that gcp credentials wrapper to the step, but it might be a clusterrole binding issue.

(I remember that previously, there was some situation where I needed to run:
kubectl create clusterrolebinding sa-admin --clusterrole=cluster-admin --serviceaccount=kubeflow:pipeline-runner
to fix something, but I can't remember the context.)

[amygdala]

Update part 1: I had no problems with tf-job using a cluster created with 'cloud-platform' scoped nodes and this pipelines bootstrapper: gcr.io/ml-pipeline/bootstrapper:0.1.7 .
So this suggests that it's some credentials/scope setup issue rather than some recent release change, which I wanted to rule out first.

Next I'll play around with a launcher-created cluster. Maybe it needs some additional cluster role during setup.

(cc @BasiaFusinska )

[IronPan]

I think the issue is caused by tf-job pod failed to talk to gcp services because tf-job pod doesn't have right GCP service account set up in launcher.

pipelines/components/kubeflow/launcher/src/launch_tf_job.py

Line 52 in df635af

content = yaml.load(f)

We need to mount the kubeflow-user GCP service account here.

[amygdala]

You're probably right, I'll try that next.

[vicaire]

Closing this issue as a duplicate of #677