[Testing] postsubmit test failing on release-1.0 branch

[Bobgy]

What steps did you take:

Look at commit: bd7f3c9

What happened:

standalone component test failed:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/kubeflow-pipeline-postsubmit-standalone-component-test/1281172742669864961

Error message in the client:

sample-test-gf4sm-3534797757:
sample-test-gf4sm-3534797757: Traceback (most recent call last):
sample-test-gf4sm-3534797757: File "/python/src/github.com/kubeflow/pipelines/test/sample-test/sample_test_launcher.py", line 236, in
sample-test-gf4sm-3534797757: main()
sample-test-gf4sm-3534797757: File "/python/src/github.com/kubeflow/pipelines/test/sample-test/sample_test_launcher.py", line 232, in main
sample-test-gf4sm-3534797757: 'component_test': ComponentTest
sample-test-gf4sm-3534797757: File "/usr/local/lib/python3.7/dist-packages/fire/core.py", line 138, in Fire
sample-test-gf4sm-3534797757: component_trace = _Fire(component, args, parsed_flag_args, context, name)
sample-test-gf4sm-3534797757: File "/usr/local/lib/python3.7/dist-packages/fire/core.py", line 468, in _Fire
sample-test-gf4sm-3534797757: target=component.name)
sample-test-gf4sm-3534797757: File "/usr/local/lib/python3.7/dist-packages/fire/core.py", line 672, in _CallAndUpdateTrace
sample-test-gf4sm-3534797757: component = fn(*varargs, **kwargs)
sample-test-gf4sm-3534797757: File "/python/src/github.com/kubeflow/pipelines/test/sample-test/sample_test_launcher.py", line 173, in run_test
sample-test-gf4sm-3534797757: pysample_checker.check()
sample-test-gf4sm-3534797757: File "/python/src/github.com/kubeflow/pipelines/test/sample-test/run_sample_test.py", line 140, in check
sample-test-gf4sm-3534797757: response = self._client.wait_for_run_completion(self._run_id, self._test_timeout)
sample-test-gf4sm-3534797757: File "/usr/local/lib/python3.7/dist-packages/kfp-1.0.0rc4-py3.7.egg/kfp/_client.py", line 680, in wait_for_run_completion
sample-test-gf4sm-3534797757: self._refresh_api_client_token()
sample-test-gf4sm-3534797757: File "/usr/local/lib/python3.7/dist-packages/kfp-1.0.0rc4-py3.7.egg/kfp/_client.py", line 242, in _refresh_api_client_token
sample-test-gf4sm-3534797757: if getattr(self, '_is_refresh_token'):
sample-test-gf4sm-3534797757: AttributeError: 'Client' object has no attribute '_is_refresh_token'

The failing component's pod: https://storage.googleapis.com/kubernetes-jenkins/logs/kubeflow-pipeline-postsubmit-standalone-component-test/1281172742669864961/artifacts/pods_info/xgboost-trainer-f9sct-3589735778.txt

• image: gcr.io/ml-pipeline-test/ml-pipeline-gcp:bd7f3c9cd60144aef953db7aa2a4b17cec1cfa73.0.0-rc.4
  imageID: ""
  lastState: {}
  name: main
  ready: false
  restartCount: 0
  state:
  waiting:
  message: Back-off pulling image "gcr.io/ml-pipeline-test/ml-pipeline-gcp:bd7f3c9cd60144aef953db7aa2a4b17cec1cfa73.0.0-rc.4"
  reason: ImagePullBackOff

It seems the component image is broken.

What did you expect to happen:

it should pass

Environment:

KFP version: 1.0.0-rc.4

Anything else you would like to add:

It seems there are two orthogonal problems here:

1. The client crashed after 1 hour, @numerology I think this should have been fixed by your change. Can you double check?
2. The image url is corrupted, it's a bug in my release script. I'll fix it tomorrow.

/kind bug
/area testing

[Bobgy]

Also post mortem, I only looked at master branch postsubmit tests passed. However, master branch has been pinned to use 0.5.1 components. We should have updated its component reference to 1.0.0-rc.4 to catch up with latest changes.

The postsubmit tests on release-1.0 branch are catching real failures.

/assign @Bobgy @numerology

[Bobgy]

Hold on, I looked at component.yaml for 1.0.0-rc.4 release, it seems good https://github.com/kubeflow/pipelines/blob/1.0.0-rc.4/components/gcp/dataproc/create_cluster/component.yaml.

Maybe the problem is a wrong mechanism that didn't replace tags properly during postsubmit test. I'll look into that tomorrow. It could be not 1.0 blocking.

But we should still look at the client stacktrace.

[Bobgy]

/reopen

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Bobgy]

We haven't seen one commit with all postsubmit tests passing, but each test has passed recently in at least one commit.
I am confident we fixed the issues and the test is just still flaky right now.