GCS is not accessible from pipeline components

[nihil0]

What steps did you take:

On executing the sample, the gcs-download step of the pipeline fails. A screenshot of the error message is attached. The input to this pipeline is a bucket which I have full control over, with the shakespeare1.txt file uploaded.

What happened:

Unexpected error when downloading from GCS in sample pipelines

What did you expect to happen:

Pipeline should execute successfully

Environment:

How did you deploy Kubeflow Pipelines (KFP)?

CLI deployment on GCP

KFP version:

ca58b22

KFP SDK version:

0.3.0

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug

// /area backend

[nihil0]

This might have something to do with Workload identity not being available in Kubeflow Pipelines when installed as part of Kubeflow. I don't know any workarounds for this. This issue might be related: #2773

I did the following and the pipeline failed with different error messages.

1. Enabled Uniform Bucket Level Access

Received an error to the effect that the workload identity does not have sufficient privileges over the bucket. I then noticed that the pipeline-runner Kubernetes service account (KSA) was not bound to the workspace identity GCP service account (GSA).

AccessDeniedException: 403 Primary: /namespaces/neaas-test.svc.id.goog with additional claims does not have storage.objects.list access to neelabh-dev.

2. Added annotation to bind pipeline-runner KSA to workload identity GSA.

Error message:

Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 289, in DecoratedFunction
    sleep_ms=sleep_ms)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 178, in RetryOnException
    TryFunc, should_retry_if=should_retry, sleep_ms=sleep_ms)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 236, in RetryOnResult
    self._RaiseIfStop(result, state)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 108, in _RaiseIfStop
    raise MaxRetrialsException('Reached', result, state)
googlecloudsdk.core.util.retry.MaxRetrialsException: last_result=(None, (<class 'googlecloudsdk.core.credentials.gce.MetadataServerException'>, MetadataServerException('The request is rejected. Please check if the metadata server is concealed.\nSee 
https://cloud.google.com/kubernetes-engine/docs/how-to/protecting-cluster-metadata#concealment
 for more information about metadata server concealment.'), <traceback object at 0x7fa762449b08>)), last_retrial=3, time_passed_ms=3032,time_to_wait=0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/bin/../lib/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 123, in <module>
    exceptions.HandleError(e, 'gsutil')
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/exceptions.py", line 527, in HandleError
    core_exceptions.reraise(exc)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 146, in reraise
    six.reraise(type(exc_value), exc_value, tb)
  File "/usr/lib/google-cloud-sdk/lib/third_party/six/__init__.py", line 693, in reraise
    raise value
  File "/usr/bin/../lib/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 121, in <module>
    main()
  File "/usr/bin/../lib/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 35, in main
    project, account = bootstrapping.GetActiveProjectAndAccount()
  File "/usr/lib/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 298, in GetActiveProjectAndAccount
    account = properties.VALUES.core.account.Get(validate=False)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/properties.py", line 2093, in Get
    required)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/properties.py", line 2392, in _GetProperty
    value = _GetPropertyWithoutDefault(prop, properties_file)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/properties.py", line 2430, in _GetPropertyWithoutDefault
    value = callback()
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py", line 263, in GetAccount
    return c_gce.Metadata().DefaultAccount()
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/credentials/gce.py", line 88, in Inner
    return f(self, *args, **kwargs)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/credentials/gce.py", line 131, in DefaultAccount
    http_errors_to_ignore=(404,))
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 292, in DecoratedFunction
    exceptions.reraise(to_reraise[1], tb=to_reraise[2])
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 146, in reraise
    six.reraise(type(exc_value), exc_value, tb)
  File "/usr/lib/google-cloud-sdk/lib/third_party/six/__init__.py", line 693, in reraise
    raise value
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 162, in TryFunc
    return func(*args, **kwargs), None
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/credentials/gce.py", line 60, in _ReadNoProxyWithCleanFailures
    'The request is rejected. Please check if the metadata server is '
googlecloudsdk.core.credentials.gce.MetadataServerException: The request is rejected. Please check if the metadata server is concealed.
See 
https://cloud.google.com/kubernetes-engine/docs/how-to/protecting-cluster-metadata#concealment
 for more information about metadata server concealment.

[Bobgy]

You are right, that's a duplicate of #2773.
The fix will be released in Kubeflow 1.0.2

[Bobgy]

/close

[k8s-ci-robot]

@Bobgy: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.