Patch Storage Bucket IAM Policy

[quentin-sensome]

I am trying to patch a storage bucket's IAM policy using the storage's GCP Type Provider. However the Cloud Storage JSON API answers back with an error 400: Bad Request.

I based my template on an example that updates IAM policies at the project level. The example can be found here: https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/master/google/resource-snippets/cloudresourcemanager-v1/patch_iam_project.yaml

This is what my template looks like:

"""Patch IAM Policy for a Storage Bucket"""

def GenerateConfig(context):

  resources = [{
    'name': 'get-bucket-iam-policy',
    'action': 'gcp-types/storage-v1:storage.buckets.getIamPolicy',
    'metadata': {
      'runtimePolicy': ['UPDATE_ALWAYS']
    },
    'properties': {
      'bucket': 'MY_BUCKET'
    }
  }, {
    'name': 'patch-bucket-iam-policy'
    'action': 'gcp-types/storage-v1:storage.buckets.setIamPolicy',
    'properties': {
      'bucket': 'MY_BUCKET',
      'policy': '$(ref.get-bucket-iam-policy)',
      'gcpIamPolicyPatch': {
        'add': [{
          'role': 'roles/storage.admin',
          'members': [ 'serviceAccount:ROBOT_SERVICE_ACCOUNT' ]
        }]
      }
    }
  }]

  return {'resources': resources} 

When executing the deployment, this is the answer I receive from DM:

ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1538660104879-5776738fb3b99-2d8cdc3d-a8609678]: errors:
- code: RESOURCE_ERROR
  location: /deployments/MY_DEPLOYMENT/resources/patch-bucket-iam-policy
  message: '{
  "ResourceType":"gcp-types/storage-v1:storage.buckets.setIamPolicy",
  "ResourceErrorCode":"400",
  "ResourceErrorMessage":{
    "code":400,
    "errors":[{
      "domain":"global",
      "message":"A policy to update must be provided.",
      "reason":"required"
    }],
    "message":"A policy to update must be provided.",
    "statusMessage":"Bad Request",
    "requestPath":"https://www.googleapis.com/storage/v1/b/MY_BUCKET/iam",
    "httpMethod":"PUT"
  }
}'

As you can see, the policy object present in the JSON does not seem to be picked up by DM. I tested the get-bucket-iam-policy operation by exposing its output and it seems fine to me:

OUTPUTS           VALUE
bucket-iam-policy[resourceId]  projects/_/buckets/MY_BUCKET
bucket-iam-policy[kind]        storage#policy
bucket-iam-policy[etag]        CAE=
bucket-iam-policy[bindings]    [
  {u'role': u'roles/storage.legacyBucketOwner', u'members': [u'projectEditor:MY_PROJECT', u'projectOwner:MY_PROJECT']},
  {u'role': u'roles/storage.legacyBucketReader', u'members': [u'projectViewer:MY_PROJECT']}
]

I couldn't find any example on how to make this work, and couldn't find documentation on this subject either. The Cloud Storage JSON API doesn't help much, it only suggests to pass a Policy entity in the request parameter when calling the setIamPolicy method.

I suspect that DM actually expects the Policy entity to be passed in another parameter, but again I couldn't find documentation on this anywhere.

Is there any information out there that could help me achieve this ?

Thanks

[ocsig]

As a quick response:
I suggest you to check this implementation:
https://github.com/GoogleCloudPlatform/deploymentmanager-samples/tree/master/community/cloud-foundation/templates/gcs_bucket

We will look into this template as well.

[quentin-sensome]

Thanks for answering so fast.

The template you linked was exactly what I was looking for, thanks for sharing it !

[raelga]

@quentin-sensome have you been able to implement the append to a binding from a getIamPolicy? Wisdom of the Ancients

I'm trying to add pull permissions to a service account on a GCR bucket, but the only way I've been able to implement this binding is by updating the entire binding list.

{% set NAME = properties.gcr | default(env.project) %}

{% if properties.region %}
  {% set BUCKET = "{}.artifacts.{}.appspot.com".format(properties.region, NAME) %}
{% else %}
  {% set BUCKET = "artifacts.{}.appspot.com".format(NAME) %}
{% endif %}

{% if properties.role == 'pull' %}
  {% set ROLE = "roles/storage.objectViewer" %}
{% elif properties.role == 'push' %}
  {% set ROLE = "roles/storage.admin" %}
{% endif %}

resources:

  - name: {{ NAME }}-gcr-bucket-get-iam-policy
    action: gcp-types/storage-v1:storage.buckets.getIamPolicy
    properties:
      bucket: {{ BUCKET }}
    metadata:
      runtimePolicy:
        - UPDATE_ALWAYS

  {# https://cloud.google.com/container-registry/docs/access-control #}
  - name: {{ NAME }}-gcr-bucket-set-iam-policy
    action: gcp-types/storage-v1:storage.buckets.setIamPolicy
    properties:
      bucket: {{ BUCKET }}
      bindings:
        - role: {{ ROLE }},
          members: [ {{ properties.serviceAccount }} ]

I have no idea how to get the $(ref.{{ NAME }}-gcr-bucket-get-iam-policy.bindings) and append a second list of bindings with Jinja.

[ocsig]

There is a getIamPolicy > setIamPolicy example here:
https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/master/community/cloud-foundation/templates/project/project.py

The Deployment Manager team is working on an update which may solves your problem without using getIamPolicy at all. ETA before the end of the year.

I highly recommend you to switch from Jinja to Python. Your marcos will start to be way too complex while limiting you compare to python.

[raelga]

Thanks for the fast response!

Using the same method as with project IAM bindings doesn't seems to be working as stated in the initial issue report. Just in case, I tried but is failing with the A policy to update must be provided. error message:

Deploying:

  - name: {{ NAME }}-gcr-bucket-set-iam-policy
    action: gcp-types/storage-v1:storage.buckets.setIamPolicy
    properties:
      bucket: {{ BUCKET }}
      policy: $(ref.{{ NAME }}-gcr-bucket-get-iam-policy)
      gcpIamPolicyPatch:
        add:
          - role: {{ ROLE }}
            members: 
              - {{ properties.serviceAccount }}

Returns:

ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1544132263835-57c614f127579-ecf78f32-c79c6a04]: errors:
- code: RESOURCE_ERROR
  location: /deployments/bigotes-pro-bonnie-cluster/resources/bigotes-pro-gcr-bucket-set-iam-policy
  message: '{"ResourceType":"gcp-types/storage-v1:storage.buckets.setIamPolicy","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"errors":[{"domain":"global","message":"A
    policy to update must be provided.","reason":"required"}],"message":"A policy
    to update must be provided.","statusMessage":"Bad Request","requestPath":"https://www.googleapis.com/storage/v1/b/eu.artifacts.bigotes-pro.appspot.com/iam","httpMethod":"PUT"}}'

make: *** [dm-update-bigotes-pro-bonnie-cluster] Error 1

Just in case may be useful, the templates used are:

https://github.com/raelga/bigot.es/blob/master/gcp/templates/storage/bucket.jinja
https://github.com/raelga/bigot.es/blob/master/gcp/templates/storage/gcr-bucket-iam-policy.jinja

I wanted to add the GKE Nodes SA as viewer to be able to pull images, but as a resource of the cluster deployment. This way, if the cluster deployment is removed, the IAM binding is removed.

https://github.com/raelga/bigot.es/blob/master/gcp/deployments/bigotes-pro/bonnie-cluster.yaml#L38

I ended adding the binding manually in the bucket deployment, otherwise is removing any other binding:

https://github.com/raelga/bigot.es/blob/master/gcp/deployments/bigotes-pro/bonnie-cluster.yaml#L38

But I would like to add the binding in the cluster deployment, as I'm already doing with the nodes role in:

https://github.com/raelga/bigot.es/blob/master/gcp/templates/container/kubernetes-cluster.jinja#L43

Glad to hear that the DM team is working on the issue.

About using python, I know it may be more powerful but so far I've been able to do everything with jinja. And to be honest, I find jinja templates prettier and cleaner than appending jsons with python.

[cytar]

hi, we are facing exactly the same problem, any update from google about that ?

[ocsig]

Apologies for not updating this thread.

The product team released a new feature:
google/resource-snippets/cloudresourcemanager-v1/policies.jinja

I keep this issue open for myself to release a tutorial for your issue, while I'm working on it, feel free to try it out yourself.

The key is the type: gcp-types/cloudresourcemanager-v1:virtual.projects.iamMemberBinding line.

[raelga]

Thanks for the update, looking forward to test it.

[cytar]

it's related to project iam bindings, not storage iam bindings ? Is there an equivalent for storage ?

[ocsig]

We are working on adding further 'virtual' endpoints for other bindings during Q2.

[lukeFalsina]

Hello @ocsig,
I just hit the very same issue described by @raelga. Any update on the virtual endpoint for storage?
Thanks!

[Cicatrice]

Helllo

I reached also that particular trouble with storage.buckets.setIamPolicy asking me for policy parameter, ignoring the one I'm passing to (exactly like in your example).
I tried to forge some templates with iamMemberBinding but sadly, there is nothing released yet for storage.

And I noticed a behavior difference between setting up by storage console/gsutil or setting up by IAM (via DM or project-wide) :
My project contains a GKE with a dedicated service account. I want this particular GKE to be able to deploy custom private images from my private GCR. If I give inherited project-wide permission role/storage.objectViewer to this service account, I cannot achieve PullImage action on Kubernetes, and it failes with ImagePullBackOff.

But, if I manually set the bucket policy using gsutil or console, I can actually pull the image and run the pod.

When I export the policy with gsutil iam get gs://<my_bucket> I see the exact same policy in both cases. the only difference (not visible) is that in first case, the permission is inherited because it is project-wide.

[ocsig]

@lukeFalsina with some delay we managed to push virtual.folders.iamMemberBinding and virtual.organizations.iamMemberBinding to production. Currently working on snipets and Cloud Foundation Toolkit integration

Our next target is the gcp-types/storage-v1:virtual.storage.buckets.* types.

[ihachani]

Hi,
Any update on this. I am using thishttps://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/master/community/cloud-foundation/templates/gcs_bucket/gcs_bucket.py and the bucket created doesn't inherit default iam policies.

[AliGouta]

any update on this ?

[ocsig]

@ihachani please use our latest template: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/dm/templates/gcs_bucket

@AliGouta gcp-types/storage-v1:virtual.storage.buckets.iamMemberBinding is rolling out to Staging today, we are planning to roll out to production soon, possibly next week.

[radevm]

Any news about the rollout to PROD?

I'm eager to test if it'll solve my issue (https://stackoverflow.com/questions/56759231/gcp-grant-a-service-account-permission-to-write-in-a-gcs-bucket-with-deployment).

[dinvlad]

+1, we'd be eager to test gcp-types/storage-v1:virtual.storage.buckets.iamMemberBinding as well!

[ocsig]

The feature passed reviews, should be part of this weeks rollout, checking back on Friday.

gcp-types/storage-v1:virtual.buckets.iamMemberBinding is the correct syntax.

[ocsig]

I am happy to announce the gcp-types/storage-v1:virtual.buckets.iamMemberBinding type.

We updated the CFT Bucket template as well.

[ihachani]

@ocsig Is there a usage example for the new types?
Thanks this is time saver.

[ocsig]

CFT IAM member binding template supports storage from now on.

Example

Is that what you were looking for?

[dinvlad]

This is really helpful! We've used it in one of our templates.
However, when I changed the bucket for the binding, Deployment Manager failed with:

- code: NO_METHOD_TO_UPDATE_FIELD
  message: No method found to update field 'bucket' on resource 'gcr-permission'
    of type 'storage-v1'. The resource may need to be recreated with the new field.

So atm it seems that the only way to update the bucket name is to rename the resource (which works).

[ocsig]

Changing type is not supported, also there are properties which can't be changed via DM or the API, only by deletion and recreation of the resource. By renaming the DM resource, it looks like you deletede the old one and created a new one and thats what DM will do. This can be inconvenient, I am pushing the DM team to support delete+recreate based updates where its possible.

[skjack3]

Hi , i am trying to set iam policy on gcs bucket but with condition , although https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/dm/templates/iam_member creating the permission but i want to add condition to allow only specific folder , can any tell how to do it

[cytar]

Hi, as far as i know, you must do it with bucket acl, not from iam policies.

1. set your bucket as fine grained (if uniform)
2. use gsutil acl (you can have more informations from cloud console)

[skjack3]

I've tried from bucket IAM permission and restricted the access but it seems i am condition not supoorted yet in deployment manager ,
now i want to give service account permission to user but still i am not able to find any way to do it via DM , it can be done easily via this command or via console
gcloud iam service-accounts add-iam-policy-binding <SERVICE_ACCOUNT>
--member="user:<USER_ACCOUNT>"
--role="roles/iam.serviceAccountTokenCreator"