Permissions problem trying to create a sink

[tseaver]

While working on a system test for Sink.create, I'm running into a permissions problem:

>>> sink.create()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "gcloud/logging/sink.py", line 145, in create
    client.connection.api_request(method='PUT', path=self.path, data=data)
  File "gcloud/connection.py", line 343, in api_request
    error_info=method + ' ' + url)
gcloud.exceptions.Forbidden: 403 The caller does not have permission (PUT https://logging.googleapis.com/v2beta1/projects/citric-celerity-697/sinks/test-sink)

Sinks.create is documented to require one of https://www.googleapis.com/auth/logging.admin or https://www.googleapis.com/auth/cloud-platform.

My client is set up with all permissions for the logging API:

>>> client.connection._credentials._scopes
'https://www.googleapis.com/auth/logging.read https://www.googleapis.com/auth/logging.write https://www.googleapis.com/auth/logging.admin https://www.googleapis.com/auth/cloud-platform'

Do I have to do something in the developer control panel to enable sinks?

[tseaver]

@jgeewax please loop in whoever would be able to point me in the right direction from the logging API team.

[tseaver]

Maybe germane: https://cloud.google.com/logging/docs/export/configure_export#exported-logs-permissions

[tseaver]

More likely an error on my part: the failed request was trying to do a PUT to the URL of the sink, which doesn't yet exist, instead of a POST to the /sinks quasi-container.

[tseaver]

But I'm betting that when I do get that part right, then the 403 will trigger because I don't have the bucket permissions set correctly.

[tseaver]

Grrr. Switching to POST doesn't fix it:

gcloud.exceptions.Forbidden: 403 The caller does not have permission (POST https://logging.googleapis.com/v2beta1/projects/citric-celerity-697/sinks)

That same URI works fine in the API Exploder.

[dhermes]

@jgeewax Any ideas?

[dhermes]

@callmehiphop @stephenplusplus @blowmage have you run into this?

[stephenplusplus]

Yes! Check this out: googleapis/google-cloud-node#1156

[stephenplusplus]

Related Ruby issue: googleapis/google-cloud-ruby#569

[blowmage]

IIRC the Pub/Sub IAM issue is different. The issue with creating Logging Sinks is that it requires a user account, not a service account. This is imposed by the API, and there isn't much we can do to get around it. @quartzmo, does that sound right?

[stephenplusplus]

Ah, my bad. We were getting errors creating Sinks with a Pub/Sub Topic as a destination, which traces back to the Pub/Sub IAM change, so I jumped at a connection between that issue and this one.

Indeed, you must be authenticated as a user (through the gcloud SDK works). Here's the official word from a Logging member: googleapis/google-cloud-node#842 (comment)

[blowmage]

There are some additional steps for creating a sink to a Bucket, and @quartzmo was chasing that info down for our docs, so I'll defer to him on that as well. 😅

[quartzmo]

Unfortunately I haven't gotten it working yet, but I'll try again today.

On Mar 16, 2016, at 7:02 AM, Mike Moore notifications@github.com wrote:

There are some additional steps for creating a sink to a Bucket, and @quartzmo was chasing that info down for our docs, so I'll defer to him on that as well. 😅

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[tseaver]

@blowmage, @quartzmo The thing I found yesterday is that the bucket's ACL must include the cloud-logs@google.com group as with the owner role. E.g.:

from gcloud import storage

storage_client = storage.Client()
bucket = storage_client.create_bucket(BUCKET_NAME)
bucket.acl.reload()
logs_group = bucket.acl.group('cloud-logs@google.com')
logs_group.grant_owner()
bucket.acl.add_entity(logs_group)
bucket.acl.save()

[blowmage]

Yep, needing to grant that account ownership of the bucket is what we've seen as well.

[quartzmo]

Actually I don't think I ever set up the ACL like that, so thanks, I'll try it.

[quartzmo]

@tseaver Adding cloud-logs@google.com as Bucket ACL owner lets us successfully create the Sink in gcloud-ruby. I am closing our similar issue with the documentation PR referenced above.

[tseaver]

@blowmage wrote:

The issue with creating Logging Sinks is that it requires a user account, not a service account. This is imposed by the API, and there isn't much we can do to get around it.

@jgeewax Is there someplace where we can at least "argue with the ref" about this restriction? It seems perverse to deny service accounts the ability create logging sinks, when they can create and delete storage buckets, pubsub topics, bigquery datasets, bigtable clusters, etc.

[blowmage]

I agree it would be nice if the service did this for us. My guess is that it has to do with Storage not using IAM for permissions.

Is adding permissions to the bucket something we should do in our libraries when creating a sink?

[tseaver]

I agree it would be nice if the service did this for us. My guess is that it has to do with Storage not using IAM for permissions.

Pubsub uses IAM, and allows service accounts to create topics / subscriptions.

Is adding permissions to the bucket something we should do in our libraries when creating a sink?

Good question. Doing so "magically" would feel wrong to me. Maybe we should expose convenience functions, e.g., prepare_bucket_for_logging, prepare_topic_for_logging, and prepare_dataset_for_logging, which would make the needed permissions / role changes.

[quartzmo]

@blowmage Great question, but I agree with @tseaver that adding permissions automatically seems like going a bit too far. I think showing that it is required in the examples is good enough.

[tseaver]

FWIW, the logging API docs say:

You must have owner permission to the project whose logs are being exported.

I thought I had read (in the docs for setIamPolicy) that system accounts could not be granted the Owner permission. However, from the console permissions page, I was able to assign it to my service account, which then allows that account to create / delete sink resources.

@jgeewax we will need to ensure that the system account used by Travis has the Owner role before merging the sink-related system tests.

[dhermes]

@tseaver Do you have access to edit that project in the Cloud Console?

[tseaver]

@dhermes The only project which shows up for me in the "Manage all projects" view is my own.

[dhermes]

OK. I can make the changes. Send me an email with the changes you want?