[Storage] "No project ID was provided" error adding notification to just-created bucket

[zynyz]

I'm creating a bucket, then trying to set a notification on it. The bucket is created fine, CORS is set fine, but when I try to add the notification, it throws an exception with "No project ID was provided, and we were unable to detect a default project ID."

My code (simplified):

$opts =
[
  'storageClass' => 'REGIONAL',
  'location' => 'us-central1',
  'predefinedAcl' => 'publicRead',
  'cors' => [ /* elided for simplicity */ ],
];

$bucket = $client -> createBucket('my-bucket-name', $opts);
$bucket -> createNotification('my-topic', ['event_types' => 'OBJECT_FINALIZE']);

Am I doing something wrong here?

[dwsupplee]

Thanks for the report @zynyz,

If you inspect the credentials file being used in your client, does it include a project ID? I believe newer credentials should include this, but in legacy situations it may not be included.

If not, there are two options that should work here,

Include the project ID explicitly when you instantiate the storage client:

$storage = new StorageClient([
    'projectId' => 'my-project'
]);

or use a fully qualified topic name, which includes your project ID:

$notification = $bucket->createNotification('projects/my-project/topics/my-topic');

Please let me know if this helps :).

[zynyz]

Hi @dwsupplee -

Okay, it turns out the project ID was not getting set in my test rig. But, the project ID is in the credentials file (for a service account, in JSON format).

I guess my big question is "how did the bucket creation work?" Are there certain actions you need an explicitly-set project ID to do?

[dwsupplee]

Okay, it turns out the project ID was not getting set in my test rig. But, the project ID is in the credentials file (for a service account, in JSON format).

Interesting, which method are you using for authentication? I would imagine we should be able to detect the project ID on your behalf given that, this may be something we need to look at fixing.

I guess my big question is "how did the bucket creation work?" Are there certain actions you need an explicitly-set project ID to do?

Generally speaking, the Storage API doesn't require knowledge of the project ID. Creating a notification is an exception because the request requires the fully qualified topic name.

[zynyz]

Hey, thanks for the quick response.

I'm not sure what you mean by auth method. I'm just calling

new StorageClient($project_id, $path_to_key_file)

Generally speaking, the Storage API doesn't require knowledge of the project ID. Creating a notification is an exception because the request requires the fully qualified topic name.

Okay, that makes sense.

[dwsupplee]

I'm just calling new StorageClient($project_id, $path_to_key_file)

Perfect, thanks 👍. What I meant by authentication was whether you were using an environment variable, application default credentials, or explicitly passing in your keyfile (as it looks like you are).

[zynyz]

Actually, I also have the $GOOGLE_APPLICATION_CREDENTIALS env var set. I had completely forgotten about that. It points to a different keyfile, one that does not have a project ID. I take it one is overriding the other?

[dwsupplee]

If you supply a keyfile to the client explicitly, it should take priority over the environment variable. The order of precedence is outlined here.

[zynyz]

I've been working more on this, and now the error that comes back from createNotification() is

"The service account '[project-name]@gs-project-accounts.iam.gserviceaccount.com' does not have permission to publish messages to to the Cloud Pub/Sub topic '//pubsub.googleapis.com/projects/[project-name]/topics/[topic-name]', or that topic does not exist."

This is confusing, since that account does not even exist in our project. The account I'm using is

[project-name]@appspot.gserviceaccount.com

which has Owner access.

The topic does exist.

This seems to be related to this issue: https://github.com/GoogleCloudPlatform/google-cloud-common/issues/231 which doesn't seem to have a clear resolution.

Any help walking through this minefield would be very much appreciated.

[illambo]

Hi, in my experience I created notification topic manually from google cloud console and for that topic I manually added (from console) publisher pub/sub permission to [project-name]@gs-project-accounts.iam.gserviceaccount.com account.
I agree with you that this step is not very clear and does not seem to me to be documented.

[dwsupplee]

@zynyz, You're right, we should definitely make this more clear. I am working on some updates which will improve this story.

In the meantime @illambo's helpful suggestion will do the trick, or you can do the same through code with the following:

If you've installed just the Storage component so far, you'll want PubSub as well:

compose require google/cloud-pubsub

Now we can update the IAM bindings through code to grant Storage access to publishing to your topic:

require 'vendor/autoload.php';

use Google\Cloud\PubSub\PubSubClient;
use Google\Cloud\Storage\StorageClient;

$topic = 'my-topic';
$projectId = 'my-project';
$bucket = 'my-bucket';
$topicIam = (new PubSubClient())
    ->topic($topic)
    ->iam();
$policy = $topicIam->policy();
// Add permissions for the service account associated with Storage to publish to your topic
$policy['bindings'] = [
    [
        'role' => 'roles/pubsub.publisher',
        'members' => [
            'serviceAccount:' . $projectId . '@gs-project-accounts.iam.gserviceaccount.com'
        ]
    ]
];
$topicIam->setPolicy($policy);

$notification = (new StorageClient())
    ->bucket($bucket)
    ->createNotification($topic, [
        'event_types' => 'OBJECT_FINALIZE'
    ]);

[zynyz]

When doing this, using the same project ID and keyfile, I get an error PERMISSION_DENIED / "User not authorized to perform this action." My code is just like yours.

            $args =
            [
                'projectId'     => $this -> project_id,
                'keyFilePath'   => $this -> key_file,
            ];

            $iam = (new PubSubClient($args)) -> topic($topic_name) -> iam();
 
            ## This call throws an exception
            $policy = $iam -> policy();

No offense, but this is not unclear, it's extremely obscure.

It doesn't make any sense that this should be necessary. With the same service account, I've created the bucket and created the topic. It seems reasonable that the bucket should already have perms to send notifications to the topic.

Honestly, this is something the API should take care of. If I use gsutil, it will add the proper permissions automatically (and silently). I would expect ths same behavior from the API, at least as an option.

I apologize if I come off as flamey here, but I'm a little frustrated about all this.

[dwsupplee]

I can empathize with the frustration. Apologies this has been a blocker. We'll make sure we get the method updated to include a sample on how to update the IAM policy to grant Storage the ability to send notifications to the topic.

If I use gsutil, it will add the proper permissions automatically (and silently). I would expect ths same behavior from the API, at least as an option.

I am definitely open to this. @frankyn does this seem like a feature that would be reasonable from the client libraries?

I get an error PERMISSION_DENIED / "User not authorized to perform this action."

Could you verify the service account being used has the correct permissions set on the topic? This can be done here. If you haven't already taken a look, the prerequisites section for registering notifications should provide a little more detail.

[zynyz]

The service account has Editor permissions - it's just the default service account for the project. My test code creates the topic, creates the bucket, then attempts to add a notification, so it's all using the same account.

The code is blowing up on the call to get the existing policy, so it's not even a problem setting the policy.

According to this page, I need roles/owner perms to see the policy for a topic I just created with roles/editor perms. Is that actually the case?

[dwsupplee]

Yes it is correct, you need the owner or admin role in order to modify/view IAM.

[zynyz]

Then how come I can do this via gsutil using the same account?

[dwsupplee]

I tried the replicate and found the following:

Using gsutil with a service account which has the project editor role:

gsutil notification create -t projects/my-project/topics/my-topic -f json gs://my-bucket

failed with AccessDeniedException: 403. After setting the service account's role to owner the notification was successfully created.

When you run gcloud auth list is the active service account the same as the one being utilized by the PHP client?

[zynyz]

gcloud auth list returns only one account, which is my email address, which has the Owner role.

$GOOGLE_APPLICATION_CREDENTIALS points to a file with a key for the account [project-name]@appspot.gserviceaccount.com, which has Editor role.

gsutil notification create -s -fjson -t[topic] gs://[bucket]

returns

PublishPermissionDeniedException: 403 The service account '[project-name]@gs-project-accounts.iam.gserviceaccount.com' does not have permission to publish messages to to the Cloud Pub/Sub topic '//pubsub.googleapis.com/projects/[project-name]/topics/[topic]', or that topic does not exist.

Without the -s option, gsutil creates the notification successfully.

Service account [project-name]@gs-project-accounts.iam.gserviceaccount.com does not even exist in the project. Does that get created automatically?

I'm using gsutil v 4.28, if that helps any.

[dwsupplee]

That makes sense, and explains why the PHP code is failing. After updating the permissions for that account you should be good to go.

Service account [project-name]@gs-project-accounts.iam.gserviceaccount.com does not even exist in the project. Does that get created automatically?

My understanding is this is a service account used by the Storage "backend". It isn't something you need to worry about having control over, it just needs permissions to your topic.

[zynyz]

Okay, I understand the problems, but the whole thing is pretty confusing. I really hope the docs can get updated with a clear explanation of the situation and the pitfalls.

If this whole business could get wrapped into createNotification(), that would be beyond terrific.

Thanks for all your help.

[nazarkazymov-devpronet]

Hello everyone. Is it correct that format of Google Cloud Storage service accounts was changed? It was - " [project-name]@gs-project-accounts.iam.gserviceaccount.com", and currently it is "service-[projectId]@gs-project-accounts.iam.gserviceaccount.com".
I check it by this API and for special new-created projects I get - this format : 'service-[project_Id]@gs-project-accounts.iam.gserviceaccount.com'.

I used json-format for topic-resource, something like that -
{ 'name': topic, 'type': 'pubsub.v1.topic', 'properties': { 'topic': topic }, 'accessControl': { 'gcpIamPolicy': { 'bindings': [ { 'role': 'roles/pubsub.publisher', 'members': [ 'service_account = [project_name]@gs-project-accounts.iam.gserviceaccount.com' ] } ] } } }
and for special new created projects it is fauilure "service account ... doesn't exist"

So, how can this service account name be fetched by deployment manager dynamically during the creation of deployment?

[frankyn]

Acking I missed this issue. Will look over. Apologies for the delay.

[dwsupplee]

@nazarkazymov-devpronet I just opened #1173 which should allow you to programmatically fetch the service account. Please take a look when you have a moment and let me know if it achieves what you're looking for.

[nazarkazymov-devpronet]

@dwsupplee thank you, sorry for some misunderstanding,
I don't use php google cloud client, I use Deployment Manager and describes my resources by Deployment Manager
in python files ( Deployment Manager allows to create configuration using Python or Jinja).
Actually, question is how we can fetch the storage service account described above dynamically in Deployment Manager config files. As I can see here
there are only several environment variables like project_name, project_id, time etc.
and there isn't any storage_service_account environment variable.
Sorry guys, if you are not chained with this part of GCP.

[dwsupplee]

Thanks for the clarification @nazarkazymov-devpronet :), sorry for the confusion. As you mentioned, this repository is more focused on the PHP client libraries and unfortunately may not be the best place to find answers to your question. We actively monitor StackOverflow if you'd like to try posting the question there. Alternatively, @frankyn might be able to help out and he's a Storage expert :).

[nazarkazymov-devpronet]

@frankyn Can you help me, please with this question about storage service account ?

[frankyn]

@nazarkazymov-devpronet responded to your question. You were caught in-between the old and the new format for the GCS service account. Reading over the other bits of this issue now.

[frankyn]

@dwsupplee, I think it would be helpful to outline an example of setting up a GCS PubSub notification in PHP client library documentation instead of performing the whole action on behalf of the user. Now that getServiceAccount is available this user story should be easier to perform.

That is one option, the better solution would be to put this on cloud.google.com/storage/docs, but that will take some time to do.

@nazarkazymov-devpronet you can also generate the GCS service account using the API explorer for Projects.serviceAccount: get .

[nazarkazymov-devpronet]

@frankyn I see. Unfortunately I don't use any client API, I want to create a topic resource using Deployment Manager and python resource file. And If I understand correctly is it impossible to call API explorer from this python resource files? I have to hardcode project's name or project's id in string parameters for now and can't call getServiceAccount: API.

[frankyn]

I missed that, what I'd recommend is try providing the new format in your Resource Manager configuration without calling getServiceAcount. Please let me know if this still blocks you.

I don't expect it to change anytime soon, but I can understand frustration of it being updated and potentially breaking things in the future.

[nazarkazymov-devpronet]

I've got it. Thank you very much for your explanation.

[frankyn]

Oh btw, you can also gsutil tool for the project you're using resource manager with:

gsutil kms serviceaccount

[frankyn]

@nazarkazymov-devpronet can we close this issue?

[dwsupplee]

@frankyn Let's please leave this open, as it will be used to track the updates to the documentation.

[nazarkazymov-devpronet]

from my side (and what about just my problem ) I don't have any questions, thank you all guys.

[frankyn]

Sgtm @dwsupplee! Thanks @nazarkazymov-devpronet!