Auto-detect a project ID

[jmdobry]

Problem

After running gcloud beta auth application-default login I should be able to use google-cloud-node to access Google Cloud Platform services, but I can't. I must set the GCLOUD_PROJECT env var or explicitly pass a project ID to a client for google-cloud-node to work.

Steps to reproduce

1. Install the Google Cloud SDK, git, Node.js and NPM
2. Run gcloud init, you'll be prompted to set a default project
3. Run gcloud beta auth application-default login and follow the OAuth 2.0 flow to login
4. Clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git
5. Run cd nodejs-docs-samples/storage
6. Run npm install
7. Run node buckets.js list

Observe the error:

Error: Sorry, we cannot connect to Google Cloud Services without a project ID. You may
specify one with an environment variable named "GCLOUD_PROJECT". See
https://googlecloudplatform.github.io/google-cloud-node/#//docs/guides/authentication
for a detailed guide on creating an authenticated connection.

1. Modify buckets.js to explicitly pass in a project ID
2. Run node buckets.js list
3. It works
4. Undo your changes to buckets.js
5. Set the GCLOUD_PROJECT env var to your project ID
6. Run node buckets.js list
7. It works

Expected

1. Open terminal
2. Run cd nodejs-docs-samples/storage
3. Run gcloud beta auth application-default login
4. Run node buckets.js list (No env vars are set, code does not pass project ID to client)
5. It works

[jgeewax]

Is there a spec for where we're supposed to look for a project ID? I think that's ... kind of scary.

[stephenplusplus]

We throw the error because a project ID wasn't specified. We get the auth client-- we can auth requests-- but we need the projectId for the construction of the URL to send the requests to. Is there an environment variable with the projectId that's set when gcloud beta auth application-default login is run?

[jgeewax]

It's definitely stored somewhere, but it's definitely not an environment variable:

$  gcloud config list project
Your active configuration is: [default]

[core]
project = jjg-cloud-research

I think it's a tiny bit scary to pull from a file. Totally fine to pull from an env variable. Can we look through how this might go wrong if we pull from a file somewhere? If there's nothing too nasty, we should add this to the list of stuff to do.

[stephenplusplus]

It looks like it's in ~/.config/gcloud/configurations/config_default for me. There's also a file a level up, active_config which is just plain text, contents: "default". Presumably, I need to use the contents of that file to read from the correct config file (~/config/gcloud/configurations/config_{{contents}}). I put that into a PR: #1656

It would be nice if this was simply an env var-- is it possible to squeeze that feature into the SDK directly?

If not, it doesn't bother me to read from the file, as long as my logic above is correct. In the worst case, it can't find it, and the program continues without the projectId, throwing the mesage that the user must set the GCLOUD_PROJECT env var or provide it.

[jmdobry]

@jonparrott Can you chime in on how google-cloud-python does it?

[jmdobry]

I think google-cloud-python (well actually, github.com/google/oauth2client) looks for a file named application_default_credentials.json.

On Mac would be located at ~/.config/gcloud/application_default_credentials.json. Here's how the python client looks for it: https://github.com/google/oauth2client/blob/master/oauth2client/client.py#L1347

[jmdobry]

So perhaps this needs to be solved in https://github.com/google/google-auth-library-nodejs

[stephenplusplus]

That file is read from google-auth-library (https://github.com/google/google-auth-library-nodejs/blob/02f587472ecdc2582f8d5fc754996aa6ef9388a0/lib/auth/googleauth.js#L231), but that file doesn't have the projectId in it-- at least not for me.

[jmdobry]

Aha, found it: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/core/google/cloud/_helpers.py#L217

[stephenplusplus]

Nice! That looks like it's one step simpler than my approach in #1656, which reads the active_config file. Is it worth clearing up if we should be using that to find the correct config file, or just always look for default file? I'm not a power-user of the SDK, but if it's possible that other config files can be made / marked as active, that might invalidate the setting from the default file.

[jmdobry]

I think first, get to feature parity with google-cloud-python on implicitly determining the project ID. Then open another issue to investigate the active_config thing.

[theacodes]

I'd recommend doing it in the auth library if possible, as the new python auth library will work like this:

import google.auth

credentials, project_id = google.auth.default()

Here's the implementation (based on the google-cloud-python _helpers.py) file.

As for default_config vs active_config, that requires a bit more attention... hmm..

[stephenplusplus]

Sounds good, thanks for the link. I'm putting this functionality in its own module so that it can be used from the auth library, maybe:

var googleAuth = new GoogleAuth();

googleAuth.getApplicationDefault(function(err, authClient) {
  authClient.projectId === already detected & populated
});

[theacodes]

The big benefit for doing it within the same code that does ADC is that the same machinery is used to get the credentials and the project. So, it could simplify the implementation to do both at once.

(But it's totally up to you)

[jmdobry]

I agree with @jonparrott, this should be added to https://github.com/google/google-auth-library-nodejs

[tmatsuo]

Re: default vs active

I think it's for supporting multiple configurations by using gcloud config configurations sub command. It should read the active config file.

But, the implementation can change, so how about to call gcloud -q config list core/project --format=json ?

[stephenplusplus]

I'm +1 on that code going into google-auth-library. Created an issue: googleapis/google-auth-library-nodejs#97

We'll keep this issue open here until that's implemented, then make the required changes here to honor it.

[Splaktar]

Looks like googleapis/google-auth-library-nodejs#97 got closed by associated PR (googleapis/google-auth-library-nodejs#98) a few minutes ago.

[stephenplusplus]

Yep, getting close! Still a few more steps #1656

[thetittenquetscher]

I think my problem belongs to this thread, because I struggled with the authentication process as jmdobry described above as well.

I am working with the tutorial sample code for the Speech API provided here: https://cloud.google.com/speech/docs/rest-tutorial (I am working with Node.js)

I have modified some lines in that tutorial sample code in order to get around the above mentioned error:

Error: Sorry, we cannot connect to Google Cloud Services without a project ID. You may
specify one with an environment variable named "GCLOUD_PROJECT". See
https://googlecloudplatform.github.io/google-cloud-node/#//docs/guides/authentication
for a detailed guide on creating an authenticated connection.

.1. I took the following lines from
https://googlecloudplatform.github.io/google-cloud-node/#/docs/google-cloud/0.41.2/speech
and added them to the tutorial sample code:

var speechClient = speech({
      projectId: 'grape-spaceship-123',    // added my own credentials
     keyFilename: '/path/to/keyfile.json'  // deployed the keyfile.json on my Desktop
    });

.2. Also I changed this line here in the tutorial sample code from / to:

const speech = require('@google-cloud/speech’)();

const speech = require('@google-cloud/speech');

.3. I changed this line here as well from / to :

var utils = require('../utils');

var utils = require('utils');

.4. And I added the utils module to the node_modules file with

npm install -g utils

These changes let me pass to the next step – to the CLI down at the bottom of the tutorial sample code:

Commands:
  sync <filename>    Detects speech in an audio file.
  async <filename>   Creates a job to detect speech in an audio file, and waits for the job to complete.
  stream <filename>  Detects speech in an audio file by streaming it to the Speech API.
  listen             Detects speech in a microphone input stream.

Options:
  --help  Show help                                                                                            [boolean]

Examples:
  node recognize sync ./resources/audio.raw    Detects speech in "./resources/audio.raw".
  node recognize async ./resources/audio.raw   Creates a job to detect speech in "./resources/audio.raw", and waits for
                                               the job to complete.
  node recognize stream ./resources/audio.raw  Detects speech in "./resources/audio.raw" by streaming it to the Speech
                                               API.
  node recognize listen                        Detects speech in a microphone input stream.

For more information, see https://cloud.google.com/speech/docs

Not enough non-option arguments: got 0, need at least 1

I tried every example, yet I get the following error :

TypeError: utils.makeHandler is not a function
    at Object.handler (/Users/me/Desktop/speech/recognize.js:312:51)
    at Object.self.runCommand (/Users/me/Desktop/speech/node_modules/yargs/lib/command.js:158:22)
    at parseArgs (/Users/me/Desktop/speech/node_modules/yargs/yargs.js:727:28)
    at Object.Yargs.self.parse (/Users/me/Desktop/speech/node_modules/yargs/yargs.js:388:12)
    at Object.speech.recognize.speech.startRecognition.module.exports.main (/Users/me/Desktop/speech/recognize.js:305:25)
    at Object.<anonymous> (/Users/me/Desktop/speech/recognize.js:332:11)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)

Why is this TypeError happening? Can anyone help me, please.
Hope this is understandable what I am writing.

[jmdobry]

@thetittenquetscher You should move your question to the nodejs-docs-samples repository.

Your error was caused by changing require('../utils'); to require('utils');. You should not have changed that, as utils.js is a file local to the repository, and isn't something that can be installed globally. In addition, before you can successfully run the Speech API sample you need to follow the prerequisite instructions in the README.md file.

[thetittenquetscher]

@jmdobry

The provided link helped me run the samples. Thank you.

[gsusI]

In case someone lands here looking for a solution when this happens in cloudshell, have a look to: https://stackoverflow.com/a/60153817/3198983

You'll basically have to run this line with your project id, which might be the name:

gcloud config set project [PROJECT_ID]