Can't get default credentials for use with datastore emulator

[sam0x17]

I am trying to use the datastore emulator locally with my rails app in development mode for testing etc.

I run the beta datastore emulator, then run gcloud beta emulators datastore env-init and make sure those environment variables are in my .bashrc file (and source my .bashrc file). Then I try to do the following from within Rails:

require 'gcloud'
gcloud = Gcloud.new('[project id here]')
dataset = gcloud.datastore

I get the following error:

RuntimeError: Could not load the default credentials. Browse to
https://developers.google.com/accounts/docs/application-default-credentials
for more information

    from /home/sam/.rvm/gems/ruby-2.2.3/gems/googleauth-0.5.1/lib/googleauth.rb:119:in `get_application_default'
    from /home/sam/.rvm/gems/ruby-2.2.3/gems/gcloud-0.7.2/lib/gcloud/credentials.rb:83:in `default'
    from /home/sam/.rvm/gems/ruby-2.2.3/gems/gcloud-0.7.2/lib/gcloud/datastore.rb:62:in `datastore'
    from /home/sam/.rvm/gems/ruby-2.2.3/gems/gcloud-0.7.2/lib/gcloud.rb:106:in `datastore'
    from (irb):3
    from /home/sam/.rvm/gems/ruby-2.2.3/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
    from /home/sam/.rvm/gems/ruby-2.2.3/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
    from /home/sam/.rvm/gems/ruby-2.2.3/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
    from /home/sam/.rvm/gems/ruby-2.2.3/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
    from /home/sam/.rvm/gems/ruby-2.2.3/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'
    from bin/rails:18:in `require'
    from bin/rails:18:in `<main>'

I was under the impression I did not need a keyfile if using the datastore emulator. If I do need a keyfile, how do I create it? The datastore emulator documentation doesn't mention this at all, and seems to suggest that merely having these environment variables in place should be enough for local clients to connect automatically.

Please advise!

[bmclean]

Hi @SamKelly,

I ran into this running locally with gcd.sh. It worked when I was already authenticated with gcloud but I wanted my devel and test environments to not have an external dependency. I am not sure if the gcloud beta emulators datastore uses gcd but based on your error message I think this might work for you as well.

I created a file in lib named 'local_datastore_no_auth:

require 'gcloud/datastore'

if Rails.env.development? || Rails.env.test?
  module Gcloud
    module Datastore
      # Override the authentication credentials needed for the actual Google Cloud Datastore.
      class Credentials
        def initialize
          # pass
        end

        def sign_http_request(request)
          request
        end
      end
    end
  end
end

I create my datastore dataset in an initializer like this:

config = Rails.application.config.database_configuration[Rails.env]['dataset_id']
if Rails.env.development? or Rails.env.test?
  require 'local_datastore_no_auth'
  @dataset ||= Gcloud::Datastore::Dataset.new(config, Gcloud::Datastore::Credentials.new)
else
  ENV['GCLOUD_KEYFILE_JSON'] = '{"private_key": "' + ENV['SERVICE_ACCOUNT_PRIVATE_KEY'] + '",
    "client_email": "' + ENV['SERVICE_ACCOUNT_CLIENT_EMAIL'] + '"}'
  @dataset ||= Gcloud.datastore config
end

[sam0x17]

This is even better! Thanks!

[jgeewax]

/cc @quartzmo Any chance we can automate this? It makes me sad to see the overrides and all that...

[quartzmo]

@bmclean Thanks for sharing this solution!

@jgeewax In my opinion, we should definitely support insecure access. (This is similar to #491.) As we transition to gRPC, the implementation details will almost certainly change, so am I right to assume that our upcoming Datastore v1beta3 milestone is the version to target?

[jgeewax]

yep, v1beta3 is the right target IMO

[quartzmo]

Possible duplication with #604?

[sam0x17]

@quartzmo I am using the updated emulator AFAIK

[quartzmo]

@SamKelly Thanks. We will definitely test/develop against the updated emulator.

[sam0x17]

@quartzmo would also be great if the documentation specifically talks about there even being an emulator and how to connect to it from ruby -- was sort of buried

e.g. http://googlecloudplatform.github.io/gcloud-ruby/docs/master/Gcloud/Datastore.html makes no mention of what to do in "development" or "testing" mode...

[sam0x17]

is gcloud components update sufficient to make sure I'm using the updated emulator?

[quartzmo]

@SamKelly Do you want to open a new issue for the docs? I strongly agree that using the emulator should be covered.

Regarding the emulator update, I don't know of any other way to do it, or how to verify the version. Please post here if you find anything helpful.

[sam0x17]

@quartzmo I posted feedback on the docs when I was on the page (I think?) but not sure how else to do it -- if you want to open something please by all means 👍

[sam0x17]

@bmclean actually this does not seem to be working for me. While I no longer get the error message when I try to connect, I always get the following message when I try to save a record:

Gcloud::Datastore::ApiError (API call to commit was not successful):

[sam0x17]

I'm sort of in a hurry with this does anyone know what I can do to get gcloud_ruby to use the datastore emulator? I must be missing something but nothing from above has helped beyond suppressing the missing keyfile error message. What is the proper way to connect? I don't understand why a monkey patch should be necessary in the first place. Please help

[bmclean]

Hi @SamKelly,
I use the gcd tool directly but I just tried it with the gcloud beta emulators datastore and it does work for me. It appears they are the same emulator. I launched it from a terminal like this:

gcloud beta emulators datastore start --host-port localhost:8180

Try this in a Rails initializer file (make sure to restart the rails server to pick up the change):

if Rails.env.development?
  ENV['DATASTORE_HOST'] = 'http://localhost:8180'
end

When I change to an incorrect port I receive the Gcloud::Datastore::ApiError error message.

[sam0x17]

OK I'll try that -- what is the gcd tool? People keep mentioning this and I"m not sure what it is.

[bmclean]

It is the google cloud datastore (gcd) emulator. If you run gcloud components list it has the ID gcd-emulator. It is installed in ~/google-cloud-sdk/platform/gcd or equivalent. I don't see it listed on the cloud datastore site anymore though. My guess is that the Google Cloud SDK is now providing a wrapper around it and it is provided through gcloud beta emulators datastore. The doc looks very similar. Previously I would create a local datastore like this:

gcd.sh create --project_id=local-datastore tmp/local_datastore

and start it like this:

gcd.sh start --port=8180 tmp/local_datastore

Now it looks like I can just do it like this:

gcloud beta emulators datastore start --host-port localhost:8180 --data-dir tmp/local_datastore

and

gcloud beta emulators datastore start --host-port localhost:8181 --data-dir tmp/test_datastore

[sam0x17]

ok thanks -- manually fixing the port will definitely make things less complicated for me as well

[sam0x17]

So turns out my issue was in my project id -- in ruby I was using an underscore in the name, but in my emulator config I was using a dash... /facepalm

[jgeewax]

Is there a way we could've made the error message more clear ? Or was it
just a regular old facepalm that we can't really do anything about?

If it's the former, I'd love to try to clarify the error message you got so
others don't get stuck in the same situation....

On Wed, Apr 6, 2016 at 12:25 AM, Sam Kelly notifications@github.com wrote:

So turns out my issue was in my project id -- in ruby I was using an
underscore in the name, but in my emulator config I was using a dash...
/facepalm

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#610 (comment)

[quartzmo]

@jgeewax As we transition to gRPC, the implementation details will almost certainly change, so we have a fresh chance to ensure that our emulator support handles error conditions clearly, in order to minimize user frustration while attempting to connect to the emulator. Also, updates to our emulator support should permit insecure access, thus preventing the need for the customization that @bmclean helpfully provided. I noted both of these issues in #604.

@SamKelly If you feel #604 will resolve everything covered in this issue going forward, please close this issue. However, if for some reason you believe you will be unable to upgrade to the next version of gcloud-ruby (using gRPC to connect to Datastore v1beta3, see #314), let us know here.

[sam0x17]

@quartzmo @jgeewax As long as the gRPC implementation has better awareness of when it is using the emulator vs when it is using a production datastore, I think everything should be fine. Maybe when there is a datastore error, gcloud_ruby should suggest looking at the emulator log -- this would have saved me some time, even though it was a bit of a facepalm.

[quartzmo]

@SamKelly We really appreciate your help as together we work out the kinks in using gcloud-ruby for Datastore development. Please do not hesitate to open issues such as this one when you run into something that you think could be better. Obviously we don't have control over everything (just this layer), but we do have control over our documentation and by adding your hard-earned knowledge there, I think we can do much to improve the user experience.

[sam0x17]

@quartzmo no problem -- I'm just glad you guys have been so responsive!