Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Naively downloading the demo and running it fails with 401 errors #245

Closed
trixing opened this issue Apr 18, 2016 · 11 comments

Comments

@trixing
Copy link

commented Apr 18, 2016

It's unclear with the instructions of how set it up that it actually works. The documentation at https://cloud.google.com/appengine/docs/go/ doesn't seem to imply extra steps.

Super frustrating experience.

./goapp serve ../gcs_app

2016/04/18 12:07:23 ERROR: createFile: unable to close bucket "app_default_bucket", file "demo-testfile-go": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:26 ERROR: readFile: unable to open file from bucket "app_default_bucket", file "demo-testfile-go": storage: can't read object app_default_bucket/demo-testfile-go, status code: 401 Unauthorized
2016/04/18 12:07:28 ERROR: copyFile: unable to copy /app_default_bucket/demo-testfile-go to bucket "app_default_bucket", file "demo-testfile-go-copy": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:29 ERROR: statFile: unable to stat file from bucket "app_default_bucket", file "demo-testfile-go": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:31 ERROR: createFile: unable to close bucket "app_default_bucket", file "foo1": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:33 ERROR: createFile: unable to close bucket "app_default_bucket", file "foo2": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:35 ERROR: createFile: unable to close bucket "app_default_bucket", file "bar": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:37 ERROR: createFile: unable to close bucket "app_default_bucket", file "bar/1": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:39 ERROR: createFile: unable to close bucket "app_default_bucket", file "bar/2": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:41 ERROR: createFile: unable to close bucket "app_default_bucket", file "boo/": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:43 ERROR: listBucket: unable to list bucket "app_default_bucket": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:44 ERROR: listBucketDirMode: unable to list bucket "app_default_bucket": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:46 ERROR: defaultACL: unable to list default object ACL for bucket "app_default_bucket": storage: error listing bucket ACL for bucket "app_default_bucket": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:48 ERROR: putDefaultACLRule: unable to save default object ACL rule for bucket "app_default_bucket": storage: error updating default ACL entry for bucket "app_default_bucket", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:49 ERROR: deleteDefaultACLRule: unable to delete default object ACL rule for bucket "app_default_bucket": storage: error deleting default ACL entry for bucket "app_default_bucket", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:51 ERROR: dumpBucketACL: unable to list bucket ACL for bucket "app_default_bucket": storage: error listing bucket ACL for bucket "app_default_bucket": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:53 ERROR: putBucketACLRule: unable to save bucket ACL rule for bucket "app_default_bucket": storage: error updating bucket ACL entry for bucket "app_default_bucket", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:55 ERROR: deleteBucketACLRule: unable to delete bucket ACL rule for bucket "app_default_bucket": storage: error deleting bucket ACL entry for bucket "app_default_bucket", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:56 ERROR: dumpACL: unable to list file ACL for bucket "app_default_bucket", file "demo-testfile-go": storage: error listing object ACL for bucket "app_default_bucket", file "demo-testfile-go": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:58 ERROR: putACLRule: unable to save ACL rule for bucket "app_default_bucket", file "demo-testfile-go": storage: error updating object ACL entry for bucket "app_default_bucket", file "demo-testfile-go", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:07:59 ERROR: deleteACLRule: unable to delete ACL rule for bucket "app_default_bucket", file "demo-testfile-go": storage: error deleting object ACL entry for bucket "app_default_bucket", file "demo-testfile-go", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/04/18 12:08:01 ERROR: deleteFiles: unable to delete bucket "app_default_bucket", file "demo-testfile-go": googleapi: Error 401: Invalid Credentials, authError

@okdave

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2016

@trixing can you add more detail about what you were trying to achieve? What do you under "gcs_app"?
The documentation you've linked above is the getting started for "Hello World" which doesn't use the GCS or any of the packages in this repo.

@ram-nadella

This comment has been minimized.

Copy link

commented May 16, 2016

@okdave I ran into this just now.

I believe @trixing is referring to the example app here: https://github.com/GoogleCloudPlatform/gcloud-golang/tree/master/examples/storage/appengine

I was trying to write some code to upload a file to Google Cloud Storage from Google App Engine (golang standard environment).

After following the tutorial in the docs which BTW was not straight forward and required quite a bit of reverse-engineering from the example mentioned above because the snippet in the docs appears to be a copy-paste of the above example code and is not a good starting point (it relies on the demo struct etc.)

Anyway, after I managed to put the code together and ran it in dev with goapp serve but I saw the following error lines:

$ goapp serve
INFO     2016-05-16 01:49:48,294 devappserver2.py:769] Skipping SDK update check.
INFO     2016-05-16 01:49:48,328 api_server.py:205] Starting API server at: http://localhost:52382
INFO     2016-05-16 01:49:48,330 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
INFO     2016-05-16 01:49:48,331 admin_server.py:116] Starting admin server at: http://localhost:8000
2016/05/16 01:49:55 ERROR: createFile: unable to close bucket &{%!q(*storage.ACLHandle=&{0xc8201ce810 app_default_bucket  false}) %!q(*storage.ACLHandle=&{0xc8201ce810 app_default_bucket  true}) %!q(*storage.Client=&{0xc8201cdb30 0xc82023c000}) "app_default_bucket"}, file "file.txt": googleapi: Error 401: Invalid Credentials, authError
INFO     2016-05-16 01:49:55,493 module.py:787] default: "GET / HTTP/1.1" 200 62
2016/05/16 01:49:56 ERROR: createFile: unable to close bucket &{%!q(*storage.ACLHandle=&{0xc82016e160 app_default_bucket  false}) %!q(*storage.ACLHandle=&{0xc82016e160 app_default_bucket  true}) %!q(*storage.Client=&{0xc820a862a0 0xc82018c000}) "app_default_bucket"}, file "file.txt": googleapi: Error 401: Invalid Credentials, authError

At this point I attempted to run the example code from this repo as it is and saw the same 401 errors.

2016/05/16 02:02:36 ERROR: createFile: unable to close bucket "app_default_bucket", file "demo-testfile-go": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:37 ERROR: readFile: unable to open file from bucket "app_default_bucket", file "demo-testfile-go": storage: can't read object app_default_bucket/demo-testfile-go, status code: 401 Unauthorized
2016/05/16 02:02:38 ERROR: copyFile: unable to copy /app_default_bucket/demo-testfile-go to bucket "app_default_bucket", file "demo-testfile-go-copy": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:39 ERROR: statFile: unable to stat file from bucket "app_default_bucket", file "demo-testfile-go": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:40 ERROR: createFile: unable to close bucket "app_default_bucket", file "foo1": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:41 ERROR: createFile: unable to close bucket "app_default_bucket", file "foo2": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:42 ERROR: createFile: unable to close bucket "app_default_bucket", file "bar": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:43 ERROR: createFile: unable to close bucket "app_default_bucket", file "bar/1": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:43 ERROR: createFile: unable to close bucket "app_default_bucket", file "bar/2": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:44 ERROR: createFile: unable to close bucket "app_default_bucket", file "boo/": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:45 ERROR: listBucket: unable to list bucket "app_default_bucket": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:46 ERROR: listBucketDirMode: unable to list bucket "app_default_bucket": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:47 ERROR: defaultACL: unable to list default object ACL for bucket "app_default_bucket": storage: error listing bucket ACL for bucket "app_default_bucket": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:47 ERROR: putDefaultACLRule: unable to save default object ACL rule for bucket "app_default_bucket": storage: error updating default ACL entry for bucket "app_default_bucket", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:48 ERROR: deleteDefaultACLRule: unable to delete default object ACL rule for bucket "app_default_bucket": storage: error deleting default ACL entry for bucket "app_default_bucket", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:49 ERROR: dumpBucketACL: unable to list bucket ACL for bucket "app_default_bucket": storage: error listing bucket ACL for bucket "app_default_bucket": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:50 ERROR: putBucketACLRule: unable to save bucket ACL rule for bucket "app_default_bucket": storage: error updating bucket ACL entry for bucket "app_default_bucket", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:51 ERROR: deleteBucketACLRule: unable to delete bucket ACL rule for bucket "app_default_bucket": storage: error deleting bucket ACL entry for bucket "app_default_bucket", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:51 ERROR: dumpACL: unable to list file ACL for bucket "app_default_bucket", file "demo-testfile-go": storage: error listing object ACL for bucket "app_default_bucket", file "demo-testfile-go": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:52 ERROR: putACLRule: unable to save ACL rule for bucket "app_default_bucket", file "demo-testfile-go": storage: error updating object ACL entry for bucket "app_default_bucket", file "demo-testfile-go", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:53 ERROR: deleteACLRule: unable to delete ACL rule for bucket "app_default_bucket", file "demo-testfile-go": storage: error deleting object ACL entry for bucket "app_default_bucket", file "demo-testfile-go", entity "allUsers": googleapi: Error 401: Invalid Credentials, authError
2016/05/16 02:02:54 ERROR: deleteFiles: unable to delete bucket "app_default_bucket", file "demo-testfile-go": googleapi: Error 401: Invalid Credentials, authError
INFO     2016-05-16 02:02:54,173 module.py:787] default: "GET / HTTP/1.1" 200 969

How do I use Google Cloud Storage from App Engine?

@okdave

This comment has been minimized.

Copy link
Contributor

commented May 16, 2016

Sorry about the confusion. The docs definitely need to be improved.

I think one source of confusion is that the example does not work in the dev environment: the google.golang.org/cloud/storage package only knows how to talk to the production Google Cloud Storage service. So, it is attempting to access your production bucket/objects using the credentials you have from signing in with the gcloud tool.

@ram-nadella

This comment has been minimized.

Copy link

commented May 17, 2016

That's good to know!

Is there an alternative? Or is the current state that code that uses Google Cloud Storage won't work in dev?

@okdave

This comment has been minimized.

Copy link
Contributor

commented May 17, 2016

As far as I know, there's no local emulator at all for working with GCS is dev.

One thing you can do is to make a bucket in a test project, and use that directly from your development environment instead. It will be a bit slower than running in prod, and you won't be able to your development offline, but it should work. You might need to tweak the example code a bit, since it's not clear to me that the oauth2 package will give the right default credentials when running against the dev_appserver.py – but outside of app engine you can definitely use gcloud auth login to access a project's resources.

One thing worth mentioning is that if you only need really simple access to your app's default bucket (and only that bucket), then the google.golang.org/appengine/blobstore is really easy to use. It also works seamlessly with dev_appserver.py so it makes that part of the story much easier too.

okdave added a commit that referenced this issue May 17, 2016

examples/storage/appengine{,vm}: clean up example
Make sure example calls out that it will not work against
dev_appserver.py (in standard).

Make error messages appear in the text output.

Issue #245.

Change-Id: Ib9cc1d0e69ae4378ad12b03d7ada5a29780b952b
Reviewed-on: https://code-review.googlesource.com/4768
Reviewed-by: Chris Broadfoot <cbro@google.com>
@weistn

This comment has been minimized.

Copy link

commented May 17, 2016

I got GCS working from the dev server. It is slow, but it works. I wrote down the steps in a blog post. Perhaps it helps.

http://researchtw.blogspot.de/2016/05/accessing-google-cloud-storage-with-go.html

@okdave

This comment has been minimized.

Copy link
Contributor

commented May 18, 2016

Thanks @weistn for the write up – it looks like the key part is using the dev_appserver flag --default_gcs_bucket_name to control which bucket your app will try to contact.

@ram-nadella

This comment has been minimized.

Copy link

commented May 18, 2016

@weistn nice!

I came up a somewhat similar alternative.

// get the bucket name
// Google Cloud Storage does not have a dev emulator and so does not work in dev
// use the staging bucket that is available with app engine projects
func bucketName(ctx context.Context) (string, error) {
    if appengine.IsDevAppServer() {
        return "staging.<APP_NAME>.appspot.com", nil
    } else {
        // determine the default bucket name
        bucketName, err := file.DefaultBucketName(ctx)
        if err != nil {
            log.Errorf(ctx, "failed to get default GCS bucket name: %v", err)
            return "", err
        }

        return bucketName, nil
    }
}

Creating a context from the current request (r *http.Request) is as simple as: ctx := appengine.NewContext(r)

Basically hard-coding the prod bucket name in code and using a function to get the right bucket name depending on the environment. This way you can continue to use goapp serve and by using the staging bucket (which is cleared once a week I believe), you keep your prod bucket clean.

To use, call the above function, pass the returned bucket name to the client.Bucket() call and do what you need to do from there.

This still required a couple gcloud commands to be run before it worked (which I can't recall anymore...), I believe it was gcloud auth login and then a way to set the current project or something.

Hopefully we'll have a GCS emulator in dev at some point to avoid all these hacks.... is this is on the roadmap? App Engine + Cloud Storage is a powerful combo but this is not an easy develop....

@fjorgemota

This comment has been minimized.

Copy link

commented Jun 18, 2016

Hello!

I found a similar bug when running unit tests in a struct that I created that uses GCS Client (basically, I can get a TokenSource without problems but when calling the API it returns an ugly 401 error..and as the tests itself are slow, I think that this occurs because the client is trying to contact GCS online, and not in dev_appserver.py).

I found this issue reporting that dev_appserver.py isn't compatible with GCS Client for Go. I think it's somewhat strange, as the version for Python works normally (it seems to just recognizes the calls to GCS as calls to blob store internally and automatically, but it works nicely), but, what I found more strange than this is this message found in this page:

gcs

So...I think that this block should be updated to clarify that Development app server is compatible with Go GCS client as long the user uses the online bucket? AND removing that Note from this part of the page, considering that, if the user is using the online bucket, the local filesystem is not used for anything in this case?

Another option, of course, is try to use the blob store as fallback to the GCS API automatically, as the Python SDK seems to do. But I think that it's not a nice option..Or, maybe there should be an IsDevAppServer() check in the client to check if the user is running in dev_appserver.py and automatically return an error or warning (?), if possible..But these are just options.

Anyway, is there any future roadmap to implement a GCS emulator in dev_appserver.py? GCS integration with AppEngine is great, specially when considering that AppEngine's users have a 5GB free bucket in GCS to use. But not having that integration locally in dev_appserver.py is....bad :/

@jba jba added the api: storage label Sep 11, 2016

@okdave

This comment has been minimized.

Copy link
Contributor

commented Sep 13, 2016

I've opened up a new issue on the App Engine repo (golang/appengine#21) to provide good emulator support. The python/java approach for GCS is to provide an App Engine-specific repo that can interface with dev_appserver, and that seems like a reasonable thing to do here.

I've also filed #368 to look at making the example in this repo more consistent with its peers in other languages.

@okdave okdave closed this Sep 13, 2016

@ram-nadella

This comment has been minimized.

Copy link

commented Sep 13, 2016

Cool, thanks for keeping track of this and creating the separate issue.

As a side note, I put together the stop-gap alternative I described in one of my comments above (use the staging prod bucket) as a blog post, hope it helps someone that runs into the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.