-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add appengine/bigquery sample #2
Changes from 2 commits
e6f5982
85f10a2
725f28d
bc66553
a1d280f
af1a967
d2b6b78
c07f926
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,168 @@ | ||
# Accessing BigQuery from App Engine | ||
|
||
A Google Cloud Platform customer asked me today how to list all the BigQuery | ||
projects that you own from a Google App Engine app. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This reads like a blog post. You may want to add your name above, or re-word so it's more like a normal README. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I removed this so it points to the medium post instead. |
||
|
||
If you don’t know what BigQuery or App Engine are this post is probably not | ||
for you … yet! Instead you should have a look at the docs for [BigQuery][1], | ||
and [App Engine][2], two of my favorite products of [Google Cloud Platform][0]. | ||
|
||
[![Google Cloud Platform Logo](https://cloud.google.com/_static/images/new-gcp-logo.png)][0] | ||
|
||
The solution for this is quite simple, but I think there’s enough | ||
moving pieces than a blog post was required. Let’s assume that the list will | ||
be displayed as part of some the handling to a request, something like this: | ||
|
||
```go | ||
func handle(w http.ResponseWriter, r *http.Request) { | ||
|
||
// create a new App Engine context from the request. | ||
c := appengine.NewContext(r) | ||
|
||
// obtain the list of project names. | ||
names, err := projects(c) | ||
|
||
if err != nil { | ||
http.Error(w, err.Error(), http.StatusInternalServerError) | ||
return | ||
} | ||
|
||
// print it to the output. | ||
fmt.Fprintln(w, strings.Join(names, "\n")) | ||
|
||
} | ||
``` | ||
|
||
Every time a new HTTP request comes the handler define above will be executed. | ||
It will create a new App Engine context that will then be used by the `projects` | ||
function to return a list with all the names of the BigQuery projects visible | ||
by the App Engine app. Finally, the list will be printed to the output `w`. | ||
|
||
In order to register the handler so it is executed on every HTTP request we add | ||
an `init` func. | ||
|
||
```go | ||
func init() { | ||
|
||
// all requests are handled by handler. | ||
http.HandleFunc(“/”, handle) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. smart quotes There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
|
||
} | ||
``` | ||
|
||
Now for the interesting part, how do we implement the `projects` function? | ||
|
||
```go | ||
func projects(c context.Context) ([]string, error) { | ||
|
||
// some awesome code ... | ||
|
||
} | ||
``` | ||
|
||
Let’s implement the body of that function in three parts: | ||
|
||
## Create an authenticated HTTP client | ||
|
||
Given a [`context.Context`][3] named `c` we can create an authenticated client | ||
using this code. | ||
|
||
```go | ||
// create a new HTTP client. | ||
client := &http.Client{ | ||
Transport: &oauth2.Transport{ | ||
Source: google.AppEngineTokenSource(c, | ||
bigquery.BigqueryScope), | ||
Base: &urlfetch.Transport{Context: c}, | ||
}, | ||
} | ||
``` | ||
|
||
In Go, HTTP clients use transports to communicate, and transports use … | ||
transports! It’s transports all the way! But what is a HTTP transport | ||
exactly? | ||
|
||
The Transport field in an HTTP client is of type [`RoundTripper`][4]: | ||
|
||
```go | ||
type RoundTripper interface { | ||
RoundTrip(*Request) (*Response, error) | ||
} | ||
``` | ||
|
||
A [`RoundTripper`][4] is responsible for generating a response given a request, | ||
but it also has the capacity of changing the request and response being set. | ||
This is very useful for monitoring, instrumenting, and also authentication. | ||
|
||
The [`oauth2.Transport`][5] type, given a request, adds authentication headers and | ||
forwards the request through its base transport. | ||
|
||
```go | ||
&oauth2.Transport{ | ||
Source: google.AppEngineTokenSource(c, bigquery.BigqueryScope), | ||
Base: &urlfetch.Transport{Context: c}, | ||
} | ||
``` | ||
|
||
The snippet above creates a new [`oauth2.Transport`][5] that authenticates | ||
requests using the default service account for the App Engine project, and then | ||
uses an HTTP transport provided by [`urlfetch`][6] — the way of accessing external | ||
resources from App Engine apps. | ||
|
||
## Create a BigQuery service and list all projects | ||
|
||
Creating a BigQuery service with [`google.golang.org/api/bigquery/v2`][7] is | ||
quite simple. Just import the package and create a new service given an HTTP | ||
client with [`bigquery.New`][8]. | ||
|
||
```go | ||
bq, err := bigquery.New(client) | ||
if err != nil { | ||
return nil, fmt.Errorf("create service: %v", err) | ||
} | ||
``` | ||
|
||
It’s important to check that the error is not nil, since the operation could | ||
fail as any other operation over the network. Once we have the service, we can | ||
list all the projects by following the documentation: | ||
|
||
```go | ||
list, err := bq.Projects.List().Do() | ||
if err != nil { | ||
return nil, fmt.Errorf("list projects: %v", err) | ||
} | ||
``` | ||
|
||
## Create a list with the project names | ||
|
||
We got list, a [`bigquery.ProjectList`][9], so we can iterate over all the | ||
projects in `Projects` and append their `FriendlyName` to our list before | ||
returning it. | ||
|
||
```go | ||
var names []string | ||
|
||
for _, p := range list.Projects { | ||
names = append(names, p.FriendlyName) | ||
} | ||
|
||
return names, nil | ||
``` | ||
|
||
I hope this will be helpful for many of you! | ||
|
||
## Questions? | ||
|
||
Feel free to file issues or contact me on [Twitter][10]. | ||
|
||
[0]: https://cloud.google.com | ||
[1]: https://cloud.google.com/bigquery/what-is-bigquery | ||
[2]: https://cloud.google.com/appengine/docs | ||
[3]: https://godoc.org/golang.org/x/net/context#Context | ||
[4]: https://golang.org/pkg/net/http/#RoundTripper | ||
[5]: https://godoc.org/golang.org/x/oauth2#Transport | ||
[6]: https://cloud.google.com/appengine/docs/go/urlfetch/ | ||
[7]: https://godoc.org/google.golang.org/api/bigquery/v2 | ||
[8]: https://godoc.org/google.golang.org/api/bigquery/v2#New | ||
[9]: https://godoc.org/google.golang.org/api/bigquery/v2#ProjectList | ||
[10]: http://twitter.com/francesc |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
// Copyright 2015 Google Inc. All rights reserved. | ||
// Use of this source code is governed by the Apache 2.0 | ||
// license that can be found in the LICENSE file. | ||
|
||
// This App Engine application uses its default service account to list all | ||
// the BigQuery projects accessible via the BigQuery REST API. | ||
package sample | ||
|
||
import ( | ||
"fmt" | ||
"net/http" | ||
"strings" | ||
|
||
"golang.org/x/net/context" | ||
"golang.org/x/oauth2" | ||
"golang.org/x/oauth2/google" | ||
"google.golang.org/api/bigquery/v2" | ||
"google.golang.org/appengine" | ||
"google.golang.org/appengine/urlfetch" | ||
) | ||
|
||
func init() { | ||
// all requests are handled by handler. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. remove this comment There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. see comment above. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I mean, it's not a very useful comment. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
http.HandleFunc("/", handle) | ||
} | ||
|
||
func handle(w http.ResponseWriter, r *http.Request) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's fine like this, ANY url will be handled by this. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Including There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
// create a new App Engine context from the request. | ||
c := appengine.NewContext(r) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we want ( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
|
||
// obtain the list of project names. | ||
names, err := projects(c) | ||
if err != nil { | ||
http.Error(w, err.Error(), http.StatusInternalServerError) | ||
return | ||
} | ||
|
||
// print it to the output. | ||
fmt.Fprintln(w, strings.Join(names, "\n")) | ||
} | ||
|
||
// projects returns a list with the names of all the Big Query projects visible | ||
// with the given context. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it seems like it'll list all the bigquery projects in the same project as the running App Engine app. |
||
func projects(c context.Context) ([]string, error) { | ||
// create a new authenticated HTTP client over urlfetch. | ||
client := &http.Client{ | ||
Transport: &oauth2.Transport{ | ||
Source: google.AppEngineTokenSource(c, bigquery.BigqueryScope), | ||
Base: &urlfetch.Transport{Context: c}, | ||
}, | ||
} | ||
|
||
// create the BigQuery service. | ||
bq, err := bigquery.New(client) | ||
if err != nil { | ||
return nil, fmt.Errorf("create service: %v", err) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
} | ||
|
||
// list the projects. | ||
list, err := bq.Projects.List().Do() | ||
if err != nil { | ||
return nil, fmt.Errorf("list projects: %v", err) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
} | ||
|
||
// prepare the list of names. | ||
var names []string | ||
for _, p := range list.Projects { | ||
names = append(names, p.FriendlyName) | ||
} | ||
return names, nil | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# Copyright 2015 Google Inc. All rights reserved. | ||
# Use of this source code is governed by the Apache 2.0 | ||
# license that can be found in the LICENSE file. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. no need for copyright header on app.yaml There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is that new? ok, done |
||
application: gae-bq-sample | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this still required for App Engine? I think you can specify it on the command line. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
version: 1 | ||
runtime: go | ||
api_version: go1 | ||
|
||
handlers: | ||
- url: /.* | ||
script: _go_app |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a short section with deployment instructions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done