Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add appengine/bigquery sample #2

Closed
wants to merge 8 commits into from
Closed
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 168 additions & 0 deletions appengine/bigquery/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
# Accessing BigQuery from App Engine
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a short section with deployment instructions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


A Google Cloud Platform customer asked me today how to list all the BigQuery
projects that you own from a Google App Engine app.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads like a blog post. You may want to add your name above, or re-word so it's more like a normal README.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this so it points to the medium post instead.


If you don’t know what BigQuery or App Engine are this post is probably not
for you … yet! Instead you should have a look at the docs for [BigQuery][1],
and [App Engine][2], two of my favorite products of [Google Cloud Platform][0].

[![Google Cloud Platform Logo](https://cloud.google.com/_static/images/new-gcp-logo.png)][0]

The solution for this is quite simple, but I think there’s enough
moving pieces than a blog post was required. Let’s assume that the list will
be displayed as part of some the handling to a request, something like this:

```go
func handle(w http.ResponseWriter, r *http.Request) {

// create a new App Engine context from the request.
c := appengine.NewContext(r)

// obtain the list of project names.
names, err := projects(c)

if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}

// print it to the output.
fmt.Fprintln(w, strings.Join(names, "\n"))

}
```

Every time a new HTTP request comes the handler define above will be executed.
It will create a new App Engine context that will then be used by the `projects`
function to return a list with all the names of the BigQuery projects visible
by the App Engine app. Finally, the list will be printed to the output `w`.

In order to register the handler so it is executed on every HTTP request we add
an `init` func.

```go
func init() {

// all requests are handled by handler.
http.HandleFunc(“/”, handle)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

smart quotes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


}
```

Now for the interesting part, how do we implement the `projects` function? 

```go
func projects(c context.Context) ([]string, error) {

// some awesome code ...

}
```

Let’s implement the body of that function in three parts:

## Create an authenticated HTTP client

Given a [`context.Context`][3] named `c` we can create an authenticated client
using this code.

```go
// create a new HTTP client.
client := &http.Client{
Transport: &oauth2.Transport{
Source: google.AppEngineTokenSource(c,
bigquery.BigqueryScope),
Base: &urlfetch.Transport{Context: c},
},
}
```

In Go, HTTP clients use transports to communicate, and transports use …
transports! It’s transports all the way! But what is a HTTP transport
exactly?

The Transport field in an HTTP client is of type [`RoundTripper`][4]:

```go
type RoundTripper interface {
RoundTrip(*Request) (*Response, error)
}
```

A [`RoundTripper`][4] is responsible for generating a response given a request,
but it also has the capacity of changing the request and response being set.
This is very useful for monitoring, instrumenting, and also authentication.

The [`oauth2.Transport`][5] type, given a request, adds authentication headers and
forwards the request through its base transport.

```go
&oauth2.Transport{
Source: google.AppEngineTokenSource(c, bigquery.BigqueryScope),
Base: &urlfetch.Transport{Context: c},
}
```

The snippet above creates a new [`oauth2.Transport`][5] that authenticates
requests using the default service account for the App Engine project, and then
uses an HTTP transport provided by [`urlfetch`][6]  —  the way of accessing external
resources from App Engine apps.

## Create a BigQuery service and list all projects

Creating a BigQuery service with [`google.golang.org/api/bigquery/v2`][7] is
quite simple. Just import the package and create a new service given an HTTP
client with [`bigquery.New`][8].

```go
bq, err := bigquery.New(client)
if err != nil {
return nil, fmt.Errorf("create service: %v", err)
}
```

It’s important to check that the error is not nil, since the operation could
fail as any other operation over the network. Once we have the service, we can
list all the projects by following the documentation:

```go
list, err := bq.Projects.List().Do()
if err != nil {
return nil, fmt.Errorf("list projects: %v", err)
}
```

## Create a list with the project names

We got list, a [`bigquery.ProjectList`][9], so we can iterate over all the
projects in `Projects` and append their `FriendlyName` to our list before
returning it.

```go
var names []string

for _, p := range list.Projects {
names = append(names, p.FriendlyName)
}

return names, nil
```

I hope this will be helpful for many of you!

## Questions?

Feel free to file issues or contact me on [Twitter][10].

[0]: https://cloud.google.com
[1]: https://cloud.google.com/bigquery/what-is-bigquery
[2]: https://cloud.google.com/appengine/docs
[3]: https://godoc.org/golang.org/x/net/context#Context
[4]: https://golang.org/pkg/net/http/#RoundTripper
[5]: https://godoc.org/golang.org/x/oauth2#Transport
[6]: https://cloud.google.com/appengine/docs/go/urlfetch/
[7]: https://godoc.org/google.golang.org/api/bigquery/v2
[8]: https://godoc.org/google.golang.org/api/bigquery/v2#New
[9]: https://godoc.org/google.golang.org/api/bigquery/v2#ProjectList
[10]: http://twitter.com/francesc
71 changes: 71 additions & 0 deletions appengine/bigquery/app.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
// Copyright 2015 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.

// This App Engine application uses its default service account to list all
// the BigQuery projects accessible via the BigQuery REST API.
package sample

import (
"fmt"
"net/http"
"strings"

"golang.org/x/net/context"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"google.golang.org/api/bigquery/v2"
"google.golang.org/appengine"
"google.golang.org/appengine/urlfetch"
)

func init() {
// all requests are handled by handler.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this comment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comment above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, it's not a very useful comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

http.HandleFunc("/", handle)
}

func handle(w http.ResponseWriter, r *http.Request) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if r.URL.Path != "/" {
// 404
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine like this, ANY url will be handled by this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Including /favicon.ico. It's wasteful to basically double up on requests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

// create a new App Engine context from the request.
c := appengine.NewContext(r)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want c -> ctx.

(ctx because we're using the actual context package now.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


// obtain the list of project names.
names, err := projects(c)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}

// print it to the output.
fmt.Fprintln(w, strings.Join(names, "\n"))
}

// projects returns a list with the names of all the Big Query projects visible
// with the given context.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems like it'll list all the bigquery projects in the same project as the running App Engine app.

func projects(c context.Context) ([]string, error) {
// create a new authenticated HTTP client over urlfetch.
client := &http.Client{
Transport: &oauth2.Transport{
Source: google.AppEngineTokenSource(c, bigquery.BigqueryScope),
Base: &urlfetch.Transport{Context: c},
},
}

// create the BigQuery service.
bq, err := bigquery.New(client)
if err != nil {
return nil, fmt.Errorf("create service: %v", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could not create service:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}

// list the projects.
list, err := bq.Projects.List().Do()
if err != nil {
return nil, fmt.Errorf("list projects: %v", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could not list projects:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}

// prepare the list of names.
var names []string
for _, p := range list.Projects {
names = append(names, p.FriendlyName)
}
return names, nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe

if len(list.Projects) == 0 {
  return nil, errors.New("no bigquery projects")
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}
11 changes: 11 additions & 0 deletions appengine/bigquery/app.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Copyright 2015 Google Inc. All rights reserved.
# Use of this source code is governed by the Apache 2.0
# license that can be found in the LICENSE file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for copyright header on app.yaml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is that new? ok, done

application: gae-bq-sample
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still required for App Engine? I think you can specify it on the command line.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

version: 1
runtime: go
api_version: go1

handlers:
- url: /.*
script: _go_app