Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

config: Add possibility to inline ServiceAccount into GCS config #963

Merged
merged 3 commits into from
Mar 26, 2019
Merged

config: Add possibility to inline ServiceAccount into GCS config #963

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6c3a22c
Add possibility to inline ServiceAccount into GCS config
metalmatze Mar 22, 2019
21e0f69
Update comment in pkg/objstore/gcs/gcs.go
bwplotka Mar 26, 2019
76f47f3
Generate docs for GCS config
metalmatze Mar 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,27 @@ Application credentials are configured via JSON file, the client looks for:

You can read more on how to get application credential json file in [https://cloud.google.com/docs/authentication/production](https://cloud.google.com/docs/authentication/production)

Another possibility is to inline the ServiceAccount into the Thanos configuration and only maintain one file:

```yaml
type: GCS
config:
bucket: "thanos"
service_account: |-
{
"type": "service_account",
"project_id": "project",
"private_key_id": "abcdefghijklmnopqrstuvwxyz12345678906666",
"private_key": "-----BEGIN PRIVATE KEY-----\...\n-----END PRIVATE KEY-----\n",
"client_email": "project@thanos.iam.gserviceaccount.com",
"client_id": "123456789012345678901",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/thanos%40gitpods.iam.gserviceaccount.com"
}
```

### GCS Policies

For deployment:
Expand Down
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ require (
github.com/prometheus/tsdb v0.4.0
go.opencensus.io v0.19.0 // indirect
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd
golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223 // indirect
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2
Expand Down
23 changes: 20 additions & 3 deletions pkg/objstore/gcs/gcs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"github.com/improbable-eng/thanos/pkg/objstore"
"github.com/pkg/errors"
"github.com/prometheus/common/version"
"golang.org/x/oauth2/google"
"google.golang.org/api/iterator"
"google.golang.org/api/option"
yaml "gopkg.in/yaml.v2"
Expand All @@ -26,7 +27,8 @@ const DirDelim = "/"

// Config stores the configuration for gcs bucket.
type Config struct {
Bucket string `yaml:"bucket"`
Bucket string `yaml:"bucket"`
ServiceAccount string `yaml:"service_account"`
}

// Bucket implements the store.Bucket and shipper.Bucket interfaces against GCS.
Expand All @@ -47,8 +49,23 @@ func NewBucket(ctx context.Context, logger log.Logger, conf []byte, component st
if gc.Bucket == "" {
return nil, errors.New("missing Google Cloud Storage bucket name for stored blocks")
}
gcsOptions := option.WithUserAgent(fmt.Sprintf("thanos-%s/%s (%s)", component, version.Version, runtime.Version()))
gcsClient, err := storage.NewClient(ctx, gcsOptions)

var opts []option.ClientOption

// If ServiceAccount provided inside configuration use it, otherwise fallback to defaults
metalmatze marked this conversation as resolved.
Show resolved Hide resolved
if gc.ServiceAccount != "" {
credentials, err := google.CredentialsFromJSON(ctx, []byte(gc.ServiceAccount))
if err != nil {
return nil, errors.Wrap(err, "failed to create credentials from JSON")
}
opts = append(opts, option.WithCredentials(credentials))
}

opts = append(opts,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a side thing (not a blocker for this PR), can we unsure, "debug" name from debug.name flag is passed here? for component for example? It would help if use agent would be actually some name you configured I guess?

option.WithUserAgent(fmt.Sprintf("thanos-%s/%s (%s)", component, version.Version, runtime.Version())),
)

gcsClient, err := storage.NewClient(ctx, opts...)
if err != nil {
return nil, err
}
Expand Down