-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor AWS object store auth to be as simple and flexible as Google FindCredentials
.
#209
Comments
Hey, first of all, we would like to aim for having a minimum amount of code required to setup each provider's clients. I know it's not easy (: For example for GCP (GCS + optional tracing) we just create // FindDefaultCredentials searches for "Application Default Credentials".
//
// It looks for credentials in the following places,
// preferring the first location found:
//
// 1. A JSON file whose path is specified by the
// GOOGLE_APPLICATION_CREDENTIALS environment variable.
// 2. A JSON file in a location known to the gcloud command-line tool.
// On Windows, this is %APPDATA%/gcloud/application_default_credentials.json.
// On other systems, $HOME/.config/gcloud/application_default_credentials.json.
// 3. On Google App Engine it uses the appengine.AccessToken function.
// 4. On Google Compute Engine and Google App Engine Managed VMs, it fetches
// credentials from the metadata server.
// (In this final case any provided scopes are ignored.)
func FindDefaultCredentials(ctx context.Context, scope ...string) (*DefaultCredentials, error) {
// First, try the environment variable.
const envVar = "GOOGLE_APPLICATION_CREDENTIALS"
if filename := os.Getenv(envVar); filename != "" {
creds, err := readCredentialsFile(ctx, filename, scope)
if err != nil {
return nil, fmt.Errorf("google: error getting credentials using %v environment variable: %v", envVar, err)
}
return creds, nil
}
// Second, try a well-known file.
filename := wellKnownFile()
if creds, err := readCredentialsFile(ctx, filename, scope); err == nil {
return creds, nil
} else if !os.IsNotExist(err) {
return nil, fmt.Errorf("google: error getting credentials using well-known file (%v): %v", filename, err)
}
// Third, if we're on Google App Engine use those credentials.
if appengineTokenFunc != nil && !appengineFlex {
return &DefaultCredentials{
ProjectID: appengineAppIDFunc(ctx),
TokenSource: AppEngineTokenSource(ctx, scope...),
}, nil
}
// Fourth, if we're on Google Compute Engine use the metadata server.
if metadata.OnGCE() {
id, _ := metadata.ProjectID()
return &DefaultCredentials{
ProjectID: id,
TokenSource: ComputeTokenSource(""),
}, nil
}
// None are found; return helpful error.
const url = "https://developers.google.com/accounts/docs/application-default-credentials"
return nil, fmt.Errorf("google: could not find default credentials. See %v for more information.", url)
} As you can see it supports different things, including specifying If we can construct similar function for AWS (or import anything existing and small), that would be really, really great! Additionally, it is worth to note that it would nice to support cases when user cannot modify |
Also it is worth maybe to ping @TimSimmons for some feedback. Tim wrote the S3 client. (: |
As long as it's not at the exclusion of the other method, specifying secret/access keys, bucket etc is how you can get access to a lot of other environments that offer S3 capability, but not the explicit IAM functionality. It seems like you could try grabbing that info in the same function where it checks to see if the S3 env vars/flags are set. That seems like a smart solution to me.
|
FindCredentials
. (remove all boilerplate flags)
FindCredentials
. (remove all boilerplate flags)FindCredentials
.
Could I take this on? |
Sure, but what's your plan? (: Also have in mind this will come soon (common tests for all providers) #327, but it should change much. |
I essentially just want to somewhat emulate the actual aws-go-sdk with regards to how it tries to resolve the credentials by default by first checking arguments/environmental variables and then using the instance profile/role/metadata/iam route (the official sdk also looksfor the shared secret config, but that doesn't seem within the scope of this issue). The necessary API already exists in minio-go. The s3 logic in Thanos might need to change a little to account for using roles, which requires an additional and different step compared to using the typical access key/secret key combination. |
SGTM @rackonnoiter |
Ideally it should detect if there's a metadata service answering and get the credentials from there. If thanos is running on kubernetes that has kube2iam or kiam or in AWS. Another thing to consider is that those credentials have a reduced lifespan I think 12 hours, so the logic should take that into consideration. @rackonnoiter does the minio-go has something similar to a CredentialsChainProvider ? |
That's exactly what I was suggesting, @jescarri, and yes. |
Is there any library for this we can reuse, without reimplementing all? In the same avoiding pulling whole AWS SDK as dependency would be nice as well (: |
All the necessary pieces should exist in |
Submitted a WIP PR. I didn't seem to find any contributing guidance, could someone point me to the right way to go about testing my changes? For an overview, these changes can use multiple auth providers and searches in this order:
|
Thanks will take a look. Any particular info you are looking for? What would you expect from contributing guidance? |
I mainly just want to know what checks I need to complete prior to submitting a PR. There also seems to be some dependencies in setting my environment for the tests to work, so explicitly listing them would be appreciated as well. Thank you, @Bplotka! |
…mmand line Support retrieving credentials from IAM instance profile, `~/.aws/credentials` and standard AWS environment variables Fixes thanos-io#209
…mmand line Support retrieving credentials from IAM instance profile, `~/.aws/credentials` and standard AWS environment variables Fixes thanos-io#209
While running in AWS, you can use the instance profile to get an auth token, but currently Thanos doesn't configure the bucket if no credentials are provided.
minio supports IAM :
Would you consider a PR ?
The text was updated successfully, but these errors were encountered: