Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloud Authorization #5903

Merged
merged 80 commits into from Sep 24, 2018
Merged

Cloud Authorization #5903

merged 80 commits into from Sep 24, 2018

Conversation

@VJalili
Copy link
Member

VJalili commented Apr 17, 2018

This PR implements the logic of accessing a user's cloud-based resources without asking for their credentials, leveraging CloudAuthz. In general, this PR implements the following:

  1. defines a cloudauthz class in Galaxy model;
  2. implements migration script to create a table in Galaxy database to store instances of the new class;
  3. updates mappings by required joins;
  4. creates API and manager so that user can define a cloudauthz instance;
  5. integrates cloudauthz with AuthnzManager;
  6. replaces credentials with a cloudauthz ID in the cloud upload API;
  7. updates CloudManager to leverage cloudauthz to get credentials.
  8. updates CloudManager to adhere with v1.0.0 cloudbridge interface.

This PR requires the following to updated:

  • Version 1.0.0 of CloudBridge; #6660
  • Update cloudauthz to resolve dependency conflicts with Galaxy;
  • Some of the raised exceptions cause 500 error (e.g., exceptions in galaxy/api/cloudauthz.py).
@VJalili VJalili changed the title Cloud Authorization Cloud Authorization #1 Apr 17, 2018
@galaxybot galaxybot added the triage label Apr 17, 2018
@galaxybot galaxybot added this to the 18.05 milestone Apr 17, 2018
@jmchilton

This comment has been minimized.

Copy link
Member

jmchilton commented Apr 20, 2018

Thanks! I have pretty much the same comment as here #5835 (comment).

@martenson

This comment has been minimized.

Copy link
Member

martenson commented Apr 23, 2018

I am pushing this back to 18.09 since we probably won't have enough time to test is and there are unaddressed comments.

@VJalili please let me know if this complicates things significantly

@martenson martenson modified the milestones: 18.05, 18.09 Apr 23, 2018
@VJalili

This comment has been minimized.

Copy link
Member Author

VJalili commented Apr 23, 2018

@jmchilton Thanks for reviewing this. The exception handling is now updated, and it uses galaxy.exceptions as you recommended.

@VJalili VJalili changed the title Cloud Authorization #1 Cloud Authorization #1 (model and API) Apr 25, 2018
@VJalili VJalili mentioned this pull request Aug 9, 2018
@afgane afgane merged commit 5355fe9 into galaxyproject:dev Sep 24, 2018
6 checks passed
6 checks passed
api test Build finished. 422 tests run, 1 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 187 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 118 tests run, 5 skipped, 0 failed.
Details
selenium test Build finished. 146 tests run, 4 skipped, 0 failed.
Details
toolshed test Build finished. 577 tests run, 0 skipped, 0 failed.
Details
@VJalili VJalili deleted the VJalili:cloudauthz branch Sep 24, 2018
@VJalili

This comment has been minimized.

Copy link
Member Author

VJalili commented Sep 24, 2018

@afgane Thanks!

@martenson martenson added area/API and removed status/WIP labels Sep 24, 2018
@afgane

This comment has been minimized.

Copy link
Contributor

afgane commented Sep 24, 2018

The high level usage model here is as follows:

  1. Head to https://console.cloud.google.com/apis/credentials and create a OAuth 2.0 client ID
  2. Enter those values into config/oidc_backends_config.xml
  3. Head to AWS IAM dashboard, create a role with desired policy (e.g. AmazonS3ReadOnlyAccess), and attach the equivalent of the trust relationship included below. Note the role ARN.
  4. Start Galaxy and GET http://127.0.0.1:8080/authnz; note down the id
  5. Create a new authz entry by issuing a POST with the content included below
  6. Test that it all works by uploading a file from a private bucket by issuing another POST request (see body below)
  • Trust relationship (where the value for accounts.google.com:aud is the respective client_id from the Google OAuth client registration):
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "accounts.google.com:aud": "893677542423-4t761afe33k3o9mu56u6p6p1ctk8o89f.apps.googleusercontent.com"
        }
      }
    }
  ]
}
  • Create authz entry body:
{
  "provider": "aws",
  "authn_id":"f2db41e1fa331b3e",
  "config": {
  	"role_arn": "arn:aws:iam::861460482541:role/s3-read-access"
  }
}
  • Upload data from S3
{
  "history_id": "f597429621d6eb2b",
  "bucket": "cm-log",
  "objects": ["cm_log2011-01-13-17-20-31-213597BDDB2013BE", "cm_log2011-01-13-17-20-46-1D133AFF95259DFE"],
  "authz_id": "f2db41e1fa331b3e"
}
@martenson

This comment has been minimized.

Copy link
Member

martenson commented Sep 24, 2018

@afgane ideally this would go to docs I think

@afgane

This comment has been minimized.

Copy link
Contributor

afgane commented Sep 24, 2018

It will, with a lot more detail, but for now these are the cliff notes.

@expose_api
def create(self, trans, payload, **kwargs):
"""
* POST /api/cloud/authz/create

This comment has been minimized.

Copy link
@jmchilton

jmchilton Sep 25, 2018

Member

Probably this should be a POST to /api/cloud/authz instead?

@@ -295,6 +295,9 @@ def populate_api_routes(webapp, app):
webapp.mapper.resource('group', 'groups', path_prefix='/api')
webapp.mapper.resource_with_deleted('quota', 'quotas', path_prefix='/api')

webapp.mapper.connect('/api/cloud/authz/', action='index', controller='cloudauthz')
webapp.mapper.connect('/api/cloud/authz/create', action='create', controller='cloudauthz')

This comment has been minimized.

Copy link
@jmchilton

jmchilton Sep 25, 2018

Member

So here you can specify actions, specify the one above to be a GET and this one to be a POST. There are examples in this file.

This comment has been minimized.

Copy link
@VJalili

VJalili Oct 15, 2018

Author Member

Please check the following PR and let me know if that address your point: #6876

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.