(POC) (8) Manage credentials according to the cloud provider #76

Closed
herlo opened this Issue Oct 14, 2016 · 10 comments

Comments

Projects
None yet
3 participants
@herlo
Member

herlo commented Oct 14, 2016

Currently, credentials are assigned in the vars of the role. This can obviously be overwritten by global vars or possibly in the linchpin_config.yml. However, many of the cloud providers (openstack, rackspace, aws/ec2, gce) have alternative ways of authenticating with configuration files they already provide.

The rationale here is that users will want to utilize things they already know. It isn't advantageous to add complexity by providing another way to configure credentials. We should provide a way to access the configs that already exist or are already documented.

As an example, check out http://docs.openstack.org/developer/os-client-config/.

@herlo

This comment has been minimized.

Show comment
Hide comment
@herlo

herlo Oct 24, 2016

Member

This is being addressed by a newly designed auth driver. @samvarankashyap, will you provide a basic design concept here in the near future?

Member

herlo commented Oct 24, 2016

This is being addressed by a newly designed auth driver. @samvarankashyap, will you provide a basic design concept here in the near future?

@herlo herlo added this to the v0.9 milestone Oct 24, 2016

@herlo

This comment has been minimized.

Show comment
Hide comment

@herlo herlo modified the milestones: v1.0.0, v0.9.0 Feb 7, 2017

@herlo herlo changed the title from Manage credentials according to the cloud provider to (POC) (8) Manage credentials according to the cloud provider May 3, 2017

@herlo

This comment has been minimized.

Show comment
Hide comment
@herlo

herlo May 4, 2017

Member

After the design discussion, an update to our topology and command-line functionality is needed. It essentially came to this example topology:

---                                                                                                                                      
    topology_name: "example_topo"                                                                                                        
    site: "qeos"                                                                                                                         
    resource_groups:                                                                                                                     
        credentials:                                                                                                                     
          - profile: e2e-openstack                                                                                                       
          - auth_type: file:secure.yaml                                                                                                  
        res_group_type: "openstack"                                                                                                      
        res_defs:                                                                                                                        
          -                                                                                                                              
            res_name: "ha_inst"                                                                                                          
            flavor: "m1.small"                                                                                                           
            res_type: "os_server"                                                                                                        
            image: "rhel-6.5_jeos"                                                                                                       
            count: 1                                                                                                                     
            keypair: "ci-factory"                                                                                                        
            networks:                                                                                                                    
              - "e2e-openstack"                                                                                                          
      -                                                                                                                                  
        resource_group_name: "testgroup1"                                                                                                
        credentials:                                                                                                                     
          - profile: ec3-awesome                                                                                                         
          - auth_type: file:ec3.ini                                                                                                      
        res_group_type: "aws"                                                                                                            
        res_defs:                                                                                                                        
          -                                                                                                                              
            res_name: "web_inst"                                                                                                         
            flavor: "m1.small"                                                                                                           
            res_type: "ec2"                                                                                                              
            image: "rhel-6.5_jeos"                                                                                                       
            count: 1                                                                                                                     
            keypair: "ci-factory"                                                                                                        
            networks:                                                                                                                    
              - "e2e-openstack" 
Member

herlo commented May 4, 2017

After the design discussion, an update to our topology and command-line functionality is needed. It essentially came to this example topology:

---                                                                                                                                      
    topology_name: "example_topo"                                                                                                        
    site: "qeos"                                                                                                                         
    resource_groups:                                                                                                                     
        credentials:                                                                                                                     
          - profile: e2e-openstack                                                                                                       
          - auth_type: file:secure.yaml                                                                                                  
        res_group_type: "openstack"                                                                                                      
        res_defs:                                                                                                                        
          -                                                                                                                              
            res_name: "ha_inst"                                                                                                          
            flavor: "m1.small"                                                                                                           
            res_type: "os_server"                                                                                                        
            image: "rhel-6.5_jeos"                                                                                                       
            count: 1                                                                                                                     
            keypair: "ci-factory"                                                                                                        
            networks:                                                                                                                    
              - "e2e-openstack"                                                                                                          
      -                                                                                                                                  
        resource_group_name: "testgroup1"                                                                                                
        credentials:                                                                                                                     
          - profile: ec3-awesome                                                                                                         
          - auth_type: file:ec3.ini                                                                                                      
        res_group_type: "aws"                                                                                                            
        res_defs:                                                                                                                        
          -                                                                                                                              
            res_name: "web_inst"                                                                                                         
            flavor: "m1.small"                                                                                                           
            res_type: "ec2"                                                                                                              
            image: "rhel-6.5_jeos"                                                                                                       
            count: 1                                                                                                                     
            keypair: "ci-factory"                                                                                                        
            networks:                                                                                                                    
              - "e2e-openstack" 
@samvarankashyap

This comment has been minimized.

Show comment
Hide comment
@samvarankashyap

samvarankashyap May 9, 2017

Collaborator

As per discussion with @herlo . We came to following agreements:

  1. The credentials will be handled by default mechanism followed by ansible modules (i.e, ENV variables) if not provided .
  2. However , end user will have an option to override the credentials by providing --cred-path LINCHPIN_CREDS environment variable which should point to one or more folder paths.
  3. If there is no override, the credentials will be searched in "credentials" folder in the Linchpin workspace.
  4. All the credentials will be resolved by the credential file name.
  5. If there are multiple credential_paths are given they are resolved on first found basis.
  6. Order of resolution:
    i. creds_path
    ii. LINCHPIN_CREDS env var
    iii. credentials folder in workspace
    iv. Environment variables ( OS_USERNAME, AWS_SECRETID etc.,)
    v. default creds, ie., clouds.yaml/ boto.ini
Collaborator

samvarankashyap commented May 9, 2017

As per discussion with @herlo . We came to following agreements:

  1. The credentials will be handled by default mechanism followed by ansible modules (i.e, ENV variables) if not provided .
  2. However , end user will have an option to override the credentials by providing --cred-path LINCHPIN_CREDS environment variable which should point to one or more folder paths.
  3. If there is no override, the credentials will be searched in "credentials" folder in the Linchpin workspace.
  4. All the credentials will be resolved by the credential file name.
  5. If there are multiple credential_paths are given they are resolved on first found basis.
  6. Order of resolution:
    i. creds_path
    ii. LINCHPIN_CREDS env var
    iii. credentials folder in workspace
    iv. Environment variables ( OS_USERNAME, AWS_SECRETID etc.,)
    v. default creds, ie., clouds.yaml/ boto.ini
@herlo

This comment has been minimized.

Show comment
Hide comment
@herlo

herlo May 9, 2017

Member

The order of resolution seems incorrect, iv and v should be swapped. This is a first found resolution, therefore Environrment variables should be resolved first, because they take precedence over the configuration files.

Member

herlo commented May 9, 2017

The order of resolution seems incorrect, iv and v should be swapped. This is a first found resolution, therefore Environrment variables should be resolved first, because they take precedence over the configuration files.

@gsr-shanks

This comment has been minimized.

Show comment
Hide comment
@gsr-shanks

gsr-shanks May 9, 2017

IMO the order should be:

  1. LINCHPIN_CREDS env var
  2. Environment variables ( OS_USERNAME, AWS_SECRETID etc.,)
  3. creds_path
  4. credentials folder in workspace
  5. default creds, ie., clouds.yaml/ boto.ini

First 2 are env vars which takes precedence over any and the next 2 require user action before the default kicks in.

- shanks

IMO the order should be:

  1. LINCHPIN_CREDS env var
  2. Environment variables ( OS_USERNAME, AWS_SECRETID etc.,)
  3. creds_path
  4. credentials folder in workspace
  5. default creds, ie., clouds.yaml/ boto.ini

First 2 are env vars which takes precedence over any and the next 2 require user action before the default kicks in.

- shanks

@herlo

This comment has been minimized.

Show comment
Hide comment
@herlo

herlo May 9, 2017

Member

I think some clarification on what these things mean might be helpful. I agree that environment variables take precedence, except when a value is passed on the command line.

The 'creds_path' value listed here is actually a command line option. It should have been --creds-path, which the click library will turn into creds_path variable. The environment variable associated with this can also be used. Thus, the order should l probably be closer to:

  1. --creds_path on the cli
  2. LINCHPIN_CREDS (which is passed via the creds_path variable using click)
  3. ENV_VARS (OS_USERNAME, etc...)
  4. credentials folder in workspace, with appropriate directory structure and config file
  5. default credentials (eg. /etc/clouds.yaml)

@samvarankashyap, @gsr-shanks what do you think?

Member

herlo commented May 9, 2017

I think some clarification on what these things mean might be helpful. I agree that environment variables take precedence, except when a value is passed on the command line.

The 'creds_path' value listed here is actually a command line option. It should have been --creds-path, which the click library will turn into creds_path variable. The environment variable associated with this can also be used. Thus, the order should l probably be closer to:

  1. --creds_path on the cli
  2. LINCHPIN_CREDS (which is passed via the creds_path variable using click)
  3. ENV_VARS (OS_USERNAME, etc...)
  4. credentials folder in workspace, with appropriate directory structure and config file
  5. default credentials (eg. /etc/clouds.yaml)

@samvarankashyap, @gsr-shanks what do you think?

@samvarankashyap

This comment has been minimized.

Show comment
Hide comment
@samvarankashyap

samvarankashyap May 10, 2017

Collaborator

With the current workflow it would be difficult to accommodate the above resolution. But, I feel following is better.

  1. --creds_path on the cli ( possible)
  2. LINCHPIN_CREDS (which is passed via the creds_path variable using click) ( possible use env var if not mentioned)
  3. credentials folder in workspace, with appropriate directory structure and config file
  4. ENV_VARS (OS_USERNAME, etc...)
  5. default credentials (eg. /etc/clouds.yaml) as these are creds configured system wide. ~/.boto.ini
    env_vars becomes the last resort of nothing is mentioned anywhere. (including the topology file making credentials parameter optional)

I feel one should resolve to defaults if there is no credentials are mentioned.
I assume, 4 and 5 are handled by default by the respective modules of openstack and aws ansible modules.
@herlo @gsr-shanks Let me know your opinion . ?

Collaborator

samvarankashyap commented May 10, 2017

With the current workflow it would be difficult to accommodate the above resolution. But, I feel following is better.

  1. --creds_path on the cli ( possible)
  2. LINCHPIN_CREDS (which is passed via the creds_path variable using click) ( possible use env var if not mentioned)
  3. credentials folder in workspace, with appropriate directory structure and config file
  4. ENV_VARS (OS_USERNAME, etc...)
  5. default credentials (eg. /etc/clouds.yaml) as these are creds configured system wide. ~/.boto.ini
    env_vars becomes the last resort of nothing is mentioned anywhere. (including the topology file making credentials parameter optional)

I feel one should resolve to defaults if there is no credentials are mentioned.
I assume, 4 and 5 are handled by default by the respective modules of openstack and aws ansible modules.
@herlo @gsr-shanks Let me know your opinion . ?

@herlo

This comment has been minimized.

Show comment
Hide comment
@herlo

herlo May 10, 2017

Member

To clarify, the option is --creds-path on the cli (with a dash, not an underscore). Click converts this to creds_path (with an underscore).

As for the order of resolution, I can agree with both points of view. What if we were to remove the credentials folder requirement? A user can still have that simply by pointing the --creds-path or LP_CREDENTIALS to the workspace path.

Member

herlo commented May 10, 2017

To clarify, the option is --creds-path on the cli (with a dash, not an underscore). Click converts this to creds_path (with an underscore).

As for the order of resolution, I can agree with both points of view. What if we were to remove the credentials folder requirement? A user can still have that simply by pointing the --creds-path or LP_CREDENTIALS to the workspace path.

@herlo

This comment has been minimized.

Show comment
Hide comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment