gce module errors on file not found when using OAuth client authentication #17075

laurentgo · 2016-08-13T00:42:17Z

ISSUE TYPE

Bug Report

COMPONENT NAME

gce module_util

ANSIBLE VERSION

ansible 2.1.1.0
  config file = 
  configured module search path = Default w/o overrides

CONFIGURATION

OS / ENVIRONMENT

N/A

SUMMARY

It is possible to connect to GCE using a clientId (feature named Installed Application authentication in apache-libcloud), which is useful when you want users to interact with GCE without creating service accounts and sharing credentials amongst your org.

In that mode, the GCE credential is not a json file, but the client secret string for the given clientId. Unfortunately a check in lib/module_utils/gce.py prevents the use of this authentication mode.

STEPS TO REPRODUCE

In GCE, create a new client id
open Google Cloud Console > API Manager > credentials
click on Create credentials > OAuth client id
choose Application Type: Other
note the client id/client secret
Run the following command:

GCE_EMAIL=<client-id> GCE_PEM_FILE_PATH=<client-secret> GCE_PROJECT=<project> ansible localhost -m gce -a name=test

EXPECTED RESULTS

ansible command should create a test instance (it might ask the user to open a browser to get a token id, but if user is already logged in, there's no interaction)

 [WARNING]: Host file not found: /usr/local/etc/ansible/hosts

 [WARNING]: provided hosts list is empty, only localhost is available

localhost | SUCCESS => {
    "changed": true, 
    "instance_data": [
        {
            "disks": [
                "test"
            ], 
            "image": "debian-7-wheezy-v20160531", 
            "machine_type": "n1-standard-1", 
            "metadata": {}, 
            "name": "test", 
            "network": "default", 
            "private_ip": "10.128.0.2", 
            "public_ip": "104.197.212.27", 
            "status": "RUNNING", 
            "tags": [], 
            "zone": "us-central1-a"
        }
    ], 
    "name": "test", 
    "state": "present", 
    "zone": "us-central1-a"
}

ACTUAL RESULTS

No config file found; using defaults
 [WARNING]: Host file not found: /usr/local/etc/ansible/hosts

 [WARNING]: provided hosts list is empty, only localhost is available

Loaded callback minimal of type stdout, v2.0
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: laurent
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1471048836.07-253572111479568 `" && echo ansible-tmp-1471048836.07-253572111479568="` echo $HOME/.ansible/tmp/ansible-tmp-1471048836.07-253572111479568 `" ) && sleep 0'
<127.0.0.1> PUT /var/folders/z0/x6glbwbn41gbb45rv3sq5tzr0000gn/T/tmpPhRg4A TO /Users/laurent/.ansible/tmp/ansible-tmp-1471048836.07-253572111479568/gce
<127.0.0.1> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /Users/laurent/.ansible/tmp/ansible-tmp-1471048836.07-253572111479568/gce; rm -rf "/Users/laurent/.ansible/tmp/ansible-tmp-1471048836.07-253572111479568/" > /dev/null 2>&1 && sleep 0'
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/var/folders/z0/x6glbwbn41gbb45rv3sq5tzr0000gn/T/ansible_YOIiDx/ansible_module_gce.py", line 640, in <module>
    main()
  File "/var/folders/z0/x6glbwbn41gbb45rv3sq5tzr0000gn/T/ansible_YOIiDx/ansible_module_gce.py", line 553, in main
    gce = gce_connect(module)
  File "/var/folders/z0/x6glbwbn41gbb45rv3sq5tzr0000gn/T/ansible_YOIiDx/ansible_modlib.zip/ansible/module_utils/gce.py", line 95, in gce_connect
IOError: [Errno 2] No such file or directory: '<client-secret>'

localhost | FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_name": "gce"
    }, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/z0/x6glbwbn41gbb45rv3sq5tzr0000gn/T/ansible_YOIiDx/ansible_module_gce.py\", line 640, in <module>\n    main()\n  File \"/var/folders/z0/x6glbwbn41gbb45rv3sq5tzr0000gn/T/ansible_YOIiDx/ansible_module_gce.py\", line 553, in main\n    gce = gce_connect(module)\n  File \"/var/folders/z0/x6glbwbn41gbb45rv3sq5tzr0000gn/T/ansible_YOIiDx/ansible_modlib.zip/ansible/module_utils/gce.py\", line 95, in gce_connect\nIOError: [Errno 2] No such file or directory: '<client-secret>'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "parsed": false
}

The text was updated successfully, but these errors were encountered:

alikins · 2016-08-15T14:50:32Z

Unfortunately a check in lib/module_utils/gce.py prevents the use of this authentication mode.

Which check?

alikins · 2016-08-15T14:55:28Z

Referring to https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/gcp.py#L88-L100 ?
(code moved from gce.py to gcp.py in devel/ branch)

        # We have credentials but lets make sure that if they are JSON we have the minimum
        # libcloud requirement met
        try:
            # Try to read credentials as JSON
            with open(credentials_file) as credentials:
                json.loads(credentials.read())
            # If the credentials are proper JSON and we do not have the minimum
            # required libcloud version, bail out and return a descriptive error
            if LooseVersion(libcloud.__version__) < '0.17.0':
                module.fail_json(msg='Using JSON credentials but libcloud minimum version not met. '
                                     'Upgrade to libcloud>=0.17.0.')
                return None
        except ValueError, e:
            # Not JSON
            pass

laurentgo · 2016-08-15T18:22:59Z

yes, that check

walbert947 · 2016-08-16T00:23:24Z

While a client ID and a client secret could technically work in place of a service e-mail and a JSON file, respectively (assuming the check were removed), neither the GCP module nor the Ansible modules that call it have ever claimed to support any authentication method other than service accounts. As such, I think this issue should be labeled as a feature idea rather than a bug report.

With respect to supporting the Installed Application auth method, I'd prefer to use new parameters for client id and client secret, rather than overloading the existing service account parameters. This would:

make it easier to validate the parameters and report relevant authentication errors,
it would be easier to avoid situations where parameters passed to apache-libcloud are mismatched (e.g., a service account e-mail with an installed app client secret),
documenting the new parameters in Ansible modules would be more straightforward than trying to explain the overloaded behavior, and
there wouldn't be a risk of breakage in modules (either Ansible or third-party) that consume the service account parameters themselves and expect them to have a particular format.

laurentgo · 2016-08-16T00:43:50Z

I'm fine using new parameters (I agree that the naming is unfortunate) but as far as I can tell, the environment variable used by ansible are the same as the ones used by libcloud in their examples. I guess they are convention based (the gce.py inventory script also use them but is not failing on client), but this might add to the confusion.

More generally, I wonder if having dfferent variables used for different types of authentication is worth it since:

the libcloud factory method is the same whatever the authentication method
libcloud does file validation for service account
in case Google decides to come up with a new authentication protocol, one will have to wait for both libcloud and ansible to be updated (instead of just libcloud).

Currently I only need to relax the gcp.py module validation of the arguments, which should be okay as libcloud performs validation too:
https://github.com/apache/libcloud/blob/trunk/libcloud/common/google.py#L478

laurentgo · 2016-11-19T20:33:43Z

I believe this check also causes issue regarding GCE internal authentication (the ability to get the credentials from a GCE VM metadata). According to Ansible documentation (https://docs.ansible.com/ansible/guide_gce.html#credentials), this is supported and you need to pass empty strings for both service_account_email and credentials_file, but obviously you cannot open an empty string

laurentgo · 2016-11-19T21:36:07Z

To confirm that internal authorization is also broken:

$ ansible -m gce localhost -a 'service_account_email="" project_id=foo-xxx credentials_file=""'
 [WARNING]: provided hosts list is empty, only localhost is available

localhost | FAILED! => {
    "changed": false, 
    "failed": true, 
    "msg": "Missing GCE connection parameters in libcloud secrets file."
}

(this error is because empty (but None) module arguments are ignored)

if using environment variables instead:

$ GCE_PROJECT='foo' GCE_CREDENTIALS_FILE_PATH='' GCE_EMAIL='' ansible -m gce localhost 
 [WARNING]: provided hosts list is empty, only localhost is available

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IOError: [Errno 2] No such file or directory: ''
localhost | FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_4XyIV6/ansible_module_gce.py\", line 671, in <module>\n    main()\n  File \"/tmp/ansible_4XyIV6/ansible_module_gce.py\", line 579, in main\n    gce = gce_connect(module)\n  File \"/tmp/ansible_4XyIV6/ansible_modlib.zip/ansible/module_utils/gce.py\", line 49, in gce_connect\n  File \"/tmp/ansible_4XyIV6/ansible_modlib.zip/ansible/module_utils/gcp.py\", line 90, in gcp_connect\nIOError: [Errno 2] No such file or directory: ''\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE"
}

GCE internal authorization or installed application authentications stopped working when checking if the credentials_file is actually a JSON file. Skipping over the check if the file doesn't exist, and also fixing module arguments not being used for internal authorization. Fixes ansible#17075

ansibot · 2016-12-19T17:09:34Z

cc @supertom
click here for bot help

GCE internal authorization or installed application authentications stopped working when checking if the credentials_file is actually a JSON file. Skipping over the check if the file doesn't exist, and also fixing module arguments not being used for internal authorization. Fixes ansible#17075

supertom · 2017-03-17T23:53:33Z

Hi Folks,

Regarding Application Default Credentials (the ability to not specify creds when running on a GCE instance), we've recently added support for it in this PR: #22723.

It's currently scheduled for the 2.4 release (unfortunately, we didn't get it finished in time for 2.3).

GCE internal authorization or installed application authentications stopped working when checking if the credentials_file is actually a JSON file. Skipping over the check if the file doesn't exist, and also fixing module arguments not being used for internal authorization. Fixes ansible#17075

ansibot · 2017-06-24T02:43:22Z

cc @erjohnso
click here for bot help

erjohnso · 2017-09-06T15:17:40Z

resolved_by_pr #22723

alikins added bug_report cloud gce labels Aug 15, 2016

alikins added this to the 2.2.0 milestone Aug 15, 2016

alikins added feature_idea and removed bug_report labels Aug 16, 2016

ansibot added the affects_2.1 This issue/PR affects Ansible v2.1 label Sep 8, 2016

laurentgo mentioned this issue Nov 19, 2016

Fix broken authentication for GCE modules (#17075) #18558

Closed

ansibot added bug_report module This issue/PR relates to a module. labels Dec 13, 2016

jimi-c removed the plugin label Jan 4, 2017

gundalow modified the milestone: 2.2.0 Apr 13, 2017

ansibot added the support:community This issue/PR relates to code supported by the Ansible community. label Jun 29, 2017

ansibot closed this as completed Sep 6, 2017

ansibot added feature This issue/PR relates to a feature request. bug This issue/PR relates to a bug. and removed feature_idea labels Mar 2, 2018

ansible locked and limited conversation to collaborators Apr 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gce module errors on file not found when using OAuth client authentication #17075

gce module errors on file not found when using OAuth client authentication #17075

laurentgo commented Aug 13, 2016 •

edited by jctanner

alikins commented Aug 15, 2016

alikins commented Aug 15, 2016

laurentgo commented Aug 15, 2016

walbert947 commented Aug 16, 2016

laurentgo commented Aug 16, 2016

laurentgo commented Nov 19, 2016

laurentgo commented Nov 19, 2016

ansibot commented Dec 19, 2016

supertom commented Mar 17, 2017

ansibot commented Jun 24, 2017

erjohnso commented Sep 6, 2017

gce module errors on file not found when using OAuth client authentication #17075

gce module errors on file not found when using OAuth client authentication #17075

Comments

laurentgo commented Aug 13, 2016 • edited by jctanner

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

alikins commented Aug 15, 2016

alikins commented Aug 15, 2016

laurentgo commented Aug 15, 2016

walbert947 commented Aug 16, 2016

laurentgo commented Aug 16, 2016

laurentgo commented Nov 19, 2016

laurentgo commented Nov 19, 2016

ansibot commented Dec 19, 2016

supertom commented Mar 17, 2017

ansibot commented Jun 24, 2017

erjohnso commented Sep 6, 2017

laurentgo commented Aug 13, 2016 •

edited by jctanner