New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GCE Dynamic Inventory script using Google API Client library #24505

Open
wants to merge 43 commits into
base: devel
from

Conversation

Projects
None yet
10 participants
@maurofaccenda

maurofaccenda commented May 11, 2017

SUMMARY

We needed to use the beta version of the API (to be able to use labels) and figured out that Apache's libcloud (used by the current gce.py) only uses 'v1' (it is hardcoded).

So, a new script that uses Google API Client library was created from scratch. It was inspired by some of the current gce.py features, however not 100% of the features are there yet.

ISSUE TYPE
  • New Module Pull Request
COMPONENT NAME

contrib/inventory/gce_googleapiclient.py

ANSIBLE VERSION
ansible 2.3.0.0
  config file =
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, Dec 18 2016, 07:03:39) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
ADDITIONAL INFORMATION
@ansibot

This comment has been minimized.

Contributor

ansibot commented May 11, 2017

@maurofaccenda this PR contains the following merge comits:

Please rebase your branch to remove these commits.

click here for bot help

@nitzmahone

This comment has been minimized.

Member

nitzmahone commented May 11, 2017

cc @supertom - thoughts?

@supertom

This comment has been minimized.

Contributor

supertom commented May 17, 2017

@maurofaccenda thanks for the PR! We like the idea of being able to specify different API versions as well, and we've been asked about label support, so your timing is good.

I haven't tried this yet, but I see that you mention that it doesn't have feature-parity with the existing gce.py. Is that something you're willing/able to add? More specifically,

  • We'd like it to use the authentication functions that we've added in module_utis/gcp.py
  • It needs to produce the same output as the existing gce.py (or at least, compatible, we could discuss)
  • Needs to support caching (which could probably be a lift-and-shift from the existing gce.py)
@maurofaccenda

This comment has been minimized.

maurofaccenda commented May 18, 2017

@supertom sure. I am willing to do that. However as I am doing that as part of my work (the customer asked explicitly to send the code upstream), it would be nice to have:

  • list of changes would be the bare minimum to get it accepted upstream
  • 'wish list' for some extra changes to do them afterwards.

With that, I can create the tasks on our internal issue tracker and they can prioritise the tasks accordingly with their needs.

Keep in mind that I am not Python 'fluent', so some input/help would be great.

From what you mentioned, the idea is to have it 100% (or as much as possible) compatible at some point.

@ansibot ansibot added stale_ci and removed stale_ci labels Jun 23, 2017

@ansibot

This comment has been minimized.

Contributor

ansibot commented Jun 27, 2017

The test ansible-test sanity --test pep8 failed with the following errors:

contrib/inventory/gce_googleapiclient.py:48:1: E303 too many blank lines (3)
contrib/inventory/gce_googleapiclient.py:53:46: W291 trailing whitespace
contrib/inventory/gce_googleapiclient.py:72:1: E302 expected 2 blank lines, found 1
contrib/inventory/gce_googleapiclient.py:170:31: E128 continuation line under-indented for visual indent
contrib/inventory/gce_googleapiclient.py:171:31: E128 continuation line under-indented for visual indent

click here for bot help

@ansibot

This comment has been minimized.

Contributor

ansibot commented Jun 28, 2017

The test ansible-test sanity --test pep8 failed with the following errors:

contrib/inventory/gce_googleapiclient.py:43:117: W291 trailing whitespace
contrib/inventory/gce_googleapiclient.py:44:87: W291 trailing whitespace

click here for bot help

@ansibot

This comment has been minimized.

Contributor

ansibot commented Jun 28, 2017

The test ansible-test sanity --test pep8 failed with the following errors:

contrib/inventory/gce_googleapiclient.py:43:117: W291 trailing whitespace
contrib/inventory/gce_googleapiclient.py:44:87: W291 trailing whitespace

click here for bot help

@ansibot ansibot removed the ci_verified label Jun 28, 2017

@ansibot

This comment has been minimized.

Contributor

ansibot commented Jun 28, 2017

@ansibot ansibot added support:core and removed needs_revision labels Jun 28, 2017

@ansibot

This comment has been minimized.

Contributor

ansibot commented Jun 30, 2017

The test ansible-test compile --python 2.6 failed with the following error:

contrib/inventory/gce_googleapiclient.py:145:15: SyntaxError: for md in instance['metadata']['items']

click here for bot help

Mauro Faccenda and others added some commits Feb 22, 2018

Fix KeyError 'sourceImage' from instances without sourceImage (#19)
Some instance on GCE might have been created from a disk snapshot. Unlike AWS, which oblige all instances to start from an AMI, GCE allows root disk from snapshots. Unfortunately, with the snapshot, we lose the source image.

To keep the gce.py retro-compactibility the instances without a source image are set in a group named 'persistent_disk'
Fix 404 on project and cache issues (#20)
* Fix ghost 404 project issue

Even if a project has been shut down, it will still be returned by the billing API for 30 days. The issue is that the project won't be returned by the compute API and produce a 404 when the zones are retrieved. So we should check if the project is active before adding it to the list.

* Fix OSError on mkdir cache when the zones are manually set

When the zones are manually configured via the command line or the config file, the script fails:

```
Traceback (most recent call last):
  File "./gce_googleapiclient.py", line 625, in <module>
    main(ARGS)
  File "./gce_googleapiclient.py", line 606, in main
    project_zone_list).get(timeout):
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 572, in get
    raise self._value
OSError: [Errno 2] No such file or directory: '.gce_cache/kube-test-eu-43-proj-xxxxxxx/europe-west1-b'
```

The list of zone per project is not cached in the `.gce_cache/project_name/zones.json` file if the zone is given to the script. So when the dir for the zone is created [here](https://github.com/KohlsTechnology/ansible/blob/29a545aa7c95274a2f348480278f143cfa72842c/contrib/inventory/gce_googleapiclient.py#L537) it fails on `.gce_cache/project_name/zone_name`, because the folder `.gce_cache/project_name` doesn't exist.

Let's do a `mkdir -p` to ensure the directory is created in any case.
Fix cache '~' path issue (#21)
By default, Python doesn't translate the '~' as the $HOME directory. We should use os.path.expanduser function to avoid this situation:

```bash
$ pwd
/home/erouan/Documents/ansible
$ contrib/inventory/gce_googleapiclient.py --cache-dir '~/.gce_cache' > /dev/null
$ ls -hla
total 484K
drwxrwxr-x.  3 erouan erouan 4.0K Mar 14 14:56 '~' # <----- Wrong place
drwxrwxr-x. 17 erouan erouan 4.0K Mar 14 14:56  .
drwxr-xr-x. 22 erouan erouan 4.0K Mar  8 16:40  ..
-rw-rw-r--.  1 erouan erouan  77K Jul 20  2017  ansible-core-sitemap.xml
-rw-rw-r--.  1 erouan erouan  317 Feb 16 13:49  ansible.iml
...
```
fix null when no public ip is available on instance and exception whe…
…n the instance disk doesnt come from another image
Change the GCE metadata format to be compatible with the previous gce…
….py (#7)

* update gce_metadata to be compatible with gce.py

* removed unecessary comment
Fix envvars and ini input parameters (#14)
* update gce_metadata to be compatible with gce.py

* removed unecessary comment

* fix syntax for python 2.6

* merged with upstream

* fix syntax for python 2.6

* improving code readability

* add gce_name on hostvars

* fixed the way the script deals with .ini config file and environment variables to get input paramenters, updated the documentation to reflect these changes

* updated comments with minor changes

* updated comments with some other minor changes

* minor changes

* minor changes on usage
@davidebelloni

This comment has been minimized.

davidebelloni commented May 7, 2018

Hi @maurofaccenda ,
thank you for the pointer!
Usually I don't use neither inventory scripts nor inventory plugins

And thanks for the fixes ;)

@ansibot ansibot added the stale_ci label May 15, 2018

@davidebelloni

This comment has been minimized.

davidebelloni commented May 17, 2018

What about add service account info?

Something like this:

'gce_svc': instance['serviceAccounts'][0]['email']

@erjohnso

This comment has been minimized.

Contributor

erjohnso commented May 31, 2018

👍 This LGTM

/cc @ryansb - think we could merge this, or are we in code freeze for 2.6 now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment