Kubernetes module

[erjohnso]

Google is going to contribute an ansible module for managing a kubernetes cluster. This module will assume the cluster exists and will require the user to specify endpoint / authorization info and will perform basic CRUD operations against the Kubernetes v1 API[1].

I suspect the best place for this module to live is under clustering/. Please voice objections if an alternate location is preferred.

[1] https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/swagger-spec/v1.json

/cc @gregdek, @jimi-c, @bcoca

[erjohnso]

Hi folks - looking for some input from you on preferred implementation approach to this module. Grab a cup of coffee, this will be lengthy...

For the sake of discussion, let's use Namespaces.

Please take a moment to look at the create Namespace API method on the swagger doc - navigate to api/v1, the use your browser's search and look for create a Namespace (should take you to POST /api/v1/namespaces. Once there, click on the create a Namespace link on the right to expand the method's detailed information. These details indicate what query params and HTTP body are required for the call (hint: it's the v1.Namespace model). As an aside, also take note of the extensive list of API methods in the swagger doc to give you a sense of scope of this effort.

Option 1: Hand-coding

One option is to hand-code API coverage in a set of Ansible modules, likely one for each resource (pods, replica-controllers, services, namespaces, etc). This would allow ansible users to write playbooks that would may contain something like,

- name: Kubernetes finance group apps
  vars:
    api_endpoint: 123.45.67.89
    api_password: redacted
  tasks:
    - name: Create the development namespace for finance group
      kubernetes_namespace:
        name: finance-dev
        labels:
          - development
          - finance

While this may be a good approach for users to map the kubernetes world into the ansible world, I'm concerned that it will be hard to cover the full kubernetes API sufficiently. It would also lead to a tremendous amount of code to write and maintain. Additionally, if a user already has existing kubernetes JSON files, they need to re-write them into the ansible YAML format.

Option 2: Playbooks utilize existing kubernetes JSON files

A second option would be for the ansible modules to just pass pure kubernetes JSON files to the kubernetes API and return the results. To illustrate this approach, let's assume we already have the kubernetes JSON file, namespace-dev.json, from the links above (copied here for convenience),

{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "development",
    "labels": {
      "name": "development"
    }
  }
}

The corresponding ansible YAML for this could look something like,

- name: Kubernetes finance group apps
  vars:
    api_endpoint: 123.45.67.89
    api_password: redacted
  tasks:
    - name: Create the development namespace for finance group
      kubernetes:
        kubernetes_file: /path/to/namespace-dev.json
        state: present

Now, the module merely reads the contents of the file, creates the POST /api/v1/namespaces request and passes the file contents in as the HTTP body. The response is already in JSON, so we just pass the response (probably) as-is to the user.

If the user wants to delete this namespace, for convenience, they should be able to reference the same file. The module will have need to be able to map a Kind from the kubernetes JSON file into the URL path. But, I think this is pretty easily done since I can dynamically read in the full API spec and introspect this mapping without having to hard-code anything other than the v1-spec file.

To handle idempotency, the module will need to know that POST, PUTresponses of 200 mean it was added (e.g. changed=True) or 409 (e.g. changed=False). Likewise on DELETE, but with 200 and 404 respectively.

My preference: Option 2

I'm leaning much more toward Option 2, but would like other's input before doing more tangible work. As I see it, the benefits are:

• Less hand-written code implies fewer bugs
• Users can take existing kubernetes JSON files and start using them out-of-the-box with ansible
• The above also allows them to continue using kubectl outside of ansible if desired
• Maintenance burden will be lower since the module dynamically parses the v1-spec and then "knows" how to interact with the API
• Also, less code means less maintenance
• Likely mostly future proof (could use a module variable called api_version) and as long as that matches the swagger URL, all should be good.

Thoughts?

[jimi-c]

My main concern with option #2 is that it removes the ability to specify variables for values in the JSON file. I know or cloudformations module can accept a list of template parameters, but this is more a built-in function of cloudformations rather than our module code. Reading the docs, I'm not seeing a comparable option, unless I missed it?

[erjohnso]

Option 1 it is. Shortly after posting this, I realized that we'd sacrifice variables with Option 2 and I can't find an elegant way of dealing with that.

The kubectl command also take YAML format. At least a few examples use YAML vs JSON, so I'll try to make the ansible YAML match kubernetes YAML syntax as much as I can. I suspect I'll run into a few gotchas, but I hope it'll generally be pretty sane.

[jimi-c]

@erjohnso yeah, the only other option I see is to support both ways. Not sure how painful that'd be, but since simply specifying an existing JSON file is pretty painless and easy it might not be too much more work.

Thoughts?

[brendandburns]

We actually prefer YAML to JSON in our examples. JSON is for machines, YAML is for humans is our general rule of thumb.

[bcoca]

a note that says 'since JSON is a subset of YAML, you can also use it if you must'

[erjohnso]

Ok, I think I'm on a good path and thanks everyone for the input. I have a working stub for creating Namespaces and it maps very cleanly to the v1.Namespace definition, so the end user won't have to do a lot of extra conversion.

For reference, here's an example of the Kubernetes v1.Namespace definition in YAML:

kind: Namespace
apiVersion: v1
metadata:
  name: my-namespace
  labels:
    env: production
    version: stable
  annotations:
    key1: value1
    key2: value2
spec:
  finalizers: []

Here's what it looks like in an ansible playbook:

---
# Create a k8s app
- hosts: localhost
  vars:
    api_endpoint: 123.45.67.89
    url_username: admin
    url_password: redacted
  tasks:
  - name: Create a new k8s namespace
    kubernetes_namespace:
        api_endpoint: "{{ api_endpoint }}"
        url_username: "{{ url_username }}"
        url_password: "{{ url_password }}"
        metadata:
          name: my-namespace
          labels:
            env: production
            version: stable
          annotations:
            a1: value1
            a2: value2
        state: present

I'm essentially handing back the JSON response as-is, and the snippet of output from ansible-playbook -vvv looks like:

No config file found; using defaults
1 plays in namespace.yml

PLAY ***************************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [Create a new k8s namespace] **********************************************
changed: [localhost] => {"apiVersion": "v1", "api_endpoint": "https://104.154.46.136/api/v1", "api_url": "https://104.154.46.136/api/v1/namespaces", "changed": true, "kind": "Namespace", "metadata": {"annotations": {"a1": "value1", "a2": "value2"}, "creationTimestamp": "2015-10-27T22:19:46Z", "labels": {"label_env": "production", "label_ver": "latest"}, "name": "ansible-test", "resourceVersion": "114451", "selfLink": "/api/v1/namespaces/ansible-test", "uid": "d4970969-7cf8-11e5-a5bc-42010af00144"}, "spec": {"finalizers": ["kubernetes"]}, "status": {"phase": "Active"}}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0   

Everyone cool with this?

I'm happy with it. 🍶

[erjohnso]

For anyone following along, here's what I have so far[1]. It's a bit different than what was last proposed, but I think overall, this is much cleaner.

It supports in-line YAML and users can effectively copy/paste the Kubernetes YAML examples into their playbooks. With this approach, they can modify the in-line YAML to allow for ansible's variable substitution.

It also supports re-using the YAML files on disk and the user's playbooks can just point to existing Kubernetes YAML files. But this comes at the cost of not allowing variable substitution.

Lastly, I've started stubbing out some fake kubectl commands. I've started with the easy ones like kubectl get [pods, replicationcontrollers, services, etc], but I think this could be extended to support some of the convenience functions like kubectl stop or kubectl scale.

TODO:

• Get SSL certificates working. Currently, since I'm using the fetch_url() method from ansible.module_utils.urls, I'm just passing in validate_certs: false during development.
• Remove debug logging

[1] https://github.com/erjohnso/ansible-modules-extras/blob/google-kubernetes/clustering/kubernetes.py

[erjohnso]

Closing this issue out now that there is a PR for the module: #1229

[eparis]

Sadly I just found this. I would have suggestted option #2 but make the json/yaml a jinja template file instead of a static file... Ok, off to see the actual code.