Skip to content

Polyglot Plugin Patterns

Jump to bottom
DeWayne Filppi edited this page Aug 21, 2018 · 13 revisions

For a long time now, I've told those that were interested that plugins for Cloudify don't necessarily have to be written in Python, and then waved my hands about how non-Python plugins could be created. Recently, I was asked from a Go shop whether plugins could be written in Go, and I did my usual assurances and hand waving. However, this time, I actually went and wrote a Go plugin. That was so satisfying that I followed it up with a Java plugin. This post explores those example plugins. This post is meant mostly for a plugin developer audience. If you aren't a developer, just retain that you don't have to write plugins in Python.

Cloudify Plugins Overview

Plugins in Cloudify are the means of adding functionality to blueprints by packaging Cloudify DSL type definitions and associated code. These plugins are ultimately called by Cloudify as Python modules. To be clear, a Cloudify plugin is a package consisting of a Python package and a definitions file ( plugin.yaml ) that links the code symbolically to types you might use in a blueprint.

Plugins define operations (among other things), which are just Python functions that get tied to life cycle methods in blueprints. What those functions do is up to the plugin author. One of the things an author might do in a plugin is call another language. Actually plugins do that all the time via REST APIs. All that is needed for a Go plugin to function is package it correctly in a Python package, associate relevant Go code, and throw in a little glue code.

The Script Plugin

Consider the Cloudify script plugin. It allows you to run arbitrary shell scripts (or any executable) as type operations. It does this by 'shelling out' in Python to the requested intepreter and running the script code. One interesting feature of the script plugin is the context proxy. This is a way of providing access inside scripts to orchestration context information like the current nodes runtime properties. This is absolutely a feature needed in a Go (or other language) plugin. I make use of the context proxy in my plugin examples.

The Go Plugin 'Template'

Overview

Relative to the "big three" ( Javascript, Java, and Python), Go is a niche language. Unlike those other languages, it doesn't use an interpreter (or GIT compiler ) to produce runnable code. It is 'C', the sequel, and like 'C' requires compilation and linking steps to produce actual runnable platform specific executables. This is significant when considering a Go plugin, as plugins are ideally portable packages.

The example Go plugin is meant to serve as a starting point for developing Go plugins. It has certain conventions and assumptions built in that hopefully aren't too limiting. As a Go programmer, you can largely ignore the Python in the project, as it only exists to launch your Go code. You may even be able to ignore the project structure somewhat, but you can't change its' organization, or the Python packaging mechanism upon which everything relies will be broken.

Organization

Python packaging is picky about the setup.py file which drives the packaging. As a Golang plugin author, you need to ignore pretty much everything except the metadata attributes, which are self explanatory:

    name='golang-test-plugin',
    version='0.0.1',
    author='dfilppi',
    author_email='dfilppi@gmail.com',
    description='example golang plugin for cloudify',
    license='LICENSE'
...

The other entries in the file are related to the packaging of the plugin Go code. You should change these entries for your own purposes. The actual meat of your plugin lives under golang_adapter/src/plugin by convention.

Plugin Authoring Nano-Framework for Go

You may have noticed the go.py file under in the golang_adapter directory. This is a little Python shim that wraps the Go code that constitutes the plugin. You normally don't want to touch this. Let's return to the golang_adapter/src/plugin directory. Here you find 3 files:

  • operations.go - The actual Go functions that will implement your plugin
  • plugin.go - The main entry point and operation caller.
  • util.go - Not meant to be modified. Currently just provides the deployment proxy client to Go.

The basic operation of runtime of a Go plugin is simple:

  • go.py is called by Cloudify based on the configuration in plugin.yaml. More on that later.
  • go.py is passed a function name
    • On the first call to the plugin, the Go code is built
    • The context proxy process is started. More on that later.
    • The function name is passed to the Go executable as a parameters, along with function parm.
  • plugin.go maps the function name and arguments to a function in the operations.go file.

The plugin.yaml file in the plugin root directory is a file that only has meaning to Cloudify. It is included in Cloudify blueprints, and typically provides types, relationships, and workflow declarations for use with the plugin code. An aspect of these definitions is the mapping from symbolic YAML names to plugin code. When you create your own plugin, you will provide you own plugin.yaml containing whatever types you want to provide; the contents provided are just for the example. Look at the example type defined in plugin.yaml:

  golang.Test:
    derived_from: cloudify.nodes.Root
    properties:
      prop1:
        type: string
      prop2:
        type: integer
    interfaces:
      cloudify.interfaces.lifecycle:
        create:
          implementation: goplugin.golang_adapter.go.callgo
          inputs:
            func:
              description: the go function to call
              type: string
            args:
              description: arg dict to pass
              default: {}

Before digging into this, you should get a basic overview of Cloudify DSL type definitions in the wiki. You should also visit the workflows and plugins areas.

The plugin.yaml defines a single type called golang.Test that has a few properties and a single mapping to a Go defined operation. Note the create lifecycle method is mapped to the goplugin.golang_adapter.go.callgo Python defined operation (in go.py). As a Go plugin author, you will always have this same mapping. The actual Go code is linked by the func input, and the valid args to the Go function are provided by the args input. In a real type, you might provide defaults for both of these, so blueprint authors wouldn't need to see it. In this example, I've left it explicit. A very important part of plugin.yaml is the plugin definition lines at the top of the file. Read and understand their use here.

Now lets switch focus to the actual example blueprint in the examples/local-blueprint.yaml file. Here we see our type being consumed:

  gotest:
    type: golang.Test
    properties:
      prop1: prop1_val
      prop2: 9
    interfaces:
      cloudify.interfaces.lifecycle:
        create:
          inputs:
            func: func1
            args:
              arg1: val1

Note the inputs to the create lifecycle method are telling the plugin to call func1 in the plugin with the supplied args. Again, if in your case it is possible to have defaults for both of these, the blueprint can eliminate the whole interfaces section. Of course, I've just demonstrated the mapping of the create lifecycle method. You can map any or all of the others. The first one that is actually called will trigger the platform build of your Go code on the target server (if it runs remotely).

The Context Proxy Client for Go

In the util.go file is a Go implementation of the context proxy client. When your Go plugin code is executed, the context proxy is started and fields requests from the client. The client and proxy form a bridge between Go and Python for the purpose of allowing your plugin to retrieve properties values from the blueprint, fetch and store runtime properties in the deployment, and log messages. See the operations.go file for usage examples.

Error Handling

Plugins communicate their status to Cloudify by raising exceptions; NonRecoverableError for errors that retrying will not fix, and RecoverableError for errors that might be fixed via retry. The Go plugin communicates these by it's exit status: 0 for success, 1 for RecoverableError, and 2 for NonRecoverableError.

Trying it out

To actually run the example, can follow these steps (on Linux):

  1. git clone -b post1 http://github.com/dfilppi/goplugin
  2. Install the Cloudify CLI.
  3. cd to the examples directory
  4. cfy init local-blueprint.yaml --install-plugins
  5. cfy ex st install -b examples

The install will call the func1 example operation to be called. It uses the context proxy to fetch properties and attributes and log some messages.

Conclusion

That's a lot of writing for something that isn't really all that complicated. By reusing the goplugin example as a starting point, you can develop your own Cloudify plugins without writing Python. I'll follow up with another, hopefully shorter post, describing the Java plugin example, which uses the same basic ideas, but is different enough to warrant it's own discussion. If you missed it above, the code is here. Comments welcome.

Clone this wiki locally