add plugin support for goss

[weakiwi]

Describe the feature:

Sometimes, we want to use some 3rd tools (like ethotool and demicode) to do some test. On current stage we can use Command to test. But I think it's better to implement plugins so that user can define their own test method.

Describe the solution you'd like

1. a plugin (in this situtation is binary file) should follow rules as below
   1. ./plugin name will output plugin's name
   2. ./plugin options will output a list with plugin's options for goss to call. Like a ethtool plugin should output speed, is-bond, is-virtual etc
   3. ./plugin test-method test-target will output the test result. E.g: ethtool plugin, ./ethotool_plugin.sh is-virtual etho0 will output true
2. the plugin pass should be passed via cli args or config file, like: ./goss --plugins /usr/local/bin/ethotool_plugins then when goss start, it will create struct via ./plugin name and ./plugin options output.

Describe alternatives you've considered

For the implement of plugins, I have an idea. I use python to explain the code logic. We can talk about the implement and then I can create a new PR for this.

plugin:
  /tmp/test.sh:
      eth0:
        is-virtual: false
        is-bond: false

import yaml
import pprint
import json
import get_command_output
with open("test.yml") as f:
        data = yaml.load(f)

def get_options(executable_file):
        return get_command_output("%s options" % executable_file).strip().split(",")

def get_output_by_option(executable_file, option, target):
        return get_command_output("%s %s %s" % (executable_file, option, target)).strip()

for executable_file in  data["plugin"]:
        for execute_target in data["plugin"][executable_file]:
                for k,v in data["plugin"][executable_file][execute_target].items():
                        print "k=%s, v=%s, result=%s" % (k,v,json.loads(get_output_by_option(executable_file, k, execute_target)))
                        print(get_output_by_option(executable_file, k, execute_target) == v)

check_virtual() {
        echo "false"
}

check_bond() {
        echo "false"
}

case $1 in
        is-virtual)
                check_virtual $2
                exit 0
                ;;
        is-bond)
                check_bond $3
                exit 0
                ;;
        options)
                echo "is-virtual, is-bond"
                exit 0
                ;;
        name)
                echo "ethtool-plugin"
                exit 0
                ;;
        *)
                echo "Usage: $0 {options|is-virtual|is-bond}"
                exit 0
                ;;
esac

[aelsabbahy]

Adding plugin support would be really beneficial to goss. So when you say it's passed in through --plugin would the goss test look as follows:

goss validate --plugin ./path/to/plugin

And the yaml (using your script for example):

plugin:
  ethtool-plugin:
    eth0:
      is-virtual: true
      is-bond: true

[weakiwi]

Yes. What you describe is better.

[aelsabbahy]

Ok, so here's some thoughts I have on this.

1. The nesting would need to change.

This doesn't match the pattern of the other tests and would need to change to be consistent (for merging and other reasons).
From:

plugin:
  ethtool-plugin:
    eth0:
      is-virtual: true

To:

plugin:
  eth0:
    plugin_type: ethtool-plugin
    is-virtual: true

2. The command is going to have some enhancements in the near future that will make it possibly overlap with this.

• Feature Request: Improve gomega matchers #538 Improving gomega matchers
• (yet to be created ticket, but here's a comment discussing it) - Jmespath gomega test

With those two features it will be possible to do the following:

command:
  eth0:
   # using echo as an example, this would need to be a script that dumps info in json format:
   # ex: ./ethotool_plugin.sh json-dump eth0
    exec: |
      echo '{"is-bound": true, "is-virtual": false, "info": {"mac": "xxx"}}'
    exit-status: 0
    stdout:
        and:
          - jmespath: is-bound
            matches: true
          - jmespath: is-virtual
            matches: true
          - jmespath: info.mac
            matches: xxx
    timeout: 10000

I would love to get the community feedback on 1 and 2.. Some high level differences I can think of:
Pros/cons of 1:
Pros:

• easier to implement for the end-user
• only runs the tests needed (e.x. ./ethotool_plugin.sh is-virtual)

Cons:

• Exec per tested attribute. So if you have 5 tests with is-virutal, is-bound and mac-id. That's 15 calls to exec (more if you include the call to "name" and "options"

Pros/cons of 2:
Pros:

• Enhances existing test. Everything is still command
• Test can print results in json, so executes only once
• ./ethotool_plugin.sh is-virtual can still be used to be more granular but it would require multiple "command" tests.
• Possible to support nested structures (e.g. info.mac above)

Cons:

• Less intuitive to learn initially (and/jmespath pattern)
• Harder to write shell scripts that print out json (not an issue with most other languages)

Would love your thoughts on this. Same with @pedroMMM, @donmstewart and anyone else interested in plugin support

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[aelsabbahy]

#576 adds a gjson matcher, this will allow option 2 above, is this sufficient?