Extend testing options for functional-test automation

[phenixblue]

What would you like to be added:

Currently there's not a good way to perform per test setup/breakdown for things outside of k8s artifacts.

Why is this needed:

Some tests may require setting up specific scenarios before/after a given functional test. Example:

NodePort test should add an annotation to the target namespace and then remove it when done.

Ideally this is done in a generic way where each resource type can have a setup/breakdown related hook.

[phenixblue]

Some examples depending on the level of granularity:

Per Resource type:

resources:
name: deployments
    pre:
      - pre-script1.sh
      - pre-script2.sh
    post:
      - post-script1.sh
      - post-script2.sh
    tests:
      pass:
        - test-deploy01.yaml
        - test-deploy03.yaml
        - test-deploy04.yaml
        - test-deploy05.yaml
        - test-deploy06.yaml
        - test-deploy07.yaml
        - test-deploy08.yaml
        - test-deploy09.yaml
        - test-deploy10.yaml
        - test-deploy11.yaml
      fail:
        - test-deploy02.yaml

Per Test:

resources:
name: deployments
    tests:
      pass:
        - test-deploy01.yaml
          pre:
            - pre-script1.sh
          post:
            - post-script1.sh
        - test-deploy03.yaml
          pre:
            - pre-script1.sh
          post:
            - post-script1.sh
        - test-deploy04.yaml
      fail:
        - test-deploy02.yaml

[ilrudie]

Accepting something like this would allow you to have a similar functionality to declaring large blocks of pre/post scripts (like having pre/post resource scope scripts but it would also allow you to declare deployment tests that have different pre/post setups as well. In this case you have tests 1-11 having the same setup as with the resource scope declaration but you also have test 12 which could now have an entirely different setup as referenced in pre/post-script3.sh scripts. It comes at a cost of increased verbosity though because instead of having a single declaration area at the resource scope which defines the pre/post scripts for all testing scenarios on that resource (i.e pass only, fail only or all) you now really should ensure that pre/post are declared inside pass and inside fail scopes even if they are the same exact setup scripts being run. This would help with cases where you run pass only on all resources types. If pre/post are not declared and cleaned up inside the pass block then cleanup would not be occurring which could lead to failures of CI because pre run scripts for testing on 1 resource type that do not get cleaned up might interfere with the setup for the next resource type.

Per Test but pre/post not required:

resources:
name: deployments
    tests:
      pass:
        - test-deploy01.yaml
          pre:
            - pre-script1.sh
            - pre-script2.sh
        - test-deploy02.yaml
        - test-deploy03.yaml
        - test-deploy04.yaml
        - test-deploy05.yaml
        - test-deploy06.yaml
        - test-deploy07.yaml
        - test-deploy08.yaml
        - test-deploy09.yaml
        - test-deploy10.yaml
        - test-deploy11.yaml
          post:
            - post-script1.yaml
            - post-script2.yaml
        - test-deploy12.yaml
          pre:
            - pre-script3.sh
          post:
            - post-script3.sh

      fail:
        - test-deploy02.yaml
          pre:
            - pre-script1.sh
            - pre-script2.sh
          post:
            - post-script1.yaml
            - post-script2.yaml

[ilrudie]

In this idea the setup scripts would need to accept a mandatory argument "setup, teardown, reset"

resources:
name: deployments
    tests:
      pass:
        group:
          setup: script1.sh
          manifests:
            - test-deploy01.yaml
            - test-deploy02.yaml
        group:
          setup: script2.sh
          manifests:
            - test-deploy03.yaml
            - test-deploy04.yaml

[ilrudie]

Should this manifest format be versioned so no version or say v1 would be the current behavior and if you specify v2 it would allow this grouping and scripting?

version: v2
resources:
name: deployments
    tests:
      pass:
        group:
          setup: script1.sh
          manifests:
            - test-deploy01.yaml
            - test-deploy02.yaml
        group:
          setup: script2.sh
          manifests:
            - test-deploy03.yaml
            - test-deploy04.yaml

[phenixblue]

I don't think that's necessary. We have a small number of tests right now and should be able to accomplish the transition atomically within a single PR. Do you see a need to keep the old logic around once we've transitioned to the new format.

[ilrudie]

I thought it might be something worth considering but if you see no real benefit to maintaining some support for the old manifest format I agree its simpler and keeps the code base cleaner to just migrate over to the new format all at once.