Skip to content
🐍 Manage and run your integration tests with efficiency - Venom run executors (script, HTTP Request, web, imap, etc... ) and assertions
Branch: master
Clone or download
matoous and yesnault fix(executor/grpc): Fix documentation and configuration (#191)
Fix documentation for gRPC executor missing some available
configuration and also accidentally copied text from README of other
executor.

Fix decoding names of configuration fields.

Signed-off-by: Matous Dzivjak <matous.dzivjak@kiwi.com>
Latest commit 670c0b7 May 3, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cli feat(executor/grpc): Add GRPC executor (#187) (#188) May 3, 2019
context feat: Add redis executor (#141) Apr 4, 2018
executors fix(executor/grpc): Fix documentation and configuration (#191) May 3, 2019
lib feat(executor/grpc): Add GRPC executor (#187) (#188) May 3, 2019
tests feat(executor/grpc): Add GRPC executor (#187) (#188) May 3, 2019
.gitignore feat : add --profiling arg Aug 6, 2018
CONTRIBUTING.md doc: fix typos and misspellings (#172) Oct 15, 2018
Gopkg.lock chore: delete vendor dir May 3, 2019
Gopkg.toml feat(executor/grpc): Add GRPC executor (#187) (#188) May 3, 2019
LICENSE Initial Revision Feb 17, 2017
README.md feat(executor/grpc): Add GRPC executor (#187) (#188) May 3, 2019
assertion.go feat: dump result if failure and outputdir != "" Aug 6, 2018
assertion_test.go feat (assertion): support quoted args when splitting assertion. (#87) Oct 13, 2017
extract.go refactor: remove unused logger in func Aug 6, 2018
extract_test.go fix: remove noPrintable char from Failure & Output (#88) Oct 13, 2017
package.sh chore: update go-dump to 1.0.8 (#182) Feb 28, 2019
process.go fix: fail if errors happened (#176) Nov 21, 2018
process_files.go fix(venom): re-templating vars (#161) Aug 8, 2018
process_files_test.go
process_testcase.go fix(venom): take extracted ts.vars only once (#166) Aug 21, 2018
process_teststep.go fix: store failure on last retry only (#181) Feb 12, 2019
process_testsuite.go fix: apply templater on vars (#164) Aug 16, 2018
templater.go fix(venom): re-templating vars (#161) Aug 8, 2018
types.go feat: dump result if failure and outputdir != "" Aug 6, 2018
venom.gif Initial Revision Feb 17, 2017
venom.go feat: improve output display Aug 6, 2018
venom_output.go chore: update go-dump to 1.0.8 (#182) Feb 28, 2019

README.md

🐍 Venom

Venom run executors (script, HTTP Request, etc. ) and assertions. It can also output xUnit results files.

Venom Demonstration

Command Line

Install with:

$ go get github.com/ovh/venom
$ go install github.com/ovh/venom/cli/venom
$ venom run -h
Run Tests

Usage:
  venom run [flags]

Flags:
      --env                    Inject environment variables. export FOO=BAR -> you can use {{.FOO}} in your tests (default true)
      --exclude strings        --exclude filaA.yaml --exclude filaB.yaml --exclude fileC*.yaml
      --format string          --format:yaml, json, xml, tap (default "xml")
  -h, --help                   help for run
      --log string             Log Level : debug, info or warn (default "warn")
      --no-check-variables     Don't check variables before run
      --output-dir string      Output Directory: create tests results file inside this directory
      --parallel int           --parallel=2 : launches 2 Test Suites in parallel (default 1)
      --profiling              Enable Mem / CPU Profile with pprof
      --stop-on-failure        Stop running Test Suite on first Test Case failure
      --strict                 Exit with an error code if one test fails
      --var strings            --var cds='cds -f config.json' --var cds2='cds -f config.json'
      --var-from-file string   --var-from-file filename.yaml : hcl|json|yaml, must contains map[string]string'

Executors

TestSuite files

Example:

name: Title of TestSuite
testcases:
- name: TestCase with default value, exec cmd. Check if exit code != 1
  steps:
  - script: echo 'foo'
    type: exec

- name: Title of First TestCase
  steps:
  - script: echo 'foo'
    assertions:
    - result.code ShouldEqual 0
  - script: echo 'bar'
    assertions:
    - result.systemout ShouldNotContainSubstring foo
    - result.timeseconds ShouldBeLessThan 1

- name: GET http testcase, with 5 seconds timeout
  steps:
  - type: http
    method: GET
    url: https://eu.api.ovh.com/1.0/
    timeout: 5
    assertions:
    - result.body ShouldContainSubstring /dedicated/server
    - result.body ShouldContainSubstring /ipLoadbalancing
    - result.statuscode ShouldEqual 200
    - result.timeseconds ShouldBeLessThan 1

- name: Test with retries and delay in seconds between each try
  steps:
  - type: http
    method: GET
    url: https://eu.api.ovh.com/1.0/
    retry: 3
    delay: 2
    assertions:
    - result.statuscode ShouldEqual 200

Using variables and reuse results

name: MyTestSuiteTmpl
vars:
  api.foo: 'http://api/foo'
  second: 'venomWithTmpl'

testcases:
- name: testA
  steps:
  - type: exec
    script: echo '{{.api.foo}}'
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldEqual http://api/foo

- name: testB
  steps:
  - type: exec
    script: echo 'XXX{{.testA.result.systemout}}YYY'
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldEqual XXXhttp://api/fooYYY

Extract variable from results and reuse it in step after

name: MyTestSuite
testcases:
- name: testA
  steps:
  - type: exec
    script: echo 'foo with a bar here'
    extracts:
      result.systemout: foo with a {{myvariable=[a-z]+}} here

- name: testB
  steps:
  - type: exec
    script: echo {{.testA.myvariable}}
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldContainSubstring bar

Builtin venom variables

name: MyTestSuite
testcases:
- name: testA
  steps:
  - type: exec
    script: echo '{{.venom.testsuite}} {{.venom.testsuite.filename}} {{.venom.testcase}} {{.venom.teststep.number}} {{.venom.datetime}} {{.venom.timestamp}}'
    # will display something as: MyTestSuite MyTestSuiteWithVenomBuiltinVar.yml testA 0 2018-08-05T21:38:24+02:00 1533497904

Builtin variables:

  • {{.venom.testsuite}}
  • {{.venom.testsuite.filename}}
  • {{.venom.testcase}}
  • {{.venom.teststep.number}}
  • {{.venom.datetime}}
  • {{.venom.timestamp}}

Testsuite Versions

Version 2

On this new version, venom use the testsuite folder as the basepath instead of location of venom execution.

Considering this workspace:

tests/
   testsuiteA/
      testsuite.yml
      testa.json

On version 1

name: TestSuite Read File
testcases:
- name: TestCase Read File
  steps:
  - type: readfile
    path: testa.json
    assertions:
      - result.contentjson.foo ShouldEqual bar

If you execute venom run * into tests/ folder, venom will try to find testa.json on tests/testa.json and will failed. You must execute venom on the testsuite dir.

On version 2, venom use as basepath the testsuite file. So no matter where you execute venom command, testa.json will be found.

To specify the version 2, add version property on the testsuite:

version: "2"
name: TestSuite Read File
testcases:
- name: TestCase Read File
  steps:
  - type: readfile
    path: testa.json
    assertions:
      - result.contentjson.foo ShouldEqual bar

RUN Venom locally on CDS Integration Tests

cd $GOPATH/src/github.com/ovh/cds/tests
venom run --var cdsro='cds -f $HOME/.cds/it.user.ro.json' --var cds='cds -f $HOME/.cds/it.user.rw.json' --parallel=5

RUN Venom with file var

vars.yaml :

cdsro: 'cds -f $HOME/.cds/it.user.ro.json'
cds: 'cds -f $HOME/.cds/it.user.rw.json'
cd $GOPATH/src/github.com/ovh/cds/tests
venom run --var-from-file vars.yaml --parallel=5

RUN Venom, with an export xUnit

venom run --format=xml --output-dir="."

Assertion

Keywords

  • ShouldEqual
  • ShouldNotEqual
  • ShouldAlmostEqual
  • ShouldNotAlmostEqual
  • ShouldResemble
  • ShouldNotResemble
  • ShouldPointTo
  • ShouldNotPointTo
  • ShouldBeNil
  • ShouldNotBeNil
  • ShouldBeTrue
  • ShouldBeFalse
  • ShouldBeZeroValue
  • ShouldBeGreaterThan
  • ShouldBeGreaterThanOrEqualTo
  • ShouldBeLessThan
  • ShouldBeLessThanOrEqualTo
  • ShouldBeBetween
  • ShouldNotBeBetween
  • ShouldBeBetweenOrEqual
  • ShouldNotBeBetweenOrEqual
  • ShouldContain
  • ShouldNotContain
  • ShouldContainKey
  • ShouldNotContainKey
  • ShouldBeIn
  • ShouldNotBeIn
  • ShouldBeEmpty
  • ShouldNotBeEmpty
  • ShouldHaveLength
  • ShouldStartWith
  • ShouldNotStartWith
  • ShouldEndWith
  • ShouldNotEndWith
  • ShouldBeBlank
  • ShouldNotBeBlank
  • ShouldContainSubstring
  • ShouldNotContainSubstring
  • ShouldEqualWithout
  • ShouldEqualTrimSpace
  • ShouldHappenBefore
  • ShouldHappenOnOrBefore
  • ShouldHappenAfter
  • ShouldHappenOnOrAfter
  • ShouldHappenBetween
  • ShouldHappenOnOrBetween
  • ShouldNotHappenOnOrBetween
  • ShouldHappenWithin
  • ShouldNotHappenWithin
  • ShouldBeChronological

Most assertion keywords documentation can be found on smartystreets/assertions README.

Write your executor

An executor have to implement this interface

// Executor execute a testStep.
type Executor interface {
	// Run run a Test Step
	Run(ctx context.Content, venom.Logger, TestStep) (ExecutorResult, error)
}

Example

// Name of executor
const Name = "myexecutor"

// New returns a new Executor
func New() venom.Executor {
	return &Executor{}
}

// Executor struct
type Executor struct {
	Command string `json:"command,omitempty" yaml:"command,omitempty"`
}

// Result represents a step result
type Result struct {
	Code        int    `json:"code,omitempty" yaml:"code,omitempty"`
	Command     string `json:"command,omitempty" yaml:"command,omitempty"`
	Systemout   string   `json:"systemout,omitempty" yaml:"systemout,omitempty"` // put in testcase.Systemout by venom if present
	Systemerr   string   `json:"systemerr,omitempty" yaml:"systemerr,omitempty"` // put in testcase.Systemerr by venom if present
	Executor    Executor `json:"executor,omitempty" yaml:"executor,omitempty"`
}

// GetDefaultAssertions return default assertions for this executor
// Optional
func (Executor) GetDefaultAssertions() venom.StepAssertions {
	return venom.StepAssertions{Assertions: []string{"result.code ShouldEqual 0"}}
}

// Run execute TestStep
func (Executor) Run(ctx context.Context, l venom.Logger, step venom.TestStep) (venom.ExecutorResult, error) {

	// transform step to Executor Instance
	var e Executor
	if err := mapstructure.Decode(step, &e); err != nil {
		return nil, err
	}

	// to something with e.Command here...
	//...

	systemout := "foo"
	ouputCode := 0

	// prepare result
	r := Result{
		Code:    ouputCode, // return Output Code
		Command: e.Command, // return Command executed
		Systemout: systemout, // return Output string
		Executor: e, // return executor, useful for display Executor context in failure
	}

	return dump.ToMap(r)
}

Feel free to open a Pull Request with your executors.

TestCase Context

TestCase Context allows you to inject data in all steps.

Define a context is optional, but can be useful to keep data between test steps on a testcase.

Write your TestCase Context

A TestCase Context has to implement this interface

type TestCaseContext interface {
	Init() error
	Close() error
	SetTestCase(tc TestCase)
	GetName() string
}

Example

// Context Type name
const Name = "default"

// New returns a new TestCaseContext
func New() venom.TestCaseContext {
	ctx := &DefaultTestCaseContext{}
	ctx.Name = Name
	return ctx
}

// DefaultTestCaseContext represents the context of a testcase
type DefaultTestCaseContext struct {
	venom.CommonTestCaseContext
	datas map[string]interface{}
}

// Init Initialize the context
func (tcc *DefaultTestCaseContext) Init() error {
	return nil
}

// Close the context
func (tcc *DefaultTestCaseContext) Close() error {
	return nil
}

Methods SetTestCase and GetName are implemented by CommonTestCaseContext

Dependencies

Individual packages were updated using the rough procedure:

  1. dep ensure
  2. dep ensure -update ${PACKAGE}
  3. dep prune
  4. go build

Hacking

You've developed a new cool feature? Fixed an annoying bug? We'd be happy to hear from you! Make sure to read CONTRIBUTING.md before.

License

This work is under the BSD license, see the LICENSE file for details.

You can’t perform that action at this time.