Skip to content

User Guide

Jump to bottom
ZhuPing edited this page Sep 15, 2018 · 52 revisions

User Guide

Contents

Syntax

Basic Test Case Json Syntax

[
  {
    "TestCase-Syntax-001": {
      "priority": "1",
      "parentTestCase": "root",
      "inputs": ["s1ParentTestCase_out.csv", "join", "s1ParentTestCase_out2.csv"],
      "request": {
        "method": "GET|POST|PUT|DELETE",
        "path": "https://api.douban.com/v2/movie/top250",
        "headers": {
          "User-Agent": ""
        },
        "queryString": {
          "pageIndex": "1",
          "pageSize": "12"
        },
        "payload": {
          "text": {
            "id": "1000000001",
            "Name": "Name",
            "province": "province",
            "city": "city"
          },
          "filename": "uploadfile.xlsx",
          "name": "excel"
        }
      },
      "response": {
        "status": {
          "Equals": 200
        },
        "headers": {
          "Content-Type": {
            "Contains": "application/json"
          }
        },
        "body": {
          "$.start": {
            "GreaterOrEquals": 0
          }
        }
      },
      "outputs": [
        {
          "filename": "s2ParentTestCase_out.csv",
          "format": "csv",
          "data": {
            "title": ["$.subjects.0.title"],
            "count2": [20]
          }
        }
      ]
    }
  }
]

Test Case Json with Variables driven by related Data

The related Data has three sources with lookup sequence: config.json -> system environment variable -> data file (.csv), but latter will override former one.

config.json has layout:

{
    "DEV": {
    	"baseUrl": "https://api.douban.com"
    },
    "QA": {
    	"baseUrl": "https://api.douban.com"
    }
}

system environment variable has prefix go4_*, which is suitable for some confidential information like token or password:

$ export
go4_authorization="Bearer eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOi ..."

For example:

  • Json file with variables {{.xxx}} 
[
  {
    "TestCase-Syntax-{{.tcid}}": {
      "priority": "{{.priority}}",
      "parentTestCase": "{{parentTestCase}}",
      "inputs": ["s1ParentTestCase_out.csv", "join", "s1ParentTestCase_out2.csv"],
      "request": {
        "method": "GET",
    ...
      "response": {
        "status": {
          "Equals": {{.statuscode}}
        },
    ...
  }
]
  • Data file (.csv)
tcid,priority,parentTestCase,statuscode
dt1-01,1,root,200
dt1-02,2,TestCase-Syntax-dt1-01,400
  • Two Test Cases will be generated based on the rows in Data file, they are:
[
  {
    "TestCase-Syntax-dt1-01": {
      "priority": "1",
      "parentTestCase": "root",
      "inputs": ["s1ParentTestCase_out.csv", "join", "s1ParentTestCase_out2.csv"],
      "request": {
        "method": "GET",
    ...
      "response": {
        "status": {
          "Equals": 200
        },
    ...
  }
]
[
  {
    "TestCase-Syntax-dt1-02": {
      "priority": "2",
      "parentTestCase": "TestCase-Syntax-dt1-01",
      "inputs": ["s1ParentTestCase_out.csv", "join", "s1ParentTestCase_out2.csv"],
      "request": {
        "method": "GET",
    ...
      "response": {
        "status": {
          "Equals": 400
        },
    ...
  }
]

1. Getting Started

Please refer to Install and Run, Quick start in page Tab Code

2. Command Options

$./go4api -h

go4api version: 0.15.0

Usage:
  go4api [command] [options]

Available Commands:
  run         Start a test
  convert     Convert a HAR file / Swagger API file to a go4api Json test case
Command: run
Usage: go4api -run [-hFMS] [-c config filename] [-t testcase path] [-d test resource path] [-r test results path] 

Options:
  -F  if to run the Fuzz test
  -M  if to run the Mutation test
  -S  if the target cases are for scenarios, which have data dependency
  -baseUrl string
      the baseUrl
  -c string
      the path which test config in (default "$GOPATH/src/go4api/testconfig")
  -cl int
      concurrency limitation (default 100)
  
  -r string
      the path which test results in (default "$GOPATH/src/go4api/testresults")

  -tc string
      the path which test json in (default "$GOPATH/src/go4api/testcase")
  -testEnv string
      the testEnv, i.e. dev, qa, uat, etc. (default "QA")
  -tr string
      the path which test resource in (default "$GOPATH/src/go4api/testresource")
Command: convert
Usage: go4api -convert [-harfile har filename] [-swaggerfile swagger api filename]

Options:
  -harfile string    	
  -swaggerfile string
    	har file name to be converted

3. Test Case Json Layout and Fields

3.1 Layout:

  • Each Json file is an Array of Test Cases
  • Each Test Case is also Json
  • Each Test Case Json has the Test Case Name as the key, make sure it is distinct

3.2 Fields:

  • priority (required)
  • parentTestCase (required, current case's parent, represents the dependency)
  • inputs
  • request (required)
    • method (required, GET|POST|PUT|DELETE)
    • path (required, url)
    • headers
    • queryString
    • payload (required for PSOT/PUT body)
  • response (required)
    • status (200, 201, etc.)
    • headers
    • body (response body)
  • outputs

4. Results lookup path in response body

4.1 response structure
In Go4Api, the response has the structure with assertion key, expected results, actual results

- response  
  - status
    - <assertion key> <expected results>
  - headers  
    - <header key> <assertion key> <expected results>
  - body  
    - <body lookup path> <assertion key> <expected results>

4.2 Go4Api supports the actual results search in response body using lookup path, the syntax like

"$.start"
"$.subjects.#"
"$.total"
"$.subjects.0.title"

5. Inputs

5.1 inputs syntax

"inputs": [<file_1.csv>, <operator 1>, <file_2.csv>, <operator 2 >, <file_3.csv>, ...]

5.1 inputs file operator
There supports file operators:

  • Union (union two csv files data rows, make sure two files have all the same header fields (not required the same order))
  • Join (join two csv files data rows using Cartesian product, make sure two files have no same header fields)
  • Append (if two files have some same header fields, will append the latter csv field(s) to former csv field(s))

6. Outputs

6.1 onputs syntax

"outputs": [
        {
          "filename": "out_file_1.csv"
          "format": "csv",
          "data": {
            "<csv header>": ["$.subjects.0.title"],
            "<csv header>": [[20, 30]]
          }
        },
        {
          "filename": "out_file_2.csv",
          "format": "csv",
          "data": {
            "<csv header>": [<body lookup path>, "$.subjects.0.title"],
            "<csv header>": [20]
          }
        }
      ]

7. Assertion Key in response and outputs

There supports the assertion keys:

  • Equals (for all types)
  • NotEquals (for all types)
  • Contains (for String)
  • StartsWith (for String)
  • EndsWith (for String)
  • Less (for Numbers)
  • LessOrEquals (for Numbers)
  • Greater (for Numbers)
  • GreaterOrEquals (for Numbers)
  • In (check if item is in Array/Slice)
  • NotIn (check if item is not in Array/Slice)
  • Has (check if Array/Slice has item)
  • NotHas (check if Array/Slice does not has item)
  • Match(uses regular expression to assert String)

8. Scenarios - Multiple APIs Data Dependency

inputs and outputs plays the channel role for data transfer. That is, parent(s) generate the outputs files, then child test case(s) use the outputs files as inputs, drive the child test case(s), the child test case(s) can generate there own outputs files, and so on.

samples can be found under scenarios samples.

9. Mutation Test

Mutation Test starts with a Snapshot of a working request/response, traverses all the key/value(s) or node(s) under request headers, queryString, payload, to mutate them (includes update, add, delete) which results in new Test Case(s) called Mutation Cases, finally, execute all these Mutation Cases to see how the API(s) works.

9.1 Fuzz Test - Mutation Test
Prepare a Test Case, samples can be found under mutation samples, then get results on console like:

...
MutationTestCase-001-M-H-S-8            1  root                          Fail      MutationTeseCase.json         .                                 200
Update/Set header key: User-Agent, `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 ...
...

MutationTestCase-001-M-QS-S-15          5  root                          Fail      MutationTeseCase.json         .                                 200
Update/Set header key: pageIndex, `1`, `true`
[{"AssertionResults":"Failed","ReponsePart":"Body","FieldName":"$.total","AssertionKey":"Equals", ...
...

MutationTestCase-001-M-QS-D-1           6  root                          Fail      MutationTeseCase.json         .                                 200
Remove querystring key: `pageIndex`
[{"AssertionResults":"Failed","ReponsePart":"Body","FieldName":"$.total","AssertionKey":"Equals", ...
...

MutationTestCase-001-M-QS-D-0           8  root                          Fail      MutationTeseCase.json         .                                 200
Remove all querystring
[{"AssertionResults":"Failed","ReponsePart":"Body","FieldName":"$.total","AssertionKey":"Equals", ...
...

9.2 Fuzz Test - Mutation Rules
Below shows a snippet about the Muatation Rules:

//----------------------------------------------
//------- Mutation Rules - Types ---------------
//----------------------------------------------
//------ mutation: if normal char 
// (1) -> empty ("")
// (2) -> blank (" ")
// (3) -> prefix blank (" " + previousValue)
// (4) -> suffix blank (previousValue + " ")
// (5) -> mid blank (previousValue[0:2] + " " + previousValue[2:])
// (6) -> only one char (previousValue[0])
// (7) -> longlong string (strings.Repeat(previousValue, 50)
// (8) -> special char(s) (~!@#$%^&*()_+{}[]<>?)
// (9) -> null
// (10) -> change type (simple, i.e. to float64/bool...)
// (11) -> change type (complex, i.e. object, arrary)
// (12) -> remove this node

//------ mutation: if not normal char -> all number
// (1) -> empty ("")
// (2) -> blank (" ")
// (3) -> not all number (char + previousValue[1:] / previousValue[0:-1] + char)
// (4) -> null
// (5) -> change type (simple, i.e. to float64/bool...)
// (6) -> change type (complex, i.e. object, arrary)
// (7) -> remove this node

...

10. Fuzz Test Based on Definition, Random

10.1 Fuzz Test - Random Test

10.2 Fuzz Test - Random Rules

11. Conversion

11.1 Har conversion

11.2 Swagger conversion

12. Reports

While running Go4Api, some reports displayed on console, like:

...
SecondTestCase-dt1-3                    2  root                          Success   SecondTeseCase.json           SecondTeseCase_dt1.csv        1   200
...
====> Priority 2 ended! 
...
====> Priority 3 starts!
SecondTestCase-dt2-3                    3  root                          Fail      SecondTeseCase.json           SecondTeseCase_dt2.csv        1   200
[{"AssertionResults":"Failed","ReponsePart":"Status","FieldName":"status","AssertionKey":"Equals","ActualValue":200,"ExpValue":500}]
...
---------------------------------------------------------------------------
----- Total 12 Cases in template -----
----- Total 12 Cases put onto tcTree -----
----- Total 1 Cases Skipped (Not Executed, due to Parent Failed) -----
----- Total 11 Cases Executed -----
----- Total 8 Cases Success -----
----- Total 3 Cases Fail -----
---------------------------------------------------------------------------
Report Generated at: /Users/testresults/2018-09-13 10:27:19.670974342 +0800 CST m=+0.001693383/index.html
Execution Finished at: 2018-09-13 10:27:29.36250866 +0800 CST m=+9.693603759

After execution finished, the Reports generated under target results folder, including the .log file with all cases' execution details in Json Lines and the UI reports.

testresults/
└── 2018-09-10\ 07:42:20.804070777\ +0800\ CST\ m=+0.001524050
    ├── 2018-09-10\ 07:42:20.804070777\ +0800\ CST\ m=+0.001524050.log
    ├── index.html
    ├── js
    └── style
Clone this wiki locally