The TestRunner (Extracted from carlin).
Clone or download
saigon and mihirkamdarcouchbase Revert "CBQE-3988 add install cb for ubuntu 18.04."
Revert this commit to create Alice branch and merge it back to master (Mad-Hatter)
This reverts commit b6ecc65.

Change-Id: Ic523034460e71a8847050382a3b4be80e923e44c
Reviewed-on: http://review.couchbase.org/99909
Reviewed-by: Mihir Kamdar <mihir.kamdar@couchbase.com>
Tested-by: Mihir Kamdar <mihir.kamdar@couchbase.com>
Latest commit 1c62eac Sep 24, 2018
Permalink
Failed to load latest commit information.
b/resources RQG part for coalesce, nvl Sep 14, 2018
cloudtest K8S-268, Fix to add Kubernetes nodes to server list in testrunner Apr 13, 2018
conf CBQE-4791 : Eventing backlog Sep 21, 2018
doc More ec2 instructions for faster dev cycle. Apr 13, 2012
enginetests CBQE-0: fix ep-engine access log tests Dec 19, 2014
lib Revert "CBQE-3988 add install cb for ubuntu 18.04." Sep 24, 2018
longevity modified testrunner code to work with ns_server Jun 27, 2011
pysystests fix ClusterStatus.update_orchestrator to reuse Feb 8, 2018
pytests fix for add user Sep 21, 2018
resources CBQE-3942 add import export with secure connection tests Jul 17, 2017
scripts Adding message with server details when installation failed Sep 18, 2018
unittests CBQE-144: Removed unused imports/variables in unittests/ May 31, 2012
.gitignore CBQE-0: added to tests for crash mater/cluster nodes during xdcr May 13, 2014
Makefile Add targets for view CI Dec 7, 2017
README.md CBQE-0 correct command line to run testrunner Jan 31, 2015
TestInput.py CBQE-4333: Refactor eventing code to support the runs through cluster… Oct 11, 2017
array_simple_table.txt changes for rqg: addition of count and min with push down. May 17, 2016
buildout.cfg MB-100: update btrc version Apr 9, 2013
logging.conf.sample Add lineno to the testrunner logs May 8, 2012
mcsoda-setup.py create a python egg for mcsoda Jan 18, 2012
mcsoda.logging.conf MB-100: fix formatting in mcsoda logging config Dec 4, 2012
scripts.logging.conf only use consolehandler while installing cb/mb Jan 11, 2012
setup.py Revert "Merge branch 'testrunner-ng'" Oct 28, 2016
testrunner Rename testrunner to testrunner.py Jun 16, 2014
testrunner.py CBQE-0: Add -p to the testrunner command printed before every test, w… Mar 21, 2018

README.md

Prerequisites

  • Python 2.6 or 2.7
  • pip or easy_install

Dependencies

General:

pip install paramiko

pip install boto

Performance tests:

pip install btrc

PDF reports:

pip install couchdbkit

Documentation:

pip install sphinx

pip install sphinx-pypi-upload

Buildout:

pip install zc.buildout

Usage

$ ./testrunner  -h
Usage: testrunner [options]

Options:
  -h, --help            show this help message and exit
  -q
  -p PARAMS, --params=PARAMS
                        Optional key=value parameters, comma-separated -p
                        k=v,k2=v2,...
  -n, --noop            NO-OP - emit test names, but don't actually run them
                        e.g -n true
  -l LOGLEVEL, --log-level=LOGLEVEL
                        e.g -l info,warning,error

  TestCase/Runlist Options:
    -i INI, --ini=INI   Path to .ini file containing server information,e.g -i
                        tmp/local.ini
    -c RUNLIST, --config=RUNLIST
                        Config file name (located in the conf subdirectory),
                        e.g -c py-view.conf
    -t TESTCASE, --test=TESTCASE
                        Test name (multiple -t options add more tests) e.g -t
                        performance.perf.DiskDrainRate

Buildout

Initiate buildout directory structure, create sandbox, build packages and scripts, fetch dependencies, and etc.:

buildout

You can execute testrunner now:

./bin/testrunner -h

Resource Files

Ini files represents the ns_server information which is accessible to the tests.

[global] section defines the rest username, password that tests use to login to ns_server.

Example:

[global]
username:Administrator
password:membase

[membase]
rest_username:Administrator
rest_password:asdasd

[servers] section lists port and ssh related information. ssh connection information is required for small subset of tests where test needs to perform installation,backup or restore. If ns_server instances are started using ns_server/cluster_run script then you only need to define ip and port for those nodes.

Example:

[servers]
1:10.1.6.104_1
2:10.1.6.104_2
3:10.1.6.104_3
4:10.1.6.104_4

[10.1.6.104_1]
ip:10.1.6.104
port:9000

[10.1.6.104_2]
ip:10.1.6.104
port:9001

[10.1.6.104_3]
ip:10.1.6.104
port:9002

[10.1.6.104_4]
ip:10.1.6.104
port:9003

Test Execution and Reporting

For every test run testrunner creates a temp folder and dumps the logs and xunit reports in the newly generated folder.

for instance if you run

$ python testrunner.py -i resources/jenkins/single-node-centos-32.ini -t setgettests.MembaseBucket.value_100b -p your_first_parameter=x,your_second_parameter=y

you will see this summary after each test is ran:

summary so far suite setgettests.MembaseBucket , pass 1 , fail 0
logs and results are available under tmp-12-11-47

and logs:

$ ls  tmp-12-11-47/
report-12-11-47.xml-setgettests.MembaseBucket.xml	value_100b.log

Development

When using git on Linux/OSX systems, you might run into issues where git incorrectly believes Windows-related files have been modified. In reality, git is merely mis-treating CRLF line endings. Try the following...

$ cd testrunner
$ git config core.autocrlf false

Running End to End Tests Before Submitting a Patch to Gerrit

Testrunner project has different test suites which can be run priori to submitting the code to gerrit. There are test suites that can be run against a single node which validates basic database operations such as persistence and bucket management. There are also key-value clustering related test cases which can be run against Membase/Couchbase 1.8 multiple nodes. Recently we have also been adding more tests which validates basic view functionalities on a cluster and on a single.

"make e2e-kv-single-node"

This make target will start ns_server using cluster_run -n1 and run all the test cases listed in conf/py-all-dev.conf. The test runtime can vary between 15-30 minutes depending on your machine.

Testrunner prints out a human readable pass/fail of the tests. Please refer to the "rerunning test" section for more information of how to re-run one single test against cluster_run.

"make test-views"

This make target will start four ns_server(s) using cluster_run -n4 and runs all the test cases listed in conf/py-view.conf. The test runtime can vary between 30-45 minutes. py-view.conf contains basic test cases which validates clustering operations such as rebalancing.Each test is also parametrized so you can easily modify this run list and change the number of docs or the load_duration.

For instance, test_get_view_during_x_min_load_y_working_set,num-docs=10000,load-time=1,run-view-time=1 test will create a view, inserts 1000 documents, mutate those documents for 1 minute and run view queries in parallel to the load for 1 min. You can easily change the parameters there to insert 1M items and keep the load running for 10 mins for example.

Using mcsoda to load json documents on to a node/cluster

mcsoda is a load generator tool that randomly generates json documents. It is available under testrunner/lib/perf_engines/ .

Mcsoda uses moxi port to distribute load in a cluster. The moxi can be set up on the server side or on a client side as follows:

"server-side moxi"

From a client, load can be run on a cluster represented by: xx.xx.xx.xxx as:

lib/perf_engines/mcsoda.py xx.xx.xx.xxx:11211 vbuckets=1024 doc-gen=0 doc-cache=0
ratio-creates=0.5 ratio-sets=0.2 ratio-deletes=0.02  ratio-expirations=0.03
expiration=3600 min-value-size=2,4 threads=100 max-items=18000000
exit-after-creates=1 prefix=KEY_ max-creates=18000000

This is going to ensure that there will be 20% sets against 80% gets. Of the 20% sets, 50% will be creates and the rest updates. There shall be 2% deletes, and minimum item size varies from 2 to 4 Bytes. 3% of the items set will be marked as expired after a duration of 1 hour. The tool will use 100 threads to generate the json load, and every item will be prefixed by "KEY_". max_creates limits the no. of items created. All the load generated is going to get loaded on bucket "default". Prefix the IP by "bucket_name:password@", to load on to standard/sasl buckets.

For more options in mcsoda, type in: lib/perf_engines/mcsoda.py -h

"client-side moxi"

Set up the moxi for a specific bucket on the client side as follows: (Make sure couchbase-server is installed on the client)

/opt/couchbase/bin/moxi -Z usr=Administrator,pwd=password,port_listen=11611,
concurrency=1024,wait_queue_timeout=200,connect_timeout=400,connect_max_errors=3,
connect_retry_interval=30000,auth_timeout=100,downstream_conn_max=128,downstream_timeout=5000,
cycle=200 http://xx.xx.xx.xxx:8091/pools/default/bucketsStreaming/[bucket_name] -d

This command sets up moxi for the cluster on the client side at port 11611. For more information on the moxi command, type in: /opt/couchbase/bin/moxi -h

Once the moxi is all set up, run mcsoda against the moxi just set up:

lib/perf_engines/mcsoda.py localhost:11611 vbuckets=1024 doc-gen=0 doc-cache=0
ratio-creates=0.5 ratio-sets=0.2 ratio-deletes=0.02  min-value-size=2,4 threads=100
max-items=18000000 exit-after-creates=1 prefix=KEY_ max-creates=18000000