Redesign / Rework test infra

[msterin]

Our current test infra ('make test-all' and CI) is comprise of python server-side tests, GO client-side tests, and Go end to end test which sometimes call .sh (e.g. refcnt_test.sh). Issue:

• It requires Docker config to accept TCP connection (tracked at refactor existing go tests which requires Docker config to accept TCP connection #1011 )
• it does not provide test coverage or lint coverage ( Add code coverage for python tests (test-esx) #404, Capture plugin code coverage #869 )
• it's hard to add simple trivial things which can be done in scripts since test APIs/libs are not clearly defined
  Examples: we do not have some test automated due to this e.g.

Notes from @shuklanirdesh82: we have started implementing test library to reuse code while writing new automation tests so marking this as addressed.

• multiple datastores test ( addressed via Identify tests for multi-esx scenario #957)

• ESX failover test ( addressed via Identify tests for multi-esx scenario #957)

• VM snapshot/vmotion test (tracked at [test automation] VM snapshot tests #1012 and [test automation] VM migration tests #1013 respectively)

• scale/perf tests (tracked at Add tests for perf/scale, per goals in spec #63)

  etc...

This Issue asks for a desgin/discussion and plan of record for reworked test infra and new test sets for GA. We may create other issues to address actual work

//CC @pdhamdhere @kerneltime

[govint]

Also, the infra should allow easy addition of new tests. My suggestions are:

a. defining a template for a test script which is coded by who ever needs to add a test case. The script/code is dropped into the repository and is then picked up by the infra and included in a test run. Developer just needs to supply the functions in the template - say, init(), run(), cleanup().

b. assign IDs for tests so we know what tests in a list have passed/failed.

c. Categorize the tests into groups so they can be run independently, developer may want to run a subset depending on what code they fix.

[kerneltime]

We have need for VMs that have been running over a long duration as well as VMs that get setup as new. #584

[shuklanirdesh82]

Govindan's comment #472 (comment)

need to find out better test automation framework which satisfies all the requirement mentioned in the above comment.

Note: keeping this issue open to address above comment.

Mark's comment #472 (comment)

separate issues are created to keep track of each action item so moving this issue in to future milestone.

[shuklanirdesh82]

Sharing some details from my research. gocheck serves our need. There are many good automation framework available and gocheck is sufficient for our e2e tests. Code contribution guideline at docker leverages the same. One more advantage of using gocheck is to reuse existing developed tests with minor modification to use gocheck library on top of golang's inbuilt testing library (some of our existing tests are written using that).

As a next step, one sample testcase should be automated using gocheck.

[shuklanirdesh82]

new test framework integration task is done through #1246 so closing this Issue.