Skip to content

proposal deterministic improvements in integration tests

Jump to bottom
amymullins edited this page May 18, 2016 · 5 revisions

Deterministic Improvements in integration tests

Background

The initial functional/integration tests running from https://github.com/RackHD/RackHD/tree/master/test are not yet being reliable, falling prey to failures due to inconsistencies and lack of specific determinism. As we build these out to improve our quality gates (PR pull request validation) efforts for RackHD, we need to drive to more deterministic tests that work the system to properly and quickly identify errors.

This proposal covers the goals and efforts required to help achieve greater stability and deterministic behavior out of the RackHD unit and smoke/functional test suite.

Goals (Short term, baseline setting)

  • A dedicated system will be setup to continuously run through the current RackHD unit and smoke/functional test suites. The test output will be reviewed, errors identified, the RackHD Dev leads will identify/prioritize issues their teams will resolve and drive to close out those issues.
    • Setup a dedicated slave node that will support continuous unit and smoke/functional testing.
    • Continuously execute the unit test suite. Suite currently averages 1 minute 9 seconds to run. Plan for ¾ to 1 full day of testing (will run > 300 iterations of the test suite for a sizeable sample set of data).
      • Data to be made available to the distributed RackHD dev teams for analysis.
      • Dev managers to identify issues their teams will investigate and/or resolve, team backlogs to be updated accordingly.
    • Continuously execute the smoke/functional test suite. Suite currently averages ~14 minutes to run. Plan for 3 days of testing (will run > 300 iterations over a weekend for a sizeable sample set of data).
      • Data to be made available to the distributed RackHD dev teams for analysis.
      • Dev managers to identify issues their teams will investigate and/or resolve, team backlogs to be updated accordingly.

Goals (Longer term)

  • Setup Jenkins to provide a daily email (or some other mechanism) to communicate all the failing tests from the day before.
    • Set up a process/expectation on the development teams to review the daily report and debug any new errors that may have been introduced with their corresponding PRs.
  • Open up Github issues for errors encountered during the PR quality gate checks (unit/smoke testing).
    • Requires more frequent Github issue triage and proactive assignment from the RackHD Dev leads.
    • Can this be done automatically?

API

Discussion

Update 4/19/16

Update 5/11/16

Update 5/18/16

Clone this wiki locally