What We Accomplished Using This Process

When we first developed this process, the results were amazing--unbelievable, really. For a very small up-front effort, we were able to realize all of the following process improvements and many more. Some are a direct result of this effort, and some are "collateral improvements" as a result of decreasing our lead time.

Direct Improvements (where this process itself had impact)

• Moved the conversation about broken test code from post-build, post-deployment to the developer's pre-check-in build. The presentation layer was synchronized to the automation code at compile-time, so a drastic change would show up on the developer's sandbox prior to check-in
• Reduced our time spend maintaining broken test code from 40% of budgeted time to around 3% (we stopped tracking it)
• Improved our test pass rate to 98.5% on a regular basis (fewer than 20 failing tests in 1,800)
• Implemented "self-tests" for areas of the product, wherein we could use reflection or some other technique to execute smart tests that could take the pulse of the application on a daily basis
• Fully integrated our own test code into the developer's environment--synchronized all tools, all languages, all processes
• Extremely complicated web controls (i.e. complex tables with multiple filterable/sortable/chart-able columns) could be addressed, changed, and referenced with single lines of code

Indirect Improvements (where this process led to other improvements)

• Code coverage against instrumented application increased steadily and consistently (because tests always passed, we weren't failing out early and skipping areas of coverage)
• Vastly improved relationship with Developers as a result of being fully integrated with their processes. We could break their build as easily as they could break ours, so they knew we weren't cheating
• The Page Object generation and WebDriver/WebElement framework itself was unit tested, allowing us to learn and propagate these good habits throughout the team

Collateral Improvements (where our increased velocity allowed us more time to do things)

• We were able to solve the problem of "waitForPageToLoad" there was neither sleep() or wait() statement anywhere in our functional test suite--it was completely abstracted into the click() method of ClickableWebElement
• We became intimately familiar with the dev platform on the underlying application, and on occasion could use this knowledge to identify and analyze defects so that reproduction steps were extremely useful for developers