Repository navigation
Accelerate sprint productivity with genAI assisted Unit Tests
Quantitative perspective: The benefits of unit tests, hyper-productivity by using genAI to generate unit-tests.
Given that:
- We had zero unit tests to begin with AND
- The speed at which genAI generates Unit Tests
The Digital Assistant (UDAN) project achieved a remarkable ~ 100% productivity increase by implementing AI-generated unit tests. This case study challenges traditional assumptions about the cost-benefit trade-offs of maintaining high test coverage and presents a compelling argument for leveraging generative AI in testing workflows.
In our digital assistant (UDAN) project, we doubled productivity just by adding Unit Tests. One of the reasons for this spectacular improvement for UDAN was because we did not have unit tests in the first place. Not having Unit Tests is a terrible omission but we were in a hurry to launch an MVP of the product/project and many best practices fell by the way-side. More here
(There are a lot of papers published quantifying benefits of Unit tests. Linked some below.)
- Early bug detection: Unit tests help catch bugs early in the development process, reducing the cost of fixing them later "The Economic Impacts of Inadequate Infrastructure for Software Testing"
- Improved code quality: High test coverage encourages better code design and maintainability.
- Faster development cycles: With a robust test suite, developers can confidently make changes and refactor code without fear of introducing regressions.
- Easier collaboration: Well-tested code is easier for other developers to understand and work with .
- +90% code coverage was always desired but is expensive. As per Microsoft research, Developers can end up spending 40% to 50% of their time writing Unit Tests to get to +90% coverage.
- Time pressure and tight deadlines (This was UDANs excuse)
- Lack of understanding of the benefits
- Perceived low return on investment (ROI)
- Inadequate testing culture and lack of management support
- Legacy codebases and technical debt
- Insufficient training and skill gaps
Generative AI fundamentally changes the testing equation by:
- Reducing test writing time by approximately 80%
- Maintaining high coverage levels (~90-100%)
- Enabling immediate test creation for new code
- 20% productivity gain from early bug detection
- 15% improvement in development velocity
- 80% reduction in test writing effort
- Net productivity increase of approximately 100%
- Staged Adoption: Begin with new features and gradually expand to legacy code. Easiest is to add unit tests whenever we pick up a bug/enhancement ticket for older code.
- Quality Control: Establish review protocols for AI-generated tests
- Team Training: Invest in training developers to effectively use and validate AI-generated tests
- Always validate AI-generated tests (Critical)
- Maintain human oversight of test quality and coverage
- Regular audit of test effectiveness
The integration of generative AI in unit testing represents a paradigm shift in software development economics. It eliminates the traditional trade-off between comprehensive testing and development speed, enabling teams to achieve both high quality and rapid delivery.
For technical leaders, this presents an opportunity to significantly improve development efficiency while maintaining or enhancing code quality. The key is to approach implementation strategically, with appropriate validation processes and team training in place.
Here is a rough illustrative calculation of productivity improvements in UDAN using genAI. The "story point" metric is for illustrative purpose but the overall benefit and conclusion is demonstrable.
- Initial productivity: 100 story points per sprint (2 weeks)
- Development time saved due to early bug detection: 20%
- Productivity gain from improved code quality and faster development cycles: 15%
- Effort ratio of unit-tests:coding reduced from 1:1 to 0.2:1 using genAI
- Unit test coverage maintained at 90%
-
Productivity gain from unit tests (as calculated earlier):
- Time saved from early bug detection: 100 * 0.2 = 20 story points
- Productivity gain from improved code quality and faster development: 100 * 0.15 = 15 story points
- Total productivity gain from unit tests: 20 + 15 = 35 story points
-
Productivity gain from reduced unit test effort using genAI:
- Original unit test effort: 100 story points (1:1 ratio)
- Reduced unit test effort using genAI: 100 * 0.2 = 20 story points
- Effort saved by reducing unit test effort with genAI: 100 - 20 = 80 story points
-
Total productivity improvement:
- Productivity gain from unit tests: 35 story points
- Effort saved by reducing unit test effort with genAI: 80 story points
- Total productivity improvement: 35 + 80 = 115 story points