Skip to content

Case Study: Generating Unit Tests to get to 95% coverage in under 30 mins

Yureshwar Ravuri edited this page Jul 1, 2025 · 7 revisions

Getting to +90% coverage under 30 mins

Generating unit tests with genAI is incredible fast and quite thorough. We can get to ~90% coverage with minimal effort. Here is an interesting comparison of LLMs performance The generation of the unit tests itself takes under 5 mins. We then need to read, validate, run and extend these unit tests. As with all genAI output manual verification and validation is critical.

  • Here is the original file for NanoHealth method for generating OTP
    • As with all startups under pressure to deliver features, Nanohealth too prioritised velocity of features and overlooked code coverage.
    • As number of customers grew, there was a demand for more changes in this functionality.
    • The team was starting to break more things with every new feature and this was causing delays and stress. This was also slowing down the pace of release of new features and hence affecting the velocity of revenue.
  • Below are the unit tests generated by genAI.
    • These tests were generated in multiple iterations using Cody and Claude Sonnet 3.5. All iterations cumulatively took less than 20 mins.
    • The developer spent about 30 mins to review, run and finalise the unit-tests.
      • The genAI generated tests were manually reviewed for relevance, accuracy, necessity (remove some unnecessary unit-tests), coverage and edge cases.
      • Some unit tests were added manually.
      • Some generated tests were edited by hand.

Prompt that can be used for Generating Unit Tests

Assume the role of a senior software architect specializing in Node.js and JavaScript testing. Analyze the provided code, which has been refined for error handling, input validation, sanitization, type strictness, security enhancements, and documentation. Your task is to design a robust suite of unit tests that thoroughly validate its functionality.

Leverage the existing documentation and inline comments to ensure that all logic branches, conditions, and edge cases are covered. Write unit tests that verify expected behaviors, handle erroneous inputs, and validate security-related checks. Ensure that all errors in the unit tests themselves are handled properly to avoid misleading results.

Additionally, document the unit tests thoroughly. Provide clear function descriptions, expected inputs/outputs, and inline comments explaining the purpose of each test case. Maintain consistency in formatting and structure, ensuring the tests are easy to understand and maintain.

Do not suggest any setup for the testing framework; assume the testing environment is already configured. Strictly focus on writing well-documented and structured unit tests with proper error handling.

Provide a detailed report outlining the test coverage, identified gaps (if any), and scenarios included. At the end of the analysis, present the complete set of unit tests with inline comments and documentation in a structured final output, ensuring clarity and ease of implementation.

For this functionality, the developer went from ~10% coverage to ~100% coverage in under an hour.

The time for manual review will vary with code complexity. Despite this added time for manual review, genAI will nonetheless reduce time/effort for unit-tests from 75%(complex functionality) to 95%(simple functionality).

Note 1: This time estimate assumes that the unit-test libraries and frameworks are installed and working. Installing these is a one-time effort for the project.

Note 2: Below tests were generated before we added input sanitisation. There were more unit tests generated after we added the input sanitisation but for illustrative purposes, the below file is adequate.

The generated unit-tests are listed below to illustrate the depth and rigour that can be expected.

# @title ABHA Web Services Controller Test Suite
# @description Comprehensive test suite for AbhaWebServicesController that handles ABHA
#             (Ayushman Bharat Health Account) web service integration testing.
#
# Key Features Tested:
# - Aadhaar OTP generation
# - Input validation
# - Error handling
# - Authentication flows
# - Response formatting
# - Rate limiting
# - Service maintenance
#
# Test Coverage Areas:
# 1. Valid parameter handling
# 2. Invalid parameter validation
# 3. Authentication scenarios
# 4. Service response handling
# 5. Error conditions
# 6. Network timeouts
# 7. Security aspects
# 8. Performance monitoring
#
# @author [Your Team Name]
# @version 1.0
# @since [Date]

RSpec.describe AbhaWebServicesController, type: :controller do
  describe '#generate_aadhar_otp' do
    # Test data setup
    let(:valid_aadhaar) { "123412341234" }
    let(:invalid_aadhaar) { "123456" }
    let(:auth_token) { "valid_token_123" }

    # @section Valid Parameters Testing
    # Tests successful OTP generation with valid inputs
    context 'with valid parameters' do
      before do
        allow(VerhoeffValidator).to receive(:validate_aadhaar)
          .with(mock_aadhaar_numbers[:valid])
          .and_return(true)
        allow(AbhaWebService).to receive(:authenticate_abha)
          .and_return(mock_auth_tokens[:valid])
        allow(AbhaWebService).to receive(:generate_aadhar_otp)
          .and_return(mock_responses[:success])
      end

      it 'generates OTP successfully' do
        post :generate_aadhar_otp, params: { aadhar: mock_aadhaar_numbers[:valid] }
        expect(response).to have_http_status(:success)
        expect(JSON.parse(response.body)["status"]).to eq("success")
      end
    end

    context 'with invalid parameters' do
      it 'returns error for missing Aadhaar' do
        post :generate_aadhar_otp, params: {}
        expect(JSON.parse(response.body)["status"]).to eq("error")
        expect(JSON.parse(response.body)["message"]).to eq("Aadhaar number is required")
      end

      it 'returns error for invalid Aadhaar format' do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(invalid_aadhaar).and_return(false)
        post :generate_aadhar_otp, params: { aadhar: invalid_aadhaar }
        expect(JSON.parse(response.body)["status"]).to eq("error")
        expect(JSON.parse(response.body)["message"]).to include("Invalid Aadhaar number")
      end
    end

    context 'with authentication failures' do
      before do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        allow(AbhaWebService).to receive(:authenticate_abha).and_return(nil)
      end

      it 'returns error for authentication failure' do
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["status"]).to eq("error")
        expect(JSON.parse(response.body)["message"]).to include("Authentication failed")
      end
    end
    
    context 'with ABHA service responses' do
      before do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        allow(AbhaWebService).to receive(:authenticate_abha).and_return(auth_token)
      end

      it 'handles 400 response' do
        allow(AbhaWebService).to receive(:generate_aadhar_otp).and_return({ resp_status: "400", resp_body: "Bad Request" })
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["code"]).to eq(400)
      end

      it 'handles 401 response' do
        allow(AbhaWebService).to receive(:generate_aadhar_otp).and_return({ resp_status: "401", resp_body: "Unauthorized" })
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["code"]).to eq(401)
      end

      it 'handles 429 response' do
        allow(AbhaWebService).to receive(:generate_aadhar_otp).and_return({ resp_status: "429", resp_body: "Too Many Requests" })
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["code"]).to eq(429)
      end

      it 'handles 500 response' do
        allow(AbhaWebService).to receive(:generate_aadhar_otp).and_return({ resp_status: "500", resp_body: "Server Error" })
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["code"]).to eq(500)
      end
    end

    context 'with malformed JSON responses' do
      before do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        allow(AbhaWebService).to receive(:authenticate_abha).and_return(auth_token)
      end

      it 'handles invalid JSON in response body' do
        allow(AbhaWebService).to receive(:generate_aadhar_otp).and_return({ resp_status: "200", resp_body: "invalid json" })
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["status"]).to eq("error")
        expect(JSON.parse(response.body)["message"]).to include("Invalid response format")
      end

      it 'handles missing transaction ID in success response' do
        allow(AbhaWebService).to receive(:generate_aadhar_otp).and_return({ resp_status: "200", resp_body: "{}" })
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["status"]).to eq("error")
      end
    end

    context 'with network timeouts' do
      before do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        allow(AbhaWebService).to receive(:authenticate_abha).and_return(auth_token)
      end

      it 'handles connection timeout' do
        allow(AbhaWebService).to receive(:generate_aadhar_otp).and_raise(Timeout::Error)
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["status"]).to eq("error")
        expect(JSON.parse(response.body)["code"]).to eq(500)
      end
    end

    context 'with encryption scenarios' do
      it 'handles encryption failures' do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        allow(AbhaWebService).to receive(:authenticate_abha).and_return(auth_token)
        allow(RsaEncryption).to receive(:oaep_mgf1_padding).and_raise(OpenSSL::PKey::RSAError)
        
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["status"]).to eq("error")
      end
    end

    context 'with rate limiting' do
      it 'handles concurrent requests' do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        allow(AbhaWebService).to receive(:authenticate_abha).and_return(auth_token)
        
        3.times do
          post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        end
        
        expect(response).to have_http_status(:success)
      end
    end

    context 'with special characters' do
      it 'sanitizes input parameters' do
        post :generate_aadhar_otp, params: { aadhar: "123412341234<script>" }
        expect(JSON.parse(response.body)["status"]).to eq("error")
      end
    end

    context 'with different Aadhaar formats' do
      it 'handles Aadhaar with spaces' do
        post :generate_aadhar_otp, params: { aadhar: "1234 1234 1234" }
        expect(JSON.parse(response.body)["status"]).to eq("error")
      end

      it 'handles Aadhaar with dashes' do
        post :generate_aadhar_otp, params: { aadhar: "1234-1234-1234" }
        expect(JSON.parse(response.body)["status"]).to eq("error")
      end
    end

    context 'with token expiration scenarios' do
      it 'handles expired authentication token' do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        allow(AbhaWebService).to receive(:authenticate_abha).and_return('expired_token')
        allow(AbhaWebService).to receive(:generate_aadhar_otp).and_return({ resp_status: "401", resp_body: "Token expired" })
        
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["code"]).to eq(401)
      end

      it 'attempts token refresh on expiry' do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        allow(AbhaWebService).to receive(:authenticate_abha).and_return(auth_token, 'new_token')
        
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(AbhaWebService).to have_received(:authenticate_abha).twice
      end
    end

    context 'with response caching' do
      it 'caches successful responses' do
        allow(Rails.cache).to receive(:write)
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(Rails.cache).to have_received(:write)
      end
    end

    context 'with performance monitoring' do
      it 'logs response time' do
        allow(Rails.logger).to receive(:info)
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(Rails.logger).to have_received(:info).with(/Response time:/)
      end
    end

    context 'with multiple OTP requests' do
      it 'handles sequential OTP requests' do
        allow(VerhoeffValidator).to receive(:validate_aadhaar).with(valid_aadhaar).and_return(true)
        
        2.times do
          post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
          expect(response).to have_http_status(:success)
        end
      end
    end

    context 'with service maintenance mode' do
      it 'handles service maintenance windows' do
        allow(AbhaWebService).to receive(:maintenance_mode?).and_return(true)
        
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(JSON.parse(response.body)["message"]).to include("maintenance")
      end
    end

    context 'with request validation' do
      it 'validates request headers' do
        request.headers['X-API-Version'] = 'invalid'
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(response).to have_http_status(:bad_request)
      end

      it 'validates content type' do
        request.headers['Content-Type'] = 'text/plain'
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(response).to have_http_status(:unsupported_media_type)
      end
    end

    context 'with response format validation' do
      it 'validates response structure' do
        post :generate_aadhar_otp, params: { aadhar: valid_aadhaar }
        expect(response.content_type).to eq('application/json')
        expect(JSON.parse(response.body)).to include('status')
      end
    end
  end
end

Clone this wiki locally