Testing

Getting Started

Running Tests using the CLI

1. Ensure Maven is installed on your machine. See installation guide.
2. Navigate to the root directory of the project and run the following command in your terminal to execute the test suite:

mvn test

Running Tests via the GUI

1. Open the project in VS Code.
2. Ensure the Test Runner for Java extension is installed.
3. Navigate to the Test Explorer or locate the desired test class in the project.
4. Click on the Run Tests button to run the test suite or individual test.

For further information, visit Testing Java with Visual Studio Code.

Test Suite

The test suite comprises of unit and performance test for the CompileCommand, Facade, Grade, PDF, MarkSchemeLoader and TestRunner interfaces along with the GradeFactory and SubmissionExtractor classes.

The test suite focuses on validating the interfaces of the application rather than their implementations. This ensures that the tests are independent of any particular implementation. As a result, the tests focus on verifying the expected behaviours and adhering to the contracts defined by the interfaces.

Furthermore, the test suite is designed for full coverage, ensuring every method and edge case defined in the interfaces is rigorously tested. This approach provides confidence in the stability and correctness of the codebase.

Unit Tests

The AutoGrader supports full coverage testing across all interfaces using JUnit 4. Unit tests validate individual components in isolation, ensuring they perform as expected.

Below is an example of a unit test for the TestRunner interface:

public class TestRunnerTest {
    @Test
    public void testRunAllTests() {
        TestRunner<?> testRunner = new AssignmentTestRunner();
        List<?> results = testRunner.runAllTests();
        for (var result : results) {
            Assert.assertTrue(result != null);
        }
    }
}

Performance Test

Full coverage testing across the test suite measures the execution time of processes/methods and compares them against the performance limits defined in TestConstants (in ms).

This is achieved using an in-house microframework, ExecutionTimer, developed for precise performance analysis.

public class ExecutionTimer {
    public static <T> PerformanceTestResult testExecutionTime(Runnable task, long threshold, String testName) {
        long startTime = System.nanoTime();
        task.run();
        long endTime = System.nanoTime();
        long elapsedTime = (endTime - startTime) / 1_000_000;
        boolean success = elapsedTime <= threshold;
        System.out.println("Execution Time of " + testName + ": " + elapsedTime + " ms");
        return new PerformanceTestResult(testName, success, elapsedTime);
    }
}

Below is an example performance test for the GradingFacade's processSubmissions method (the system's main module, providing full-coverage testing):

public class FacadePerformanceTest {
    private static final PrintStream originalOut = System.out;

    @Test
    public void testFacadePerformance() throws Exception {
        System.setOut(new PrintStream(OutputStream.nullOutputStream()));
        Facade facade = new GradingFacade();
        PerformanceTestResult result = ExecutionTimer.testExecutionTime(
                () -> {
                    facade.processSubmissions("submissions.zip");
                },
                TestConstants.MAX_ALLOWABLE_THRESHOLD_MS,
                "Facade - Process Submissions");
        System.setOut(originalOut);
        assertTrue("Execution of processSubmissions took too long", result.isSuccess());
    }
}