Compatibility and regression testing for AI models in Java.
ModelMatrix4J is a Java testing framework for running the same behavioral scenario against multiple model configurations and comparing the results in a deterministic, test-friendly way.
The project is designed primarily for Spring AI applications, while keeping the core execution engine independent from Spring, model providers, and external services.
Current development stage: M2 — Minimal Core Execution + JUnit Integration
Next milestone: M3 — Spring AI + Ollama vertical slice
An AI application may work correctly with one model and behave differently with another.
Differences can appear in:
- textual output
- structured output
- tool selection and arguments
- retrieval behavior
- latency
- failures
- model availability
- repeatability
Traditional unit tests usually validate one configured model path.
ModelMatrix4J aims to make the model itself part of the test matrix.
Conceptually:
Scenario
|
+----------+----------+
| |
v v
Model A Model B
| |
+----------+----------+
|
v
Run Results
|
v
CompatibilityResult
The same scenario is executed against multiple model configurations and the results are classified separately from execution failures.
The current M2 implementation provides:
- provider-neutral scenarios
- model descriptors
- a small
ModelAdapterextension port - adapter-backed models under test
- deterministic matrix execution
- repetitions
- timeout handling
- cancellation handling
- unavailable-model classification
- execution-failure classification
- normalized textual comparison
- immutable run results
- bounded and sanitized diagnostics
- JUnit Jupiter integration
- deterministic offline tests
- parallel test isolation
The core does not depend on Spring, Spring AI, Ollama, MCP, PostgreSQL, or any model-provider SDK.
A matrix produces a CompatibilityResult.
The current compatibility statuses are:
| Status | Meaning |
|---|---|
COMPATIBLE |
All successful runs produced equivalent normalized output |
MISMATCH |
Runs completed successfully but produced different normalized output |
UNAVAILABLE |
At least one requested model could not be used |
EXECUTION_FAILURE |
At least one execution failed, timed out, or was cancelled |
Individual executions also retain their own RunStatus.
COMPLETED
FAILED
UNAVAILABLE
TIMED_OUT
CANCELLED
A behavioral mismatch is therefore different from an infrastructure or execution failure.
The core can be used without Spring or any real model provider.
import com.modelmatrix4j.core.execution.ModelMatrix;
import com.modelmatrix4j.core.model.ModelDescriptor;
import com.modelmatrix4j.core.model.ModelUnderTest;
import com.modelmatrix4j.core.result.CompatibilityResult;
import com.modelmatrix4j.core.scenario.Scenario;
import java.time.Duration;
import java.util.List;
public class Example {
public static void main(String[] args) {
Scenario scenario = new Scenario(
"greeting",
"Say hello"
);
ModelUnderTest first = new ModelUnderTest(
new ModelDescriptor("model-a"),
ignored -> "Hello world"
);
ModelUnderTest second = new ModelUnderTest(
new ModelDescriptor("model-b"),
ignored -> " Hello world "
);
ModelMatrix matrix = new ModelMatrix(
List.of(first, second),
1,
Duration.ofSeconds(1)
);
CompatibilityResult result = matrix.run(scenario);
System.out.println(result.status());
}
}The result is:
COMPATIBLE
because basic textual normalization treats the two outputs as equivalent.
Changing the second model:
ModelUnderTest second = new ModelUnderTest(
new ModelDescriptor("model-b"),
ignored -> "Goodbye world"
);produces:
MISMATCH
Both models executed successfully, but their behavior differed.
This distinction becomes increasingly important when future milestones add structured output, tool calls, RAG, and MCP observations.
ModelMatrix4J also provides a JUnit Jupiter integration layer.
A test supplies its scenario and model matrix through ModelMatrixSource.
import com.modelmatrix4j.core.model.ModelDescriptor;
import com.modelmatrix4j.core.model.ModelUnderTest;
import com.modelmatrix4j.core.result.CompatibilityResult;
import com.modelmatrix4j.core.result.CompatibilityStatus;
import com.modelmatrix4j.core.scenario.Scenario;
import com.modelmatrix4j.junit.ModelMatrixSource;
import com.modelmatrix4j.junit.ModelMatrixTest;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
class GreetingCompatibilityTest implements ModelMatrixSource {
@Override
public Scenario scenario() {
return new Scenario(
"greeting",
"Say hello"
);
}
@Override
public List<ModelUnderTest> models() {
return List.of(
new ModelUnderTest(
new ModelDescriptor("model-a"),
ignored -> "Hello world"
),
new ModelUnderTest(
new ModelDescriptor("model-b"),
ignored -> "Hello world"
)
);
}
@ModelMatrixTest
void modelsAreCompatible(CompatibilityResult result) {
assertEquals(
CompatibilityStatus.COMPATIBLE,
result.status()
);
}
}The JUnit extension:
- resolves the scenario and model configuration,
- executes the matrix through the core engine,
- creates a
CompatibilityResult, - injects the result into the test method.
JUnit does not contain provider-specific execution logic.
ModelAdapter is the primary extension point between ModelMatrix4J and model implementations.
@FunctionalInterface
public interface ModelAdapter {
String invoke(Scenario scenario) throws Exception;
}For deterministic tests, an adapter can simply be a lambda:
scenario -> "expected response"A future Spring AI integration can implement the same port:
Spring AI
|
v
SpringAiModelAdapter
|
v
ModelAdapter
|
v
ModelMatrix
Provider-specific behavior stays outside the core execution engine.
The project follows a small Ports and Adapters style architecture.
modelmatrix-junit
|
v
+-------------+
| ModelMatrix |
+------+------+
|
v
ModelAdapter
PORT
^
|
+---------+---------+
| |
Test adapters Future adapters
Spring AI / Ollama
Inside the core, execution and comparison remain separate responsibilities:
ModelMatrix
|
v
model execution
|
v
ExecutionOutcome
|
+----> CompatibilityEvaluator
|
v
safe RunResult
|
v
CompatibilityResult
Important architectural rules:
- core remains provider-neutral
- core remains framework-neutral
- JUnit depends on core, never the opposite
- provider adapters depend on the core port
- scenarios describe execution input, not assertions
- assertions do not invoke models
- no hidden model retries
- external services are opt-in
- public APIs remain intentionally small
RunResult represents a completed model execution.
It contains information such as:
- run identity
- scenario identity
- model descriptor
- repetition index
- terminal status
- normalized textual output
- duration
- bounded diagnostics
Comparison is performed on the meaningful normalized execution output before lossy redaction is applied to the public result.
This prevents secret redaction from incorrectly making different model outputs appear compatible.
Public result surfaces sanitize common sensitive forms and diagnostics are bounded.
Raw provider payloads are not part of the core result contract.
Pure Java execution and compatibility kernel.
Responsibilities include:
- scenarios
- model descriptors
- model adapter port
- model execution
- timeout and cancellation semantics
- repetitions
- normalized run results
- compatibility evaluation
It intentionally has no Spring or provider dependency.
JUnit Jupiter integration for ModelMatrix4J.
Responsibilities include:
@ModelMatrixTest- test lifecycle integration
- model/scenario resolution
CompatibilityResultparameter injection
It depends inward on modelmatrix-core.
Current core packages are organized by responsibility:
com.modelmatrix4j.core
├── scenario
│ └── Scenario
│
├── model
│ ├── ModelAdapter
│ ├── ModelDescriptor
│ ├── ModelUnderTest
│ └── ModelUnavailableException
│
├── execution
│ └── ModelMatrix
│
└── result
├── RunResult
├── RunStatus
├── CompatibilityResult
└── CompatibilityStatus
Execution implementation details remain package-private where possible.
Requirements:
- Java 25
- Maven Wrapper included in the repository
Run the complete default verification:
./mvnw -B verifyWindows:
mvnw.cmd -B verify
The default build is deterministic and does not require:
- Ollama
- Docker
- PostgreSQL
- cloud credentials
- paid model providers
- external runtime services
Normal Maven dependency resolution may still access configured artifact repositories.
ModelMatrix4J follows a few deliberate constraints.
The normal test suite must run without external model providers.
Real-model tests are explicit opt-in integrations.
No hidden retries
One requested execution means one model invocation.
Retries change observed model behavior and latency, so the core does not silently retry failed calls.
The core does not contain logic such as:
if (provider.equals("ollama")) {
...
}Providers integrate through adapters.
A scenario describes what should be executed.
It does not decide whether the result is correct.
A model returning different output is different from:
- a timeout
- a provider failure
- cancellation
- an unavailable model
These outcomes remain explicitly distinguishable.
Internal execution mechanisms are not automatically extension points.
An abstraction becomes public only when an actual consumer requires it.
Product boundaries, architecture, and implementation roadmap.
Java 25 multi-module Maven build, CI, dependency boundaries, and offline verification.
Current development stage.
Deterministic scenarios, model adapters, matrix execution, RunResult, CompatibilityResult, and JUnit Jupiter integration.
Next major milestone.
One Spring AI scenario will run against two explicitly configured local Ollama models through the same ModelMatrix4J execution path.
The integration will remain opt-in so the default build stays offline.
Normalized structured-output and tool-call compatibility testing.
Provider-neutral retrieval testing with optional PostgreSQL/pgvector integration.
MCP tool and resource behavior testing.
Stable reporting, serialization, CI artifacts, and richer comparison presentation.
Public API review, documentation, licensing, release metadata, compatibility policy, and release preparation.
See docs/ROADMAP.md for the complete milestone definitions.
ModelMatrix4J is still under active development.
The current implementation does not yet provide:
- Spring AI integration
- Ollama integration
- structured-output comparison
- Java tool-call comparison
- RAG support
- pgvector integration
- MCP support
- cloud-provider adapters
- stable reporting formats
- Maven Central releases
These capabilities belong to later milestones rather than being speculative parts of the current core.
More detailed project decisions are available in:
docs/PRODUCT_SPEC.md— product goals and boundariesdocs/ARCHITECTURE.md— architecture and dependency rulesdocs/ROADMAP.md— milestone plan
ModelMatrix4J is currently a development-stage project.
The immediate goal is to stabilize the deterministic core and JUnit execution path before implementing the first real Spring AI + Ollama vertical slice.
The project is not yet published as a stable library.