Validation

Validation

This package is a port, so the question that matters is whether it reproduces the MATLAB reference. Agreement is established solver by solver against golden fixtures and checked by the test suite on every change.

How the fixtures are made

validation/fixtures/ holds outputs produced by running every solver in MATLAB R2026a through validation/generate_fixtures.m, at fixed configurations that include the MATLAB repository's baseline examples. The test suite re-runs each configuration in Python and asserts agreement against the stored numbers.

python -m pytest                        # fast suite, about a minute
python -m pytest -m "slow or not slow"  # including the 36,000-pulse scanning baseline
python validation/benchmark.py          # wall-time comparison against MATLAB

What agreement looks like

The solvers fall into two groups, and they are held to different standards for a reason.

Step-identical kernels. The surface point, radial profile, and scanning beam solvers are hand-rolled ports whose arithmetic follows the MATLAB line by line. They agree to round-off, with relative errors near 1e-15 and a worst case of 1.6e-10 K across a 36,000-pulse scan. Any drift here is a bug.

Stiff integrator solvers. The depth profile, single pulse, and inversion quantifier solvers integrate the coupled equations with a stiff solver. MATLAB uses ode15s and Python uses SciPy BDF, the same solver family at the same tolerances, so they agree at integrator-tolerance level rather than bit level. Relative errors are near 1e-4 on peak temperatures.

This split is why the repository treats solver arithmetic as frozen. A refactor is expected to be a provable code move, and the fixture suite is what proves it.

One deliberate deviation: the energy-conserving deposit

Since 0.2.0 the pulse deposit in the surface point, radial profile, and scanning beam solvers conserves the absorbed energy on any grid, and this is the one place the Python package deliberately deviates from the MATLAB reference by default.

The reference deposit raises the surface node to the post-pulse equilibrium temperature and lets the rise decay over Leff. On a depth grid that resolves Leff this is nearly exact. On a coarser grid the surface control volume absorbs roughly (dz/2)/Leff times the intended energy on every pulse. At the default grid spacing of 500 nm and Leff of 100 nm that inflates the deposited energy by about 2.5x per pulse, and for a 23 nm layer by about 11x, which compounds into large errors in heat accumulation. The energy balance printed in every report shows the effect directly: the bookkeeping mismatch is a few hundred percent under the reference deposit and about 0.01 percent under the corrected one.

Setting legacyDeposit: True in any affected solver's config reproduces the MATLAB behavior exactly. The golden fixture tests run in that mode, so the agreement figures above continue to hold and continue to prove the port line-faithful. The corrected default is pinned by its own tests, which assert that the deposited energy equals the layer energy to one part in a trillion on both fine and coarse grids.

Performance against the reference

Warm-run wall times on the same machine. MATLAB times were recorded when the fixtures were generated, with details in validation/fixtures/manifest.json.

Case	MATLAB	Python	Speedup
surface point baseline, 50 pulses	7.11 s	0.07 s	108x
single pulse baseline	1.36 s	0.04 s	32x
radial profile baseline, 100 pulses	1.41 s	0.10 s	15x
depth profile baseline, 100 pulses	17.60 s	12.9 s	1.4x
scanning, 3,600 pulses	29.96 s	22.9 s	1.3x
scanning baseline, 36,000 pulses	927.2 s	543.6 s	1.7x
inversion baseline, 20 pulses	4.35 s	6.1 s	0.7x

The largest gains are in the solvers built on the Numba kernels. The depth-solver family currently mirrors the MATLAB integration windows exactly, which is why its margin is smaller. The source term is identically zero beyond ten pulse widths from each pulse centre, so relaxing the step cap outside that window is a planned optimization.

The first call in a fresh environment pays a one-off Numba compilation cost. Compiled kernels are cached on disk, so the numbers above describe warm runs, which is what a real study sees.

Reporting a discrepancy

If you find a case where the two implementations disagree beyond these tolerances, that is worth an issue. Include the config, both outputs, and the versions of each package. A configuration that reproduces it inside the fixture harness is the most useful form.