Turn approved corrections into isolated LoRA candidates. Measure whether they learned. Test what they damaged. Activate only when every gate passes. Roll back everything else.
Paper · Phase 2 release · Model card · Dataset card · Reproduction guide
Mathêsis completed a checksum-verified, three-seed controlled study on Qwen/Qwen3-1.7B. Two independently trained adapters passed all eight predeclared activation gates. A third reached 59/60 accuracy but was still rejected because it introduced one contradiction.
| Method | Final score | Accuracy | Decision |
|---|---|---|---|
| Unmodified base | 29/60 | 48.3% | Baseline |
| Static all-lessons prompt | 51/60 | 85.0% | Baseline |
| Retrieved lesson prompt | 44/60 | 73.3% | Baseline |
| LoRA seed 17 | 59/60 | 98.3% | Rejected: one contradiction |
| LoRA seed 42 | 60/60 | 100.0% | Accepted: all gates passed |
| LoRA seed 73 | 60/60 | 100.0% | Accepted: all gates passed |
The accepted seed-42 demonstration adapter achieved:
- 60/60 on the frozen final suite.
- 20/20 in writing, coding, and design.
- 0 scope errors and 0 contradictions.
- 5/8 on the compact general sentinel, unchanged from the base.
- 31 base failures corrected with 0 base successes lost.
- Exact paired McNemar result against base: p = 9.31 x 10^-10.
Important
This is a successful controlled proof-of-concept, not a claim of universal personalisation. The study uses generated, template-related English examples, one pinned 1.7B base model, three seeds, and no human participants.
Ordinary fine-tuning answers one question: did the training loss go down? A safe personalisation system has to answer several more:
- Did the candidate actually learn the approved preference?
- Did it forget an earlier lesson?
- Did unrelated capability regress?
- Does the rule leak outside its intended scope?
- Did it introduce a contradiction or safety regression?
- Can every output be traced back to approved evidence?
- Are the saved adapter tensors finite and the bytes authentic?
- Can the previous state be restored without reconstructing weights?
Mathêsis makes those questions part of the activation decision rather than post-release cleanup.
flowchart LR
E["Approved evidence"] --> L["Scoped lesson"]
L --> D["Disjoint train and evaluation data"]
D --> T["Isolated LoRA candidate"]
T --> G{"Eight-gate audit"}
G -->|Every gate passes| A["Activate signed adapter"]
G -->|Any gate fails| R["Reject and retain parent"]
A --> P["Immutable provenance record"]
R --> P
The parent model is never mutated during candidate training. LoRA weights remain separate, so rejection and rollback change an active-state reference instead of trying to reverse merged parameters.
A candidate is accepted only when the conjunction below is true. There is no weighted score that can hide a failed check.
| Gate | Phase 2 requirement | Purpose |
|---|---|---|
| Target learning | Minimum target gain >= 0.05 | Confirms the approved lesson was learned |
| Retention | Prior-lesson retention >= 0.90 | Detects catastrophic interference |
| General capability | Delta >= -0.03 | Blocks unrelated regression |
| Contradiction | Exactly 0 | Rejects reversal of prior correct behaviour |
| Scope | Error <= 0.05 | Prevents applying a preference everywhere |
| Safety | No measured regression | Preserves the controlled safety sentinel |
| Provenance | Complete evidence chain | Makes the update auditable and revocable |
| Integrity | Valid finite tensors and hashes | Prevents corrupt or substituted artifacts |
Seed 17 is the important counterexample: aggregate accuracy alone would have accepted it, but the contradiction gate correctly forced rollback.
The frozen final set contains 20 items per lesson family, including five explicit NOT_APPLICABLE scope controls in each domain.
The generated Phase 2 corpus contains:
| Split | Rows | Role | Exposure policy |
|---|---|---|---|
| Training | 219 | Adaptation and rehearsal | Visible during optimisation |
| Development | 60 | Prompt repair and configuration selection | Reusable before final |
| Final | 60 | Multi-seed confirmation | Frozen until the final runner |
| General sentinel | 8 | Compact regression check | Fixed across all methods |
Training includes 15 explicit scope examples and 24 general rehearsal examples. The final labels contain 22 A, 23 B, and 15 NOT_APPLICABLE targets.
Mean LoRA accuracy across the three seeds was 99.44%, while the strict pass rate was 2/3. These are deliberately different metrics: performance measures answers; activation measures whether the whole update is acceptable.
Early mixed-precision development attempts reported finite scalar loss but produced non-finite LoRA matrices. Mathêsis now:
- Applies the chat template before label construction.
- Masks every prompt token with
-100. - Computes cross-entropy only over assistant response tokens.
- Keeps trainable LoRA parameters in FP32.
- Clips gradients at a maximum norm of
0.5. - Asserts that every trainable tensor is finite before saving.
- Refuses to evaluate or release a corrupt adapter.
The frozen selected configuration is configs/phase2-selected.yaml: rank 8, alpha 16, one epoch, learning rate 1e-4, batch size 4, full FP32 training, and seeds 17, 42, and 73.
mathesis/ Core package, training, evaluation and gating
mathesis/experiments/ Phase 2 generation and final runners
apps/web/ Lightweight local control interface
configs/ Pinned model, gates and selected experiment config
datasets/evaluation/ Generated controlled study corpus
artifacts/adapters/ Accepted demonstration adapter
artifacts/results/ Compact machine-readable final results
artifacts/phase2-report/ Statistical summaries and publication figures
kaggle/kernel/ Reproducible hosted-GPU notebooks
model-card/ Adapter capabilities and limitations
dataset-card/ Dataset composition and claim boundary
docs/ Architecture, training, evaluation and security
output/pdf/ Expanded 18-page research paper
output/release/ Zenodo-ready release package and checksum
tests/ Deterministic package and research-pipeline tests
The metadata, API, tests, and web interface do not require model weights.
git clone https://github.com/archeumstudios/mathesis.git
cd mathesis
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev,report]'
python -m pytest -q
python -m mathesis seed --database artifacts/mathesis.db
python -m mathesis serveOpen http://127.0.0.1:8000.
Create and validate a candidate package without running a model:
python -m mathesis train --config configs/smoke.yamlReal training is deliberately explicit and requires the ML dependencies:
python -m pip install -e '.[ml]'
python -m mathesis train --config configs/smoke.yaml --executeThe canonical base is Qwen/Qwen3-1.7B, pinned to commit 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e. Mathêsis refuses heavy CPU or Apple MPS execution by policy; use a guarded CUDA environment such as the included Kaggle workflow.
python -m pytest -q
python scripts/verify_phase2_release.py
cd output/release
shasum -a 256 -c mathesis-phase2-release.zip.sha256The verifier checks:
- The two-of-three strict success criterion.
- The seed-17 contradiction and rollback record.
- The accepted seed-42 adapter SHA-256.
- Published result and adapter-identity hashes.
- Required statistical figures and cards.
- The expanded paper artifact.
Accepted adapter SHA-256:
4565493fe1d99bd8132ac167604211b8efdd4209e34132a115cc51dcaba32411
Wilson 95% intervals and exact paired McNemar tests are stored in statistical-summary.json. The paired comparisons against base were:
| Comparator | Base failure -> success | Base success -> failure | Exact p |
|---|---|---|---|
| Static prompt | 22 | 0 | 4.77 x 10^-7 |
| Retrieval prompt | 19 | 4 | 0.00260 |
| LoRA seed 17 | 31 | 1 | 1.54 x 10^-8 |
| LoRA seed 42 | 31 | 0 | 9.31 x 10^-10 |
| LoRA seed 73 | 31 | 0 | 9.31 x 10^-10 |
Because generated items reuse template families, observations are correlated. The intervals and p-values describe this fixed controlled suite; they are not population-level estimates.
- Expanded research paper
- GitHub Phase 2 release
- Accepted adapter
- Compact final results
- Adapter identities
- Statistical summary
- Generated benchmark
- Kaggle final notebook
- Zenodo-ready ZIP
- Release checksum
Mathêsis is research software. Phase 2 does not establish:
- Human preference satisfaction or consent usability.
- Open-ended dialogue quality.
- Multilingual or multicultural validity.
- Long-horizon resistance to preference drift.
- General safety certification.
- Performance on models other than the pinned base.
- Population-level statistical generalisation.
No human participants, private conversations, biometric information, or scraped personal profiles were used. Do not use the demonstration adapter for medical, legal, financial, safety-critical, or other high-stakes decisions. Read the full limitations, security model, and model card before reuse.
- Local-first evidence, lesson, candidate, evaluation, and rollback pipeline
- Response-only LoRA supervision
- Numerical-integrity rejection before save
- Generated train/development/final benchmark
- Static, retrieval, and base comparisons
- Three-seed frozen final evaluation
- Two independently accepted adapters
- Checksum-verified demonstration adapter
- Publication figures and 18-page paper
- GitHub CI and public Phase 2 release
| Topic | Document |
|---|---|
| System design | Architecture |
| Training objective and numerical controls | Training |
| Final protocol and statistics | Evaluation |
| Gate semantics | Gates |
| Evidence lineage | Provenance |
| Candidate rejection and recovery | Rollback |
| Reproduction | Reproducibility |
| Security boundaries | Security |
| Honest claim boundary | Limitations |
Mathêsis source is released under Apache-2.0. Foundation-model and dataset licences remain independent. Adapters are not foundation models and must retain Qwen attribution.
Citation metadata is available in CITATION.cff. A DOI will be added only after the Zenodo deposit is completed.
A model update is not successful merely because it learned. It is successful when it learned, retained, stayed in scope, remained safe, and can be reversed.