MARULHO

MARULHO is a local research system for building a continual language model whose tokenizer, learned weights, memory, learning rules, generation, checkpoints, and evaluation are owned by this repository. The research target is a model that can learn from an ongoing stream, recall useful past experience under bounded active compute, and remain rollbackable while it changes.

MARULHO is not currently an AGI or a frontier model. Its strongest research checkpoint produces coherent multi-sentence English and has passed one narrow continual-learning test: it learned held-out synthetic relations while retaining its general-language loss. It remains repetitive, semantically unreliable, and uneven across relation types. There is still no admitted long-term memory read interface or generally capable continual model.

Current architecture

There are eight different levels of truth:

1. Installed runtime: MarulhoBrain owns a 21M-parameter decoder-only causal Transformer and its checkpoint-owned BPE tokenizer. This remains the stable runtime baseline.
2. Retained sparse research evidence: V11 is an uninstalled 36.18M-parameter causal Transformer whose replaced feed-forward block contains deterministic hashed singleton micro-experts. Its strict checkpoint has trained for 1.0B update tokens.
3. Memory evidence boundary: V25 proves that a bounded selector plus one exact archived episode improves disjoint causal likelihood. Raw prepending fails anchored generation, while V26 and V27 gated readers fail even with oracle evidence. No memory model, checkpoint, or runtime integration is currently admitted.
4. Active optimizer experiment: V29 keeps the exact 20.976M Transformer fixed and compares AdamW with matrix-orthogonalized Muon at two learning rates. At 16.78M tokens, Muon 1e-3 beats same-rate AdamW by 0.1645 heldout loss and 12.11 points of exact free generation. This admits checkpoint reproduction, which passes bit-exact reload. Unseen prose remains 0/8 and semantically unstable, so Muon is retained as a training improvement rather than a quality-qualified model or runtime optimizer.
5. Strongest general base candidate: V30 selects context 72 after removing synthetic relation updates. V31 then trains the same fresh 20.976M model on 67.11M non-repeated tokens and improves common loss/perplexity from 4.0093/55.11 to 3.6291/37.68. FineWeb-Edu/Cosmopedia unseen loss also falls to 4.2053/3.4896. Text is more grammatical, but remains 0/8 on anchored cases, generic, and factually unstable, so the checkpoint is retained for a larger data curve rather than installed as a qualified base. V32's fresh 201.32M-token point improves loss only to 3.4983, a +0.1308 gain that misses its frozen +0.20 gate. V34 then expands the same general recipe to a fresh 100.68M-parameter Transformer. At the same 67.11M-token budget it reaches loss/perplexity 3.3902/29.67 versus V31's 3.6291/37.68, passing its +0.20 gate with exact checkpoint reload. Unseen prose is clearly more grammatical and multi-sentence, but remains generic and 0/8 source-anchored; the checkpoint is retained for a new-data continuation, not installed. V35's continuation reaches diagnostic loss 3.1654, but its manifest schedules 58,255 of 58,257 prepared unique batches. The full-coverage gate correctly marks it invalid and no checkpoint survives. V35R reruns from V34 with exactly the two missing batches and no quality-gate change. The valid rerun reaches loss/perplexity 3.1649/23.69 after 201.34M cumulative tokens and strict-reloads its checkpoint. Unseen FineWeb and Cosmopedia loss improve to 3.8020 and 2.9282; controlled generations are now coherent multi-sentence paragraphs. Exact source anchoring remains 0/8, so this qualifies a base-language research checkpoint, not grounded memory or runtime installation.
6. Architecture search: V28's particle field and V33's local-attention plus editable matrix state are retired. V33 was a valid exact-parameter test and nearly tied loss (4.0056 versus 4.0082), but its 0.0025 gain missed the 0.02 gate while retaining only 75.5% throughput and using 1.41 times peak memory. No checkpoint or event-control phase survives. No biological metaphor or many-small-units design is a requirement; every mechanism must earn its place.
7. Strongest continual research checkpoint: V39 starts from V35R and mixes new relation examples with fresh general-language replay. A normalized 4x answer-span objective reaches 50.00% strict free recall and 98.44% candidate accuracy while improving heldout general loss from 3.1649 to 3.1134. The 100.68M-parameter checkpoint reloads exactly after 218.11M cumulative tokens. This is a narrow learning-without-forgetting result, not general reasoning: property and event-order relations are strong, while container and ownership remain weak.
8. Qualified sustained research runtime: V40 loads that exact V39 artifact and generates 256 independent 2,048-token streams on the RTX 3060. All 524,288 tokens complete in 74.84 seconds at 7,005 tokens/s with 3.17 GB peak allocation and unchanged model tensors. Execution hooks observe every one of the 100.68M parameters, so the current path is measured dense, not sparse.

flowchart LR
    Stream["Causal text stream"] --> Tok["Checkpoint-owned BPE"]
    Tok --> Local["Bounded local cortex"]
    Tok --> Archive["Exact episodic archive<br/>tokens + provenance + compact keys"]
    Prefix["Visible current prefix"] --> Select["Bounded evidence selector"]
    Archive --> Select
    Select --> Evidence["One older exact span"]
    Evidence --> Boundary["No admitted read interface<br/>after V26/V27"]
    Local --> Output["MARULHO-owned next-token generation"]
    Local --> Save["Atomic checkpoint and rollback"]
    Archive --> Save
    Brain["MarulhoBrain"] --> Local
    Service["Thin /brain service"] --> Brain

Loading

The division of labor is deliberate:

• the cortex learns language and reasons over the small amount of evidence that is active now;
• no current reader connects the archive to the cortex; both tested gated cross-attention placements are retired;
• the archive preserves potentially important experience without forcing every detail through a fixed-size recurrent state;
• keys and indexes may be compressed, but valuable episode content stays exact until evidence supports a safe consolidation rule;
• selection limits active context instead of pretending that an ever-growing prompt is free.

The validated selector is currently lexical TF-IDF, not a learned semantic memory and not the intended final answer. It is a causal instrument that has replicated a likelihood win. That signal is retained as evidence, not as an active architecture; the next work returns to base-language computation before another memory interface is justified.

MARULHO is not using an SNN, GRU, cortical-column simulation, Hopfield network, or reservoir as its active language core. Those ideas remain available only when they express a measurable computational role and can beat matched controls.

What the evidence supports

Result	Evidence	Decision
V11 base cortex	36.18M parameters; heldout loss 3.0805 after 1.0B update tokens; about 121.9k training tokens/s and 1.97 GB peak allocation on the RTX 3060	Retain as the strongest sparse research base, but do not call it language-qualified
V19/V19b latent memory	Recurrent and partitioned banks reach 30.1% and 31.4% paired source-following and remain more than 16 points behind exact history	Retire the latent memory-token interface
V20 addressing audit	Lexical top-one fails its gate; lexical top-two includes the required episode in 98.83% of cases while reading half of the available history	Admit a separate top-two language screen
V21 language screen	Lexical top-two reaches 51.6% free exact and 52.0% paired source-following versus all-history at 39.5% and 38.0%; it reads 96 instead of 192 source tokens	Advance exact episodic retrieval to causal document streams
V22 document audit	Oracle-one improves loss by 0.0341, but lexical-one's 75.0% retrieval recall yields only +0.0017 and top-two hurts; wrong episodes are about three times as costly as correct episodes are useful	Replace unconditional top-k with a calibration-frozen retrieve-or-abstain gate
V22b abstention audit	The frozen gate transfers at 97.84% precision and gains 0.0356 loss, but always-on lexical gains 0.0388 on the same cases	Retire detached correctness gating and co-train the cortex to interpret selected evidence
V23 joint document screen	Oracle and true-vs-wrong tests prove learned source use, but lexical's +0.0192 interval crosses zero and general loss regresses +0.1200/+0.1346	Reject the 75/25 top-one curriculum; run one balanced top-two falsifier
V24 balanced top-two	Replay restores retention, but top-two is 0.0064 worse than top-one. The lexical-one control gains a significant +0.0255 while retaining general loss	Retire top-two and replicate top-one against balanced random-one
V25 top-one replication	Lexical memory gains +0.0430 over off, beats random, improves both corpora, and retains general loss; all 8 anchored continuations still fail	Preserve the likelihood signal, retire raw concatenation, and build a separate evidence reader
V26 final-layer reader	All reader/cortex tensors train, but oracle gain is only +0.00010 and the gate remains near 0.119	Retire final-layer injection; test interleaved evidence before later cortex layers
V27 interleaved reader	Raw context gains +0.0426, but lexical and oracle readers are both about 0.0392 worse than gate-zero; all tensors train and both gates remain near 0.119	Retire cross-attention document memory and return to the base-language architecture

V21 also keeps both general-language holdouts within the preregistered 0.10 loss regression bound and uses about 0.90 GiB peak allocation versus all-history's 1.03 GiB. Its elapsed training time is tied with the controls, so MARULHO makes no speed claim from this experiment.

The important V21 result is not “TF-IDF solved memory.” It is that selected exact evidence can outperform both lossy learned compression and indiscriminate full history. That is the first memory architecture admitted in the current research iteration.

What remains unproved

The selected direction still has to show all of the following:

• an evidence interface that converts the retained V25 likelihood signal into anchored free generation;
• lower heldout continuation loss and better source-anchored free generation at the same time;
• a semantic or learned key that transfers beyond relation templates;
• strict checkpoint fidelity for a future archive, index, provenance, optimizer, and rollback state;
• sequential-domain learning that generalizes beyond the narrow synthetic V39 relation family;
• a conditional memory or compute path that beats the measured 100%-dense V39 runtime on quality per local compute.

Until those are demonstrated, this is an architecture hypothesis with one positive controlled result—not a replacement for frontier Transformers.

Current research program

1. Treat V33 as closed evidence: replacing half the token Transformer with an editable matrix state did not buy enough language quality, speed, or memory.
2. Retain V34 as the parent of the first local 100.68M trajectory: its +0.2389 heldout gain justified continuation, but V35R supersedes it as the live research checkpoint.
3. Treat V35's 3.1654 loss as diagnostic only: two prepared batches were absent from its frozen schedule, so the evidence is invalid and no checkpoint exists. V35R reruns from V34 on all 58,257 hash-pinned batches (134,224,128 new; 201,335,040 cumulative tokens), passes the unchanged +0.15 gate by 0.0753, and produces coherent unseen paragraphs. Preserve the 0/8 anchoring failure as the boundary between base-language qualification and grounded memory.
4. Keep dynamic byte hierarchies as a later scale-aware direction. Published H-Net evidence begins around 680M parameters and tens of billions of bytes, so a 21M imitation is not the next credible 3060 experiment.
5. Use V36's quality-safe RTX 3060 recipe for the next durable run. On 2.36M identical ordered tokens, physical batch 256 with whole-QKV Muon at 3e-4 reaches loss 3.1423 at 25.07k tokens/s versus batch 32's 3.2455 at 11.08k: 2.262 times the throughput and better loss. Higher learning rates are worse. Per-head Muon is useful at batch 32 but does not justify replacing whole-QKV Muon at the advancing batch size.
6. Reopen exact episodic memory, online learning, consolidation, forgetting, active compute, and the sustained runtime ladder from V35R. V37's full-width depth assembly is retired after exceeding a fixed one-hour run and reaching 11.74/12.29 GiB observed device allocation without terminal quality evidence. A successor may test fused low-rank depth channels, but may not restore all-depth activation retention. Do not call the still-unanchored generator runtime-qualified.
7. Treat V38 as a near-positive continual result, not a promoted checkpoint. The 50/50 replay arm reaches 100% relation recognition, 46.88% strict free answers, and improves old-language loss to 3.1124, but misses the 50% free gate. V39 must improve exact answer formation under the same replay/compute budget; do not add capacity, weaken evaluation, or repeat replay ratios.
8. Retain V39 as the first continual-qualified 100M checkpoint. Its 4x answer-emphasis arm reaches exactly 50% strict free relations, 98.44% ranked, improves general loss to 3.1134, and reloads exactly after 218.11M cumulative tokens. Preserve the uneven boundary: ownership is 4/64 and container 15/64, so this is not yet general binding competence.
9. Retain V40 as the same-checkpoint runtime qualification. Its 256 independently prompted CUDA streams each produce 2,048 consecutive tokens, totaling 524,288 in 74.84 seconds at 7,005 tokens/s. Model state is immutable and bounded; observed parameter coverage is 100%, so there is no sparse-compute win yet.
10. Retire V41 hidden-state episodic memory. Its 65,536 training-only keys move strict free accuracy only from 50.00% to 51.56%, entirely through property; ownership stays 6.25% and container falls to 20.31%. Shuffled values prove the intervention is causal, while 740.95M dense key comparisons and no binding gain reject this interface. No checkpoint or code survives.
11. Test V42 role-contrastive continual learning from V39. Keep replay and 4x answer loss fixed, but explicitly suppress wrong entity/container/color/ event fillers at answer positions. A short three-arm pilot must show a material weak-binding gain before any full control/candidate confirmation.

A negative result is allowed to kill or redesign the archive path. Breaking changes are expected; failed live machinery is deleted after its evidence is retained.

Scientific boundaries

• external_llm_used=false: no downloaded model owns language generation.
• MarulhoBrain owns cognition; service/status code only exposes it.
• Labels, target slots, oracle routes, and future tokens are metrics-only unless a training objective explicitly allows them.
• Every candidate faces matched local, random, recency, full-history, or dense controls appropriate to its claim.
• Throughput, one benchmark row, and readable samples do not substitute for unseen quality.
• Durable mutation must be checkpointed, hashable, reloadable, and rollbackable.
• CUDA/Triton and sparsity claims describe observed execution, not architecture diagrams.

Repository map

• CONTEXT.md — Runtime Truth, current decisions, and evidence pointers.
• RESEARCH.md — research synthesis, competing hypotheses, and retired ideas.
• IDEAS.md — creative architecture notebook and explicit falsifiers.
• src/marulho/brain/ — runtime ownership and installed generation path.
• src/marulho/training/ — tokenizer, causal language model, training, and checkpoint machinery.
• src/marulho/evaluation/ — matched experiments and promotion boundaries.
• src/marulho/service/ — thin API projection over MarulhoBrain.
• reports/language_scaling/ — local evidence artifacts; large reports and checkpoints are intentionally not versioned.

Read CONTEXT.md before changing the system, then read the nearest package README for the machinery being changed.

Development

python -m pip install -e ".[dev,cuda]"
python -m pytest -q
python -m compileall -q src tests

The focused tests for the selected V20/V21 branch are:

python -m pytest -q `
  tests/test_language_hashed_micro_experts.py `
  tests/test_language_exact_episodic_retrieval_audit.py `
  tests/test_language_exact_episodic_retrieval_screen.py