Skip to content

b11712

Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 17 Oct 14:01
b11712
10b8b75
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

tests: give the random model generator a per-tensor filler table (#30674)

  • tests: give the random model generator a per-tensor filler table

Muse Spark makes test-llama-context fail on state_load, cp_h and cp_d on about
six percent of the random seeds, with an NMSE of 1.0e-05 to 1.4e-05 against a
threshold of 1.0e-05. The sliding window is what exposes it: a restored state
keeps the window alone, so the attention of an SWA layer reduces over a
different cell layout and the continuation is not bit exact. Every SWA model
carries that, at 1e-08. Muse Spark carried it at 1e-05.

The generator is what turns 1e-08 into 1e-05. A norm preserving residual gate
holds alpha^2 + beta^2 == 1 and the conversion resolves both weights from one
raw weight, while the generator draws every tensor on its own. The pair then
scaled the stream and the branch independently, the stream lost an arbitrary
factor at each layer, and the model became an amplifier. The weights the test
ran on were not a Muse Spark.

Per-tensor rules lived inside the write path, where a flag picked the values of
a tensor inside each of the two branches that store them, so a new rule meant a
new flag threaded through both. The file asks to be rid of that in a TODO of its
own. Give each rule a filler, pick one from a table keyed on a substring of the
tensor name, and keep a single write path behind it. Mamba moves into the table
and Muse Spark joins it: the gate filler draws the raw weight once per block and
resolves alpha and beta the way the conversion does.

Over a thousand seeds on a CPU build the worst case drops from 1.36e-05 to
2.48e-06 and the median from 4.10e-06 to 5.48e-07, with no seed above the
threshold. The other 131 models come out identical to the bit, since no other
architecture carries these tensors. ctest is green on a CPU build and on a CUDA
build.

  • tests: draw every tensor filler from fill_normal

The fillers repeated the generator and the distribution each time. The
normal draw now takes its key and the other fillers transform its
values, so the generator lives in one place. The generated models are
identical to the bit.

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Assets 38
Loading