laya-apple 1.7.0

laya-apple 1.7.0: an opt-in W8 ANE artifact (experimental), serve reports Jev's confidence

A minor release. It adds an 8-bit weight-palettized (W8) ANE artifact for laya-typed-decisions
L64, which you choose explicitly; FP16 stays the default and nothing changes unless you ask for
it. laya-apple serve adds a field so Jev-compatible clients can read the confidence Jev would
report, and a checkpoint trained for a non-sequential option layout now fails loudly. The parity
tolerances, the routing rule and the prebuilt-artifact pin are unchanged.

pip install -U 'laya-apple[ane]'

Opt-in W8 ANE artifact (experimental)

Added in #176, after the research in
#174 and
#175.

laya-apple artifacts build laya-typed-decisions --length 64 --quantize w8

laya = Laya.from_pretrained("laya-typed-decisions", device="auto", ane_quantization="w8")

The artifact is built beside the FP16 one and runs the unchanged parity gate (probability error
<= 0.02, 0 hard mismatches), the 100% Neural Engine placement check, the runtime probe and the
hash check. L64 loads it; the other buckets keep their FP16 artifacts. It works with workers and
adaptive execution. Scope, how it fails and the full limitations are in
docs/guide.md.

Measured on macOS 27.0, Apple M4 Max, coremltools 9.0
(research/ane-w8-l64-gate/,
research/ane-w8-l64-s5-open-loop/):

	FP16	W8
Single L64 request, end-to-end P50	8.2 ms	5.4 ms
Forward predict P50	1.0	0.651 (pooled, 3 processes)
Throughput, single client	1.0	1.52x
Weights	738.9 MB	369.7 MB
Max probability error (gate 0.02)	0.0046	0.0170
Hard mismatches	0	0

Why it is experimental and off by default

The preregistered production gate for W8 was not passed; its recorded verdict is Research
Only / No Ship, and W8 ships here only as this explicit opt-in.

• GPU LLM interference. With a continuous local LLM on the GPU and a closed-loop client, the
  LLM generated about 4% fewer tokens/s next to W8 (ratio 0.953 to 0.962 in 12 of 12 cells,
  bound 0.97). An open-loop run at equal offered load found no W8-specific slowdown (1.012 to
  1.013) and a 25% higher offered load cost about 3% in both arms, so the closed-loop drop most
  likely comes from the extra GPU work the faster ANE path completes. The closed-loop cost is
  still real for a client that sends as fast as it is served.
• No real L64 traffic data. No recorded request corpus exists, so how often real requests
  fall in the L64 bucket is unknown. This is why W8 is not the default.
• Adaptive breaker. No real GPU-busy episode tripped it with W8, so its trip behaviour is
  untested.
• Two near-tie flips against the reference (listed in the research record); one machine,
  thread-level concurrency.

Limits of the option

Only laya-typed-decisions L64 on Apple M4 Max with macOS 27 (other models and machines raise),
not with device="gpu", ane_startup="background" or from_pretrained("auto"), and not
supported by artifacts export, import, fetch or laya-apple serve. The routing thresholds
are the FP16 ones. A missing, stale or failing W8 artifact raises; it never falls back to FP16 or
the GPU.

x_jev_confidence in serve

Added in #170
(#168). Each choice and score
answer carries x_jev_confidence: the confidence Jev would report for the same
probabilities, as upstream Laya #1066 defines it. confidence is unchanged and the two are
never aliased. A noul answer, or one without usable probabilities, gets no field. See
docs/serve.md.

Non-sequential option_layout is refused

Fixed in #171
(#167). Upstream stores the layout as a
top-level option_layout key of rl_agent_config.json. laya-apple validates only sequential
semantics, so running a checkpoint trained for another layout gave wrong decisions, and an
artifact built from it looked valid. Loading such a checkpoint, and building or checking its
artifact, now raises LayaAppleError. A checkpoint without the key is sequential, as before.

Limitations

• The serve and option_layout changes were not benchmarked; neither touches the model forward
  pass. The seven Jev clients tested for serve were not re-run against the extra
  x_jev_confidence field (docs/serve.md).
• The W8 numbers are one machine (Apple M4 Max, macOS 27.0). The release benchmarks of FP16 are
  not re-run for this release.
• The 1.6.3
  limitations not repeated here still apply.

Upgrading

No code change needed and FP16 behaves as before. A checkpoint with a non-sequential
option_layout now raises. To try W8, build the artifact locally as above; it is not part of the
prebuilt-artifact pin.

laya-apple 1.6.3

laya-apple 1.6.3: a first artifacts fetch compiles once, not twice

A patch release. A first artifacts fetch or artifacts import of a bucket now compiles the
model for the Neural Engine once instead of twice, and the artifacts it registers no longer
carry a stale BUILDING.json. artifacts prune and build swaps also survive interruptions.
predict, the parity tolerances, the routing rule and the prebuilt-artifact pin are
unchanged.

pip install -U 'laya-apple[ane]'

One compile per import

Core ML compiles a model for the Neural Engine per path. Until now an import ran the compute
plan, the parity gate and the placement probe on a staged copy, moved the model to its
registered path, and then loaded and probed it again there. A first install paid for two
compiles (#164).

Now the staged copy moves to the registered path first, with its manifest withheld. The
checks run there, and the manifest is published with one atomic rename only after they all
pass. Until then the runtime sees no artifact at that path, and a failed or killed import
registers nothing. The checks, the hooks and PROBE_MAX_RATIO are the same. A --force
replacement still validates in staging first, so it still compiles twice.

Measured on one Mac Studio M4 Max (macOS 27.0, coremltools 9.0) with laya-typed-decisions
L128, each import into an empty cache, three imports per side
(results.md):

	Before	After
Import time (fetch's checks)	145.7 / 146.1 / 145.9 s	75.4 / 75.6 / 75.3 s
New Core ML compile bundles	6	3
Compile cache added	2.98 GB	1.49 GB
First load by another process, afterwards	0.34 s (median)	0.33 s

Before is main one commit after 1.6.2. After is the research prototype of this change; the
shipped artifacts import was measured separately (below). With the method of
#121, three artifacts import runs of
the same bucket went from 151.1, 154.0 and 149.2 s to 76.4, 91.4 and 78.9 s. The 91.4 s import
is not explained.

BUILDING.json stays out of registered artifacts

Fixed in #161
(#160).

• artifacts import, artifacts fetch and artifacts build wrote an in-progress marker into
  their staging directory and then renamed the whole directory into place, so every artifact
  they registered kept it. The marker is now removed just before the rename.
• artifacts prune reads that marker before it checks a staging directory's age, and checks
  both again just before it deletes, so a directory whose import has just finished or is still
  running is not deleted.
• A forced artifacts build whose move into place fails now puts the previous artifact back.
  It used to leave it at .old-<pid> with nothing registered.

artifacts prune and build swaps survive interruptions

Fixed in #166. artifacts prune claims a
path with one rename before it deletes anything, so an import that lands in between is not
deleted, and an interrupt right after a build's move into place keeps the new artifact
registered. A published copy stranded by a crashed prune is reported with an mv hint, never
deleted.

Limitations

• One machine, one model, three imports per side. Only macOS 27.0 (26A428) was measured; the
  macOS 26 import path was not. No second machine has checked this.
• Compiles are counted from Core ML's e5rt cache directories, a proxy, not an API. The
  artifacts import re-run was not preregistered, and its 91.4 s import is unexplained.
• It does not remove the compile, only the second one. The import still pays it, and the
  first runtime load of an artifact at a new location still compiles: about 210 s for
  laya-typed-decisions (64/96/128) on macOS 27.0
  (results.md).
  ane_startup="background" serves on MLX meanwhile.
• Core ML's compile cache still grows without eviction, and Core ML keeps it per process
  name, so a process with a different name may compile again. laya-apple never evicts it,
  and artifacts prune does not touch it.
• Artifacts registered before 1.6.3 keep their copy of BUILDING.json. Nothing reads it, and
  it goes away when the artifact is replaced.
• The 1.6.2
  limitations not repeated here still apply.

Upgrading

• No code change needed, and nothing to rebuild or re-fetch.
• The next artifacts import or artifacts fetch of a bucket removes an unpublished
  directory that a killed import left behind. artifacts prune lists one once its import has
  exited and it is older than an hour.

laya-apple 1.6.2

laya-apple 1.6.2: prebuilt ANE artifacts for M4 Max on macOS 27

On an Apple M4 Max with macOS 27 and coremltools 9.0, you can now fetch the ANE artifacts
instead of building them:

pip install -U 'laya-apple[ane]'
laya-apple artifacts fetch laya-typed-decisions

In 1.6.1 that command raised ArtifactMissingError on macOS 27. Nothing else changed: the
code, the parity gate, the routing rule and the macOS 26 archives are the same as in 1.6.1.

What changed

• 1.6.2 pins
  tc3oliver/laya-apple-artifacts at
  commit 44a54765f89d434e9d85f401126cfee7d2059e7d (1.6.0 and 1.6.1 pinned 93181067…).
• That commit adds 10 archives built on macOS 27.0 (26A428): laya 64/96/128,
  laya-multilingual 64/96/128/256 and laya-typed-decisions 64/96/128.
• Its index has 20 entries. The 10 macOS 26 entries are identical to the old pin's.
• The macOS 27 routing profile shipped in 1.6.1, so device="auto" uses fetched artifacts
  without a calibrate step. laya-multilingual L256 stays explicit-only.

How the archives were checked

Two clean-cache runs on the build machine (Apple M4 Max, macOS 27.0, coremltools 9.0):

• released 1.6.1 from PyPI, with --revision 44a54765…;
• the 1.6.2 wheel, with no --revision, so it read its own pin.

Both runs fetched all 10 pairs. Each one matched its SHA-256 in the index, passed
artifacts verify and the FP16 parity gate with 0 hard mismatches (max probability error
0.0128), ran 100% on the Neural Engine with 0 transitions, and had a probe ratio of 0.8 or
less. Per-bucket results and timings:
benchmarks/prebuilt-artifacts-1.6.2-macos27.md.

Limitations

• Only macOS 27.0 was measured. The profile matches every 27.x on an M4 Max with coremltools
  9.0, so 27.1 and later will select these archives too. Every machine still re-runs the
  parity gate, the compute-plan check and the placement probe before it registers one.
• No second machine has checked them. Both runs used the machine that built the archives.
  The same is still true of the macOS 26 archives.
• Fetching skips the build, not Core ML's on-device compile. A cold start of
  laya-typed-decisions (64/96/128) took about 210 s on macOS 27.0
  (results.md).
  ane_startup="background" serves on MLX meanwhile.
• Core ML's compile cache still grows without eviction. One clean fetch of all three models
  added 21.4 GB under ~/Library/Caches/python/com.apple.e5rt.e5bundlecache (each import
  compiles twice: once for the parity gate, once at the registered path).
• The 1.6.1 limitations
  not repeated here still apply.

Upgrading

• No code change needed.
• M4 Max, macOS 27, coremltools 9.0: run laya-apple artifacts fetch MODEL. Artifacts you
  already built locally keep working; fetch skips any bucket whose local artifact already
  passes verification (load_verified).
• On 1.6.1 without upgrading:
  laya-apple artifacts fetch MODEL --revision 44a54765f89d434e9d85f401126cfee7d2059e7d
  (checked in run A of the record).
• Everywhere else, nothing changes.

laya-apple 1.6.1

laya-apple 1.6.1: a safer artifacts fetch, and auto on macOS 27

A patch release for the 1.6 line. Two things change for users:

• laya-apple artifacts fetch now registers an artifact only after every check has passed.
• On an Apple M4 Max with macOS 27 and coremltools 9.0, device="auto" uses the ANE once the
  artifacts are built.

predict, the parity tolerances, the routing rule, the macOS 26.6.2 routing table and the
prebuilt-artifact pin are unchanged.

pip install -U 'laya-apple[ane]'

artifacts fetch keeps only what passed

Fixed in #146.

• An existing bucket is skipped only if it passes load_verified, the runtime's own checks.
  Before, any manifest.json, even an empty one, counted as registered
  (#132).
• The placement probe runs on the staged copy before it moves into place, and again at the
  registered path, since Core ML compiles per path. If anything fails, the new artifact is
  dropped and the previous one stays. Before, a failed probe left the artifact registered, and
  --force had already deleted the old one
  (#133,
  #147).
• --offline (or HF_HUB_OFFLINE=1) now covers the index and archive downloads too. Fetch
  reads them from the Hugging Face cache, still runs the full import checks, and raises
  BackendUnavailableError if they are not cached
  (#143).

For how artifacts prune cleans up after an interrupted import, see
CHANGELOG.md.

auto on macOS 27 (M4 Max)

1.6.0 had no routing profile for macOS 27, so auto stayed on MLX there
(platform_not_validated). 1.6.1 ships one for Apple M4 Max, macOS 27, coremltools 9.0
(#152).

• On macOS 27.0, all three models built for every explicit ANE bucket, ran 100% on the Neural
  Engine with 0 transitions, and passed the FP16 parity gate with 0 hard mismatches
  (research/macos27-validation/).
• Two laya-apple calibrate passes picked the same auto buckets as on 26.6.2: 64, 96 and 128
  for every model. laya-multilingual L256 stays explicit-only
  (benchmarks/routing-macos27/).
• Build the artifacts locally with laya-apple artifacts build MODEL (convert extra). There
  are no prebuilt ones for macOS 27.

Also in 1.6.1

• laya-apple artifacts list --format table prints one row per artifact: bucket, status,
  parity and compute plan. JSON stays the default, unchanged
  (#4).
• scripts/hardware_report.py --json PATH writes a compact summary of a run
  (#7,
  docs/community-benchmarks.md).
• laya-apple info adds an environment block: Python, MLX and the effective offline mode
  (#9).
• laya-apple serve uses one loopback test for the bind address and the Host header, so
  --host 127.0.0.2 no longer answers 421. Remote binding still needs --allow-remote and
  LAYA_API_KEY (#135).
• SECURITY.md covers the 1.6 line, serve and artifacts fetch
  (#134). CI runs with a read-only token
  (#137).
• CI rejects new tracked files over 1 MiB. Larger raw data goes out as external evidence with a
  checksummed manifest
  (#136,
  docs/evidence.md).
• The READMEs open with install and a runnable quickstart
  (#138). The guide has worked score
  and noul examples (#5).

Research, not shipped

• Compile-cache reuse on macOS 27.0
  (#121,
  results.md).
  A cold start of laya-typed-decisions (buckets 64/96/128) took 210.152 s (median of 3)
  against about 2 s warm. Touching the files kept the compile (5.7 s); re-copying them to the
  same path lost it (210.9 s); a move was inconclusive (88.0 s). An import compiles twice:
  about 66 s for the registered load after a 27 s parity gate. No code change.
• The host-side slow state of the no-GIL ANE thread, screened on macOS 27.0 with 1.4's
  synchronous path (#89,
  results.md).
  The positive control was slow in 5 of 6 windows, the negative control in 0 of 6. Holding the
  GIL, a fixed warm-up and a 1 ms probe thread each removed the slow state. No single cause is
  named: holding the GIL also changed the GPU reply wait, so that contrast is wider than the
  GIL alone, and background load was a possible confound. It says nothing about macOS 26.6.2,
  where the slow state was first seen, or about 1.5's adaptive execution.

Limitations

• No prebuilt artifacts for macOS 27. They still cover only Apple M4 Max, macOS 26,
  coremltools 9.0. On macOS 27, artifacts fetch raises ArtifactMissingError; build locally
  (docs/support-matrix.md).
• On macOS 27 only build, parity and routing were measured. The release benchmarks, serve
  beside a local LLM and adaptive ANE execution were not
  (docs/compatibility.md).
• The shipped profile matches every macOS 27.x on an M4 Max with coremltools 9.0, but only 27.0
  (26A428) was measured. Third-party work saw an enumerated-shape Core ML package run on the GPU
  on macOS 27.2. laya-apple ships fixed-shape artifacts, and the compute-plan check and the
  placement probe still run on every machine, but 27.2 is unmeasured
  (docs/support-matrix.md).
• The macOS 27 margin at L96 is thin. For laya and laya-typed-decisions the ANE at 96
  tokens beats MLX at 64 tokens by 0.21–0.27 ms. On 26.6.2 the margin was 0.37–0.41 ms, but
  that table came from the Phase -1 harness and this one from calibrate: different methods,
  not a measured change
  (benchmarks/routing-macos27/).
• A cold start still takes minutes: about 210 s on macOS 27 and 273.5 s in one 26.6.2 run
  (different OS and run count; not compared). Fetching skips the build, not the compile.
  ane_startup="background" serves on MLX meanwhile.
• Core ML's compile cache still grows without eviction: 4.36 GB per 3-bucket cold load on
  macOS 27.
• No second machine has checked the prebuilt artifacts yet.
• Every release measurement comes from one Apple M4 Max, now on two macOS versions.
• The 1.6.0 limitations
  not repeated here still apply.

Upgrading

• No code change needed.
• M4 Max, macOS 27, coremltools 9.0: auto now uses the ANE once artifacts are built, and with
  execution="workers" it also runs 1.5's adaptive ANE execution, which is unmeasured on
  macOS 27. The shipped profile replaces any local laya-apple calibrate profile there. Pass
  device="gpu" to keep 1.6.0's behaviour.
• On that profile laya-apple info reports platform_validated_for_auto_ane: true (was
  false), and Laya.info()["routing_profile"] becomes "shipped". Everywhere,
  laya-apple info also gains the environment block. artifacts list JSON is unchanged.
• artifacts fetch may re-download a bucket that 1.6.0 skipped, if the registered copy fails
  verification.

Release gate

Two full scripts/release_gate.py runs on Apple M4 Max, macOS 27.0, coremltools 9.0. Run 1
failed on a timing-sensitive test, fixed by
#156 with a fake clock. Run 2 passed every
required step (pytest: 1643 passed, 9 skipped).

Differences from 1.6.0's gate, all explained in
benchmarks/release-gate-1.6.1-macos27-notes.md:
it ran on macOS 27.0 instead of 26.6.2, with the shipped macOS 27 profile instead of
routing.json, and before the version bump. It covered fewer artifacts (10 manifests against
15: L64–L128 for every model, plus laya-multilingual L256). derive_routing --check does not
cover the macOS 27 profile; unit tests do. Other load ran on the machine during run 2.

Evidence: coreml-slow-state-trigger

Raw data for research/coreml-slow-state-trigger (#89, #154). SHA-256 and sizes are listed in that directory's evidence.json.

laya-apple 1.6.0

laya-apple 1.6.0: language routing, prebuilt ANE artifacts, opt-in shortlisting

The 1.6 flow:
laya-apple artifacts fetch → from_pretrained("auto") → predict / predict_shortlist.

• Laya.from_pretrained("auto") returns a LayaRouter that loads laya and
  laya-multilingual and picks one per request by the language of the context, with the same
  function laya-apple serve --model auto uses.
• laya-apple artifacts fetch MODEL downloads a prebuilt Neural Engine artifact from
  tc3oliver/laya-apple-artifacts
  and registers it through the validating artifacts import. It replaces the local build
  and its PyTorch dependency: the convert extra is not needed.
• predict works as before. predict_shortlist is opt-in, for choice questions
  with many labels.

pip install -U 'laya-apple[ane]'
laya-apple artifacts fetch laya
laya-apple artifacts fetch laya-multilingual

from laya_apple import Laya

with Laya.from_pretrained("auto") as model:
    result = model.predict(context=context, questions=questions)
    print(result.runtime.model, result.runtime.model_routing, result.runtime.device)

Prebuilt artifacts exist for exactly one platform profile: Apple M4 Max, macOS 26,
coremltools 9.0. On every other Mac, fetch raises ArtifactMissingError naming the build
command, and you build locally as before (laya-apple artifacts build MODEL, with the
convert extra).

A fetched artifact is trusted no more than a local build

Every archive goes through the same checks on the receiving machine before it is registered:

• its SHA-256 against the repository's index.json;
• the manifest against the pinned checkpoint, and the build platform profile;
• the compute plan: 100% ANE, 0 transitions;
• the full FP16 parity gate against the shipped goldens;
• the runtime placement probe, once per fetched bucket.

The default repository is read at the commit this release was validated against,
93181067cfee9c6117a7919321eb303ec36fcbd4, not at its mutable main. A later upload reaches users only through a
release that moves the pin. --revision overrides it; --repo (or
LAYA_APPLE_PREBUILT_REPO) selects another repository, which defaults to main.

Measured results

Result	Source
The published artifacts, downloaded into an empty cache on the profile they were built for: fetch, verify, parity and placement passed for all 10 model/bucket pairs (laya 64/96/128, laya-multilingual 64/96/128/256, laya-typed-decisions 64/96/128). This ran on the build machine with an empty cache; no independent second-machine check yet	benchmarks/prebuilt-artifacts-1.6.0.md
Cold start: the first start of laya-typed-decisions (buckets 64/96/128) at a new artifact location took 273.5 s, against 2.7 s warm. One run on one M4 Max, macOS 26.6.2	research/coreml-compile-cache/screen.md
MLX fast path: about 1–4% on laya-typed-decisions, in a screen only (no full release_bench)	benchmarks/mlx-fast-path-screen/README.md

What else is in it

• Language routing in Python. The choice is in RuntimeInfo.model_routing (stable codes:
  language_english, language_non_latin_script, language_not_english,
  language_undecided_default, no_text_default) and in Result.extra["routing"], which
  to_dict() returns as a top-level routing key, as upstream's Router does.
• predict_shortlist(..., embed_fn, k=20), adapted from upstream laya v0.3.20. For each
  choice question with more than k labels it keeps the k labels whose embed_fn vectors are
  most cosine-similar to the request's, then runs one predict. predict itself is
  unchanged and still scores every label. embed_fn_from_laya builds an embed_fn from the
  checkpoint's own MLX encoder. No accuracy claim is made: upstream's reported figures are
  not remeasured here.
• The MLX fast path, for all three checkpoints:
  • a token-id cache, on by default. It is exact by construction and bounded to about
    16 MB; laya.tokenizer.cache = None turns it off (on a LayaRouter from from_pretrained("auto"), set it on each of router.instances.values());
  • a compiled forward (mx.compile), opt-in: LAYA_APPLE_MLX_COMPILE=1.
    artifact_revision gains :compiled when it runs. Bitwise identical to the eager
    forward on every golden row;
  • length-bucketed batching, opt-in: LAYA_APPLE_MLX_LENGTH_BUCKETS=1, for requests
    with more than 16 questions. Bitwise identical to in-order batching except
    laya-multilingual FP32 (max |Δlogit| 9.5e-7, same decisions).
• laya-apple serve reports adaptive ANE execution in /health (ane.<model>.handoff).
• CI runs the MLX-only integration tests on a hosted Apple silicon runner.
• A new example,
  examples/auto_fetch_shortlist.py.

Research recorded, not shipped

• W8-palettized ANE artifacts. Only the laya-typed-decisions L64 w8-pt result is
  reproduced: 0.650 of FP16 predict latency over three runs. Every laya and laya-multilingual
  cell failed the parity gate. Nothing W8 ships in 1.6.0
  (research/ane-w8/README.md,
  replication.json).
• Splitting one multi-question request across the ANE and the GPU: stopped, no promotion
  (research/intra-request-split/results/verdict.md).

Limitations

• One platform profile has prebuilt artifacts: Apple M4 Max, macOS 26, coremltools 9.0.
  Every other Mac builds locally.
• The prebuilt artifacts have no independent check on a second machine yet. The
  clean-cache fetch, verify, parity and placement check ran on the machine that built them.
  A second machine of the same profile is recommended, not required, for publishing
  (docs/publishing.md); every receiving machine re-validates each artifact before
  registering it.
• Fetching does not make a cold start fast. The first Core ML load still compiles on the
  device, about 4.5 minutes for laya-typed-decisions in the one run measured (273.5 s against
  2.7 s warm). Fetching removes the build, not this compile. ane_startup="background" serves
  on MLX in the meantime.
• Core ML's compile cache grows without eviction: 0.7–1.4 GB per bucket for every new
  artifact location, under ~/Library/Caches/<process name>/com.apple.e5rt.e5bundlecache/.
  laya-apple never evicts it.
• from_pretrained("auto") loads two checkpoints, laya and laya-multilingual, with the
  same arguments. It covers English against multilingual only; upstream's LAYA_AUTO_TASK
  routing and caller language hints are not implemented.
• The MLX speedups come from a screen, not a full benchmark, and are not a headline.
• Every measurement is from one Apple M4 Max.

Upgrade notes

• No code change is needed. predict, routing thresholds, parity tolerances and the default
  execution paths are unchanged.
• The token-id cache is the only new default. Set laya.tokenizer.cache = None (on a LayaRouter from from_pretrained("auto"), set it on each of router.instances.values()) to tokenize
  every text every time, as 1.5 did.
• Result.to_dict() now adds each Result.extra entry as a top-level key, for example
  routing and shortlist. It never replaces model, answers, usage or runtime.
• To use prebuilt artifacts, install the ane extra and run laya-apple artifacts fetch MODEL. --repo or LAYA_APPLE_PREBUILT_REPO selects another repository.

The release gate is in benchmarks/release-gate-1.6.0.md.

Details: docs/guide.md
and CHANGELOG.md.

laya-apple 1.5.0

laya-apple 1.5.0: GPU + Neural Engine serving adapts to system load

GPU + Neural Engine serving now adapts to system load. When the Mac is quiet, laya and
laya-typed-decisions run the Neural Engine on a faster asynchronous path, so it no longer
holds up the GPU's results. When the Mac becomes contended, they fall back automatically to
the stable 1.4 execution path. It is on by default and needs no code change:

pip install -U 'laya-apple[ane]'

with Laya.from_pretrained("convaiinnovations/laya-typed-decisions", execution="workers") as model:
    futures = [model.submit(context=c, questions=q) for c, q in requests]

Measured results

On one Apple M4 Max, against the 1.4 path in the same session, with a
closed-loop short stream (128 tokens, one question) beside a long one:

• 154 production episodes of GPU + Neural Engine overlap: laya and laya-typed-decisions;
  product mix, bursts, repeated transitions and a 66-episode typed soak;
• all 154 episodes stayed on the fast path.

Result	laya	laya-typed-decisions	Source
GPU result return, P50 (1.4 path)	0.035–0.037 ms (4.28–4.29 ms)	0.043 ms (8.60 ms)	val_tables.md
Throughput against the 1.4 path	1.042×	1.038×	same
Median episode P99, against the 1.4 path	0.18–0.19 ms lower	0.53 ms lower	same
Mismatches, routing failures, lost requests, crashes	0	0	same

When the fast path slows down:

• On a contended Mac the asynchronous path can fall into a slow state: a median short latency
  of 12.3 ms against 10.0 ms.
• In the recovery experiment, 12 episodes were already in that state. In every one, the
  breaker tripped within 40 ms, and latency was back to the 1.4 path's within 164–414 ms
  (phase1_tables.md).
• No slow state occurred naturally in the 154 validation episodes.

What else is in it

• ane_handoff:
  • None (the default): on where eligible and available;
  • False: always the 1.4 path;
  • True: required.
  • info()["ane_handoff"] shows its state.
• laya-multilingual is unchanged. Its Neural Engine runs in a worker process and does not
  use adaptive execution.
• If pyobjc-framework-CoreML is missing (it ships in the ane extra), or the
  asynchronous models fail their load checks, the instance runs the 1.4 path and warns once.
• A new parity test runs the asynchronous path against the goldens: every output equals
  coremltools' exactly.
• The research behind it is published:
  • research/coreml-adaptive-breaker/:
    the detector replay, the recovery experiment, and the release validation;
  • research/coreml-staged-handoff/:
    the static route that was closed first.

Limitations

• Measured on one Apple M4 Max only.
• The first 64 Neural Engine requests of every GPU + Neural Engine overlap run the 1.4
  path.
• The fallback's recovery comes from a separate 12-episode experiment, not from the
  validation runs, where no slow state occurred.

Details: docs/guide.md.
The release gate is in benchmarks/release-gate-1.5.0.md.

laya-apple 1.4.0

No runtime change. This release publishes the measurements and research recorded since
1.3.0: laya-apple serve measured beside a busy local LLM, the FAIL of GIL-released Core ML
predict on the product mix and the prebound-predict result (no production change), new
research tooling, and Chinese READMEs. The package code, routing thresholds, parity
tolerances and the published v1.0 measurements are unchanged.

Added

• laya-apple serve measured beside a local LLM (run 2,
  benchmarks/serve/m4-max-r2/). One run on an
  Apple M4 Max, Qwen3.8-27B-oQ4e-mtp on oMLX generating at saturation, 8 decision requests
  per second offered beside it (80% short), serve --model laya; the default
  --model auto was not measured:
  • short-decision P99 with the LLM busy: 47.2 ms with --device auto, 122.3 ms with
    --device gpu; with the LLM idle, 43.2 ms and 55.9 ms;
  • LLM tok/s: −4.0% beside serve auto, −5.3% beside --device gpu, from 42.2 tok/s alone.
    The 1.31-point gap is just above the LLM-alone window spread of 1.28 points;
  • 0 hard mismatches and 0 errors over 7,728 decisions. About 4% of auto short decisions
    went to the GPU by the designed backlog spill, and the P99 includes them;
  • every preregistered criterion passed (G1, G2, L1, L2, C1, E1), each reported as its own
    result. Run 1 (benchmarks/serve/m4-max/) was
    invalid under its own validity checks, and no result is drawn from it. Run 2 used
    revised V2 and V4 definitions, preregistered before its data
    (benchmarks/serve/README.md).
• Chinese READMEs: README.zh-TW.md (Traditional Chinese) and
  README.zh-CN.md (Simplified Chinese), translated from README.md,
  with a language switcher at the top of all three. The English README stays authoritative.
• Research tooling (not part of the package; no results claimed from it):
  • a same-machine comparison harness for Apple-silicon Laya runtimes
    (benchmarks/compare-v1.4/): method, pins and
    tooling; no campaign has run yet;
  • the no-sudo energy sampler and its J/decision harness
    (research/energy-sampler/). Energy sampler run 1
    recorded: it is invalid under its own criteria, and run 2 is preregistered. Energy use
    stays unmeasured.

Changed

• README leads with serve beside a busy local LLM ("Short decisions stay fast while
  your local LLM is busy"), measured with --model laya. The new first section has the
  three-line quickstart and a chart of short-decision P99 and LLM tok/s, --device gpu
  against auto, with the LLM idle and busy, labelled with the machine, the LLM model and
  that it is one run. The chart, docs/readme/serve-llm-load.svg, is drawn by
  scripts/generate_readme_svgs.py from benchmarks/serve/m4-max-r2/results.json (and
  covered by its --check); it refuses a run that is not valid under its own checks. The
  1.3 serve section follows, unchanged.
• The serve "no performance claim" limitation is replaced by what run 2 measured and
  what it does not cover (other LLM servers and models, prefill-heavy loads, other request
  rates, serve's maximum decision throughput, the other checkpoints and --model auto),
  plus the costs it did measure: auto still costs
  the LLM 4.0% tok/s, and multi-question decisions, which always run on the GPU, reach a P99
  of 117.1 ms with the LLM busy. docs/serve.md gains a "Beside a local LLM" section;
  docs/support-matrix.md, docs/compatibility.md and docs/reproducibility.md are
  updated to match.
• GIL-released Core ML predict on the product mix: FAIL, production unchanged
  (research/coreml-nogil-product-mix/).
  Neither candidate (C: GIL-released predict with the GPU in its worker process; D: the same
  with the GPU on a thread) passed the preregistered production gate on any of the three
  models. Correctness and GPU completion isolation passed everywhere; short (ANE) P99
  regressed in every cell (+11.8% to +83.0%) and aggregate throughput failed in four of six
  (down to −15.6%). Production stays thread-placed coremltools for laya and
  laya-typed-decisions, and process placement for laya-multilingual.
  • The study's stop rule named a Swift-worker study as the next step. A post-hoc diagnosis
    of the recorded data found dozens of PyObjC/GIL handoffs per ANE forward on the request
    path, so a smaller, request-path study (prebound predict, #80) was preregistered first;
    the Swift worker is deferred.
• Prebound Core ML predict: no production change
  (research/coreml-prebind-predict/, #80).
  PB prebinds the inputs' and output's backings: 4 PyObjC/GIL crossings per ANE forward
  against C's 108. Preregistered paired verdict against production (ratio, 95% interval):
  PASS on laya (aggregate 1.060 [1.057, 1.062], short P99 0.957 [0.953, 0.962]), PASS on
  laya-typed-decisions (aggregate 1.049, short P99 0.963 [0.952, 0.974]), INCONCLUSIVE on
  laya-multilingual (short P99 0.851 [0.524, 1.384]). As preregistered for this outcome,
  production is unchanged and a larger preregistered replication decides. PB's short P99 was
  about 3–5% lower than C's in the same campaign.
  • Unplanned observation: under this experiment's hetero-only protocol, C (#77's binding)
    also passes on laya and laya-typed-decisions, where #77 recorded it as FAIL. #77's
    regression therefore depends on the protocol; its verdict stands, and the protocol
    dependence is an open question.

Release notes: docs/releases/v1.4.0.md

laya-apple 1.3.0

Adds laya-apple serve: a local, loopback-only decision server with a Jev-compatible API,
so an existing Jev client can use Laya on the Mac by changing only its base URL. The answers
are upstream Laya's, not Jev's: 792 of 792 requests matched unmodified upstream laya.serve
0.3.20 within the FP16 parity gate. Parity now tracks upstream Laya 0.3.20. Routing
thresholds and the published v1.0 measurements are unchanged.

Added

• laya-apple serve, a local Jev-compatible decision server (pip install 'laya-apple[serve]'). It exposes POST /v1/systemone with upstream laya.serve's wire
  format (v0.3.20): request fields, limits, status codes, and the routing block.
  • Jev clients work unchanged: point a client's base URL at http://127.0.0.1:8642.
  • Default --model auto: the checkpoint is chosen per request by the state's language,
    as upstream's Router does. Upstream's language detection is vendored unchanged
    (laya_apple/lang.py). Every response also carries a laya_apple block saying which
    checkpoint and device answered, and why.
  • Where we differ from upstream:
    • it binds loopback only by default;
    • there is no global inference lock, so the GPU and the Neural Engine serve
      concurrently;
    • the Neural Engine warms in the background;
    • the default port is 8642, not 8000.
  • Fidelity: against unmodified upstream laya.serve 0.3.20 serving the same pinned
    weights, 792 of 792 requests matched in wire shape, language routing and values
    within the FP16 parity gate; 365 of them were answered on the Neural Engine
    (benchmarks/serve-compat/).
  • Clients: seven Jev clients, at their released versions and unmodified, completed
    their requests against it
    (integrations/jev-plugins/README.md). This is
    wire compatibility, not Jev-level decisions.
  • Docs: docs/serve.md.
• RuntimeInfo.truncated. It is true when a state was cut to fit the checkpoint's
  maximum length. Upstream truncates too, without reporting it.

Fixed

• Upstream Laya 0.3.20 parity. Prompt building, question validation and the answer
  format now follow upstream Laya 0.3.20 (NandhaKishorM/laya@23a1752), not 0.3.5. Token ids
  are unchanged for every request that 0.3.5 and 0.3.20 already treated alike. The rest
  differs as follows:
  • A conversation context (a list of turns) that exceeds max_len keeps its newest
    turns. It is now truncated from the start, not the end.
  • Non-string instructions keep non-ASCII text, where they were \uXXXX-escaped before.
  • noul accepts optional labels ({"false": ..., "true": ...}), which replace the
    false: / true: option prefixes. labels on choice or score, or invalid labels,
    raise InvalidRequestError.
  • noul criteria keys are case-insensitive. Keys other than true/false raise
    InvalidRequestError; before, they were replaced by the default texts without a
    word.
  • Every answer gains answer_confidence, max(p), and the answer fields follow upstream's
    order.
  • questions={} returns empty answers with zero usage instead of raising. Nothing is
    tokenized, routed or run: runtime records backend="none", device="none" and
    routing_reason="no_questions", and workers mode emits no trace.
  • A checkpoint temperature that is not a number now loads with a warning and falls back
    to 1.0, where it raised before. Out-of-range values in temperature are now warned
    about as well as those in temperature_by_options.
• Goldens recorded from upstream 0.3.20. The [reference] extra pins laya==0.3.20.
  scripts/make_reference.py records the references into
  research/upstream-reference/raw/laya-0.3.20/. Every Phase -1 case is identical to its
  0.3.5 record, which scripts/make_goldens.py enforces. Five new cases per model (13
  rows) cover the changes above, and each golden now stores upstream's answers. The v1.0
  parity tables predate these cases (docs/correctness.md).

Changed

• README leads with laya-apple serve. The new first section has the three-line
  quickstart, the client and fidelity results, and a terminal figure,
  docs/readme/serve-demo.svg. The figure is rendered by scripts/generate_readme_svgs.py
  (and covered by its --check) from docs/readme/serve-demo.json, which
  scripts/capture_serve_demo.py records from a real run: the released typesafe-sdk 0.7.1,
  unmodified, against a local server. The Switchyard section follows it, unchanged. The
  README's limitations gain the serve limits.
• Switchyard launch media. docs/media/switchyard-launch.gif and .mp4 are re-cut for
  readability. Both are cropped to the rail yard, with the whole-round result in large type.
  The result card is simplified to trains late and P99 decision latency. The MP4 is now 20 s
  and opens on the result; the GIF drops the command, which the README shows above it. Same
  run (run-003), same numbers; sources in
  docs/media/switchyard-launch.md. The README states the
  result in text under the GIF.

Release notes: docs/releases/v1.3.0.md

laya-apple 1.2.0

Adds laya-apple switchyard: a one-command, headless open-loop benchmark of GPU-only
against GPU + ANE serving, replayed in the browser as a rail-yard game from the recorded
request traces. The runtime, routing and published v1.0 measurements are unchanged.

Added

• laya-apple switchyard. A headless benchmark of the frozen switchyard-v1 workload
  (bursty seeded arrivals, single-question routing decisions as "trains", background
  long-context requests loading the GPU), replayed as a rail-junction game. The CLI runs the
  timed gpu_only and (when the Neural Engine is ready) hybrid rounds, writes result.json
  and trace.jsonl, builds a self-contained replay.html and opens it (--no-open to skip).
  --setup-ane builds and parity-validates the ANE artifact, calibrates if needed, then runs.
  Missing ANE never fails the command — it runs gpu_only and prints the setup command. See
  docs/switchyard.md for the workload, measurement boundaries and
  result schema.
  • Official campaign (benchmarks/switchyard/v1-m4-max/, 3 runs, Apple M4 Max, macOS
    26.6.2, laya-typed-decisions, standard workload: seed 11, 60 s, 40 req/s nominal, 1,422
    trains, 100 ms deadline): gpu_only 1,407–1,408/1,422 late (P99 decision
    3,107.7–3,170.6 ms, P99 queue 3,095.9–3,158.8 ms); hybrid (GPU + ANE) 0 late (P99
    decision 54.5–54.7 ms, P99 queue 39.6–42.8 ms); 0 decision disagreements, 0 misrouted in
    all 3 runs. Round order is fixed by seed parity (hybrid first in this campaign),
    design.counterbalance is "none", and these numbers are not comparable with the v1.0
    open-loop table (different rate, measurement boundary and workload) — see
    docs/switchyard.md. Report:
    benchmarks/switchyard/README.md.
• Switchyard launch media. docs/media/switchyard-launch.gif (README) and
  docs/media/switchyard-launch.mp4, a capture of the recorded replay of campaign run-003;
  sources in docs/media/switchyard-launch.md. The README
  now opens with Switchyard.

Fixed

• laya-apple switchyard opens the replay on macOS. webbrowser.open() reports success
  for a file:// URL without opening anything, so the replay never appeared. It is now
  opened with /usr/bin/open, and the path is printed if no browser opens.