Skip to content

Releases

PLI 5.6.5

PLI 5.6.5 Pre-release
Pre-release

Choose a tag to compare

@nicopingu nicopingu released this 17 Nov 21:04

PLI 5.6.5: one local server for text, voice, images and video

PLI 5 turns a Mac Studio into a self-contained AI server: Qwen3.8-27B for chat, code and tool calls, plus voice, transcription, images and video with sound, behind one OpenAI- and Anthropic-compatible endpoint, all on the machine.

New in 5.6.4 and 5.6.5

  • Video LoRA in memory, several at once: h3 adds the LoRA to each block's weights as it loads them (tools/h3c-lora.patch for h3.c), converted once by tools/pli-lora-natif. No fused copy of 33 to 62 GB per LoRA, up to 4 LoRA in one video. Same prompt and seed: the MP4 is byte-for-byte identical to the one rendered from a fused copy, with no measurable time cost.
  • Room asked of an outside scheduler: a generation child short of memory can ask a scheduler on the same Mac to pause its own jobs (place_fichier), and waits up to 120 s.
  • /health shows every model served: per generation family, the model of the last request and the requests in progress.

What's new in PLI 5 (from 5.6.2 and 5.6.3, never published separately)

  • One process, one endpoint: /v1/chat/completions, /v1/messages, /v1/audio/speech, /v1/audio/transcriptions, /v1/images/generations, /v1/videos. Each generation family runs in its own child process, started on demand and unloaded when idle.
  • The LLM keeps the right of way: images, voices and videos pause between two steps while the LLM writes (97.7 % of its writing speed kept), and the LLM is capped while a child runs.
  • Memory-aware: each model declares what it needs, idle families make room, and a request that does not fit gets a clear 503 instead of sending the Mac into swap.
  • Fast restart (--startup-cache on): the 27B starts in 9 s instead of 71 s, same figures, same text.
  • Cancellation, a prompt guard for images and video, self-maintenance (PLI_TASKS), exact drafts (PLI_TREE_EXACT=1).
  • Security review: no secret in the history (gitleaks), static analysis clean (semgrep p/default).

Measured on an M1 Ultra 64 GB

  • Writing: 111 tok/s on a short prompt, 34 tok/s at 96k tokens.
  • A 1024² image in 38 to 44 s (Qwen-Image 2.1 + Lightning); a 22-frame video with sound in 2 to 2.5 minutes.

Before you upgrade

  • Pre-release, validated on one machine (Mac Studio M1 Ultra, 64 GB) and one model family (Qwen3.8-27B).
  • The package is pli; saved conversations move to their new key with python -m pli.additions.rekey.
  • Generation is off by default (PLI_GEN=1 and PLI_GEN_CONFIG; examples in docs/gen/).

Everything else: CHANGELOG.md.

PLI 4.2.1

PLI 4.2.1 Pre-release
Pre-release

Choose a tag to compare

@nicopingu nicopingu released this 12 Nov 22:36
  • Copy log for head rounds too (PLI_COPY_LOG=<file>, off by default): a round where a copy was offered but the
    learned choice (PLI_HEAD_BEFORE_COPY=3) took the head's tree now writes the copy, the whole tree (tokens, parents)
    and the choice's moving average, so the choice can be checked offline against the text written. The choice itself
    does not change. PLI_COPY_LOG stays out of the checkpoint key's settings fingerprint.
  • python -m tensorfold.pli.rekey plan-code: the newest checkpoint key with the digest of this tree's code, for a
    code change that does not change a checkpoint's bits. First release whose number changes without touching the
    checkpoint key (4.2.0 took the release number out of it).

PLI 4.2.0

PLI 4.2.0 Pre-release
Pre-release

Choose a tag to compare

@nicopingu nicopingu released this 11 Nov 22:54
  • Checkpoints written in the background (PLI_SPILL_BACKGROUND=1): a conversation's checkpoint goes to disk from
    its own thread instead of the engine's. Over 24 hours of service on an M1 Ultra the engine waited 143 s for the disk
    instead of 424 s the day before (spills 328 -> 88 s). Same text.
  • Tool calls follow their schema (PLI_TOOL_GRAMMAR=1, needs xgrammar): when a request offers tools, the reply's
    calls are constrained by a grammar built from the tools' JSON schemas (name, required keys, valid values). On 24 agent
    turns of a real session, 24 valid calls against 20 without the grammar; same text elsewhere.
  • End of a reply read back from disk (PLI_DISK_REPLY_END=1): when the checkpoint of a finished reply was evicted
    from memory but is on disk, the next turn resumes from it instead of reading the conversation again (first word of a
    resumed turn 1 546 s -> 6.9 s on the bench, 9 texts of 9 identical).
  • The code enters the checkpoint key (PLI_CODE_KEY=1): the sources of PLI and of the prompt path are hashed into
    the key, so a checkpoint computed by other code is never resumed. With it on, the release number leaves the key: a
    new release that does not change the computation keeps its checkpoints.
  • python -m tensorfold.pli.rekey re-keys checkpoints on disk without recomputing them (header and file name
    rewritten, tensors copied, modification time kept), for a key change that does not change their bits. Used to move
    the checkpoints of 4.1.2 to the key of 4.2.0.
  • Request tokens kept for training (PLI_FALCON_TOKENS=<dir>, off by default): the prompt and reply tokens of each
    request, one file per request (directory mode 700), deleted after PLI_FALCON_TOKENS_DAYS days (7), for a night job
    that recomputes the target's states and adjusts the draft on real traffic. Nothing changes in the computation.
  • The tool-grammar tests read the tokenizer from PLI_TEST_QWEN38_MODEL (skipped when absent).

PLI 4.1.2

PLI 4.1.2 Pre-release
Pre-release

Choose a tag to compare

@nicopingu nicopingu released this 05 Nov 16:45
  • pli setup turns A1 on. With the m1-ultra profile, the list shown before downloading has one more item: A1's
    map (Latent Sparse Attention, release d-lsa-weights-v1, 6 MB, Apache-2.0). On yes, pli setup downloads it from
    the GitHub release, checks its SHA-256 (nothing is kept if it differs), converts it to NumPy in <prefix>/models,
    and writes PLI_A1_LSA_MAP, PLI_S3B_THRESHOLD=16384 and PLI_S3B_ROWS=4096 in the service. It is left out with
    --no-a1, on other Macs, and when the model's config.json is not Qwen3.8-27B's; a failed download leaves PLI
    without A1 and the install goes on. Before, A1 had to be set by hand.
  • The map is converted without torch (src/tensorfold/install/lsa_map.py, used by tools/convert_lsa_map.py
    too): the PyTorch file is read as the zip of arrays it is, with a pickle reader that accepts only the names of a
    tensor file and refuses any other, so no code from the file runs. On the release's file it writes the same .npz,
    byte for byte, as the torch path of 4.1.1 and the map served in production.

PLI 4.1.1

PLI 4.1.1 Pre-release
Pre-release

Choose a tag to compare

@nicopingu nicopingu released this 05 Nov 14:11
  • Name of A1's map. 4.1.0 named the file to download qwen38_27b.pt; the file of LSA's release
    d-lsa-weights-v1 is qwen38_27b__long-16k-1M-heavy-key-and-query-d2048.pt (SHA-256 24cee934f969756db228a8df7f2bc7613979b63ac3b1d5ac3d9aa42e4339bb9b). README,
    docs/switches.md and tools/convert_lsa_map.py give that name and the download command.
  • tools/convert_lsa_map.py checked: run on that file (torch 2.13.0, NumPy 2.4.6), it writes the 32 arrays (16
    layers, W_Q_idx and W_K_idx, [768, 64] float32) equal to the map served in production since 4.1.0's A1.
    No code change.

PLI 4.1.0

PLI 4.1.0 Pre-release
Pre-release

Choose a tag to compare

@nicopingu nicopingu released this 05 Nov 14:04
  • checkpoint field of /health and /v1/models. The served weights (folder, or Hugging Face repo and revision),
    their model type and quantization, the draft, the MTP head and the PLI version, so a client sees which model answers
    without reading the server's settings. Same text.
  • Checkpoints kept in 4 bits (PLI_CHECKPOINT_BITS=4 in the m1-ultra profile, off in the code; in production
    since 4 Nov). A checkpoint evicted from memory or written to disk is
    rotated (Hadamard transform, as in QuaRot) and quantized in groups of 64 (PLI_CHECKPOINT_GROUP); on resume it is restored to 16
    bits. The checkpoint key carries the format, so a file written in 16 bits is not read as 4 bits, and back. Under
    memory pressure (agent sessions that evict their checkpoints): disk of the checkpoints divided by 4, time to the
    first token of resumed turns -54%; on other resumes -6 to -13%. Text can change on resumed turns: on conversations
    judged turn by turn (code, admin, maths), no right answer became wrong; one maths answer wrong at turn 2 was corrected
    at turn 3. 8 (8 bits, no rotation) is accepted; it was measured and not kept. The production name
    PLI_POINTS_BITS is still accepted.
  • A1 in the prefill (PLI_A1_LSA_MAP, PLI_A1_PREFILL_ONLY=1; in production since 5 Nov). Past
    PLI_S3B_THRESHOLD positions, each row of a prompt chunk reads the 10% of the tiles of 32 keys that the index of
    Latent Sparse Attention scores best, plus the last 1,024
    positions; decode and trees keep the attention of 4.0.1. The distilled map is not shipped: download qwen38_27b.pt
    from LSA's release d-lsa-weights-v1 (Apache-2.0) and convert it with tools/convert_lsa_map.py (needs torch).
    It goes with PLI_S3B_THRESHOLD=16384 and PLI_S3B_ROWS=4096 in the environment; without the map, the profile's
    65,536 stays and nothing changes. On Qwen3.8-27B (Sabi-Noema 4-bit), M1 Ultra: time to the first token at 146k
    tokens 712.5 s instead of 844.6 s (-15.6%), -6% at 61k; offline, the NLL of the reduced prefill equals the full
    one (0.72229 against 0.72127 at 48k, 1.01024 against 1.0106 at 97k). Text can change: on conversations judged turn
    by turn, two answers changed from right to wrong and three got better. Experimental, not in production:
    PLI_A1_MAX_TILES (a cap on the tiles per row: -22 to -27% at 146k but the NLL rises, +0.0046 at 160 tiles) and
    PLI_A1_PREFILL_ONLY=0 (A1 for decode too: decode 5 to 21% slower on long conversations). The production names
    PLI_ANCRE1_LSA, PLI_ANCRE1_LECTURE_SEULE, PLI_A1_TRANCHE, PLI_A1_PETITE and PLI_A1_GARDES_MAX are still
    accepted.

PLI 4.0.1

PLI 4.0.1 Pre-release
Pre-release

Choose a tag to compare

@nicopingu nicopingu released this 04 Nov 09:50
  • pli serve prints the PLI mark (folded paper) with the version when it starts. The README gets the PLI logo,
    a speed summary, a Hugging Face link for each model family, and says that PLI is tuned for the M1 Ultra.
  • pli setup installs PLI; install.sh only makes the virtual environment. ./install.sh install finds Python
    3.11 to 3.14, creates ~/pli/venv (or --prefix), installs PLI in it and runs pli setup with the same options.
    pli setup does what the rest of install.sh did, with the same checks, messages, defaults and service settings
    (profile m1-ultra on an Ultra chip with 64 GB, PLI_ANE_FFN, MLX_MAX_MB_PER_BUFFER=500,
    MLX_MAX_OPS_PER_BUFFER=100, --context 229376 --parallel 4, --spill-gib 40 with 70 GB free, the hybrid draft),
    through pli pull and pli service. pli status, start, stop, restart, logs and uninstall replace the
    commands of install.sh, which still run them (./install.sh status is pli status).
  • Nothing is downloaded without consent. Before downloading, pli setup lists each item: Hugging Face repo, page
    and revision (main, not pinned, with its commit), license from the model card, size read from the Hub's API
    ("about" when the Hub does not answer), where it goes, the free disk, and what it brings. Files already in the
    Hugging Face cache are marked "already downloaded" and not counted. It then asks [y/N]; --yes answers yes, and
    without a terminal and without --yes it stops and says what to add. It stops too when the disk is too small.
  • Each item is optional. --no-hybrid (no MTP head), --no-drafter (new: no draft at all, --drafter none --no-drafts), --no-models (nothing downloaded; the service now starts when the model and the draft are already on
    the Mac), --model DIR, --drafter REPO|DIR (new) and --mtp-head FILE (new: a head already written).
  • pli bench runs the public suite pli-bench-v1 against the local server (the port of pli setup), thinking off.
    The bench and its texts (811 KB) are now part of the package (src/tensorfold/bench/); tools/pli_bench.py runs the
    same code from a clone, with standard Python only. tools/quantize_mtp_head.py likewise runs
    tensorfold.install.mtp_head, which pli setup uses.
  • pli alone prints a short help with examples instead of an argument error.
  • Checkpoint at the start of the last message (PLI_LAST_MESSAGE_CHECKPOINT, 1,024 in the m1-ultra profile,
    off in the code; in production since 3 Nov). Besides the history checkpoint, PLI keeps the state at the start of
    the prompt's last message (the turn marker that opens the reply, looked for backwards in the history) when at least
    N tokens are read there since the resume. The next request that keeps the history but not that message (a question
    replaced, a reply thrown away, a sub-agent on the same history) starts from there instead of from the previous
    request's checkpoint. Same text. On the agent session of the public suite pli-bench-v1: the same 8 replies, prompt
    served from the cache 73.5% instead of 58.9%, sum of the times to the first token 238.0 s instead of 357.3 s
    (-33%), writing speed unchanged. The production name PLI_DEBUT_DERNIER is still accepted.

PLI 4.0.0

PLI 4.0.0 Pre-release
Pre-release

Choose a tag to compare

@nicopingu nicopingu released this 03 Nov 08:08

PLI 4.0.0 is a pre-release. It is validated on one machine (Mac Studio M1 Ultra, 64 GB, macOS 27) and one model
family (Qwen3.8-27B dense, MLX 4-bit). Other Macs and models run, but are not measured.

PLI is an LLM inference server for Apple Silicon, forked from TensorFold
0.6.6 (Apache-2.0). It keeps TensorFold's OpenAI- and Anthropic-compatible API and its exact speculative decoding, and
adds work that spreads Qwen3.8-27B across the GPU, the CPU and the Neural Engine, its own prompt and decode kernels,
and a draft that picks, per conversation, between DFlash2 and the model's own MTP head.

Since 3.6.0, the last public version (details in the CHANGELOG, sections 3.7.0 and 4.0.0):

  • Drafts: hybrid DFlash2 + MTP head chosen per conversation, DFlash2 alone for one-turn questions, draft trees for
    one-turn questions, copy of the context or DFlash2 chain chosen per stream, extended draft lists, faster draft
    kernels.
  • Verify and decode: Kapitsa's 8-row verify GEMM, attm on short rows, reduced attention anchored to positions, and
    for small prompt chunks beyond 65,536 tokens of context.
  • Neural Engine: weights requantized on the CPU, kept awake between layers, units chosen by chunk size (private Apple
    APIs: a macOS update can break it; PLI then computes on the GPU).
  • Agent turns: the reply kept for the next turn, checkpoints without a copy of the KV cache, checkpoints at fork
    points, evicted checkpoints written to disk, control tokens in a request kept as text.
  • --profile m1-ultra sets every production setting; install.sh installs PLI as a per-user service with its models.
  • A public bench with fixed prompts (tools/pli_bench.py): run it on your Mac and compare.
  • Settings renamed in English; the older names are still accepted.

Measured with the public suite on the validated machine (prompt reading / writing, tokens per second):

Prompt PLI 4.0.0 Kapitsa 0.2.24 llama.cpp b11926 mlx-lm 0.32.2
1k 294 / 53.2 210 / 65.3 193 / 25.1 159 / 24.0
32k 285 / 49.2 189 / 43.1 168 / 19.9 153 / 18.8
64k 245 / 40.8 167 / 32.9 144 / 17.1 136 / 17.2
96k 213 / 34.1 151 / 33.6 126 / 15.5 not measured
agent session, writing (first 6 answers) 47.0 46.7 24.3 20.9

Setup, every size and the agent session: README, section Measurements. Every change and its measurement:
CHANGELOG.md. Install: git clone https://github.com/nicopingu/pli.git && cd pli && ./install.sh install.