Skip to content

fix(server): support sampled requests in target layer split#295

Merged
davide221 merged 2 commits into
Luce-Org:mainfrom
weicj:fix-layer-split-sampling
Jun 3, 2026
Merged

fix(server): support sampled requests in target layer split#295
davide221 merged 2 commits into
Luce-Org:mainfrom
weicj:fix-layer-split-sampling

Conversation

@weicj

@weicj weicj commented May 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

From the user side, target layer split currently works for greedy requests but rejects normal sampled OpenAI-compatible requests with sampling_unsupported. In practice this means requests using temperature > 0 such as temperature=0.7, or requests using repetition / frequency / presence penalties, fail even though the same model can sample correctly on the single-card backend.

This PR fixes that gap for target layer split. Adapters that can return final-token logits may opt into the existing CPU sampler path, so split and non-split generation follow the same sampling rules. Adapters that have not implemented logits output still fail explicitly instead of silently falling back to inconsistent behavior.

Changes

  • Add a LayerSplitAdapter::supports_cpu_sampling() capability gate.
  • Keep the existing sampling_unsupported protection for adapters that do not opt in.
  • Extend Qwen35 layer-split projection so it can return final-token logits as well as argmax tokens.
  • Extend Gemma4 layer-split projection in the same way, including Gemma4 final logit softcap before CPU sampling.
  • Make Qwen35 and Gemma4 layer-split adapters cache the final prefill logits and use the existing shared sample_logits() path during AR decode when temperature > 0, repetition penalty, frequency penalty, or presence penalty is active.
  • Persist and restore the cached final prefill logits with layer-split snapshots, so sampled restore paths do not fail or reuse stale logits when no new prefill is needed before decode.
  • Keep Qwen35 DFlash speculative decode on the greedy path only; sampled / penalty requests now stay on AR decode where CPU sampling is available.
  • Add unit coverage for the layer-split sampling capability gate.

Notes

  • This does not add a new sampler. It reuses the existing shared CPU sampler so split and non-split paths follow the same sampling rules.
  • Current validation: CUDA SM61 dflash_common, test_server_unit, and dflash_server build passed; test_server_unit passed with 1606 assertions and 0 failures. HIP gfx906 dflash_server build passed, and a dual Pro VII Gemma4 E4B Q4 layer-split smoke returned an OpenAI-compatible completion with temperature=0.7.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed May 28, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 11 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/src/qwen35/qwen35_layer_split_adapter.cpp
Comment thread server/src/gemma4/gemma4_layer_split_adapter.cpp
@weicj weicj changed the title fix(server): enable sampling for target layer split fix(server): support sampled requests in target layer split May 28, 2026
@weicj weicj force-pushed the fix-layer-split-sampling branch from 742b86c to a9aedf7 Compare May 28, 2026 18:08
easel pushed a commit to easel/lucebox-hub that referenced this pull request May 28, 2026
Merge PR Luce-Org#295 into auto-integration
98da34f 
# Conflicts:
#	server/src/gemma4/gemma4_layer_split_adapter.cpp
#	server/src/qwen35/qwen35_layer_split_adapter.cpp
easel pushed a commit to easel/lucebox-hub that referenced this pull request May 28, 2026
fix(ci): remove duplicate CMake test registrations
72355a1 
Remove the repeated drafter/anchor test target block so CMake can configure the integration stack after merging upstream and PR Luce-Org#295.
This was referenced May 28, 2026
Merged
Merged
@davide221 davide221 merged commit 080b501 into Luce-Org:main Jun 3, 2026
2 of 3 checks passed
This was referenced Jun 4, 2026
Closed
Merged
easel added a commit to easel/lucebox-hub that referenced this pull request Jun 4, 2026
@easel
feat(server): GgufMetadata reader + SHA-256 sidecar for /props schema-4
f7d8278 
## What

Adds a richer GGUF identity reader on top of Howard Su's existing
`gguf_inspect` (PR Luce-Org#305):

- `GgufMetadata` struct — captures `general.*` + `<arch>.*` header fields
  (architecture, name, file_type, quantization_version, block_count,
  embedding_length, context_length, vocab_size) with -1 / "" sentinels
  distinguishing "not in GGUF" from legitimate zero.
- `read_gguf_metadata(path, compute_sha256)` — best-effort header read;
  optional SHA-256 of the whole file.
- Self-contained SHA-256 mini-impl (RFC 6234) — no OpenSSL dependency
  added for one hash.
- `<path>.sha256` sidecar caching — first server start hashes the file
  (~30s for a 17 GB GGUF on NVMe), subsequent starts read the sidecar.
  Sidecar I/O failures are non-fatal.
- `llama_ftype_name` decode — maps `general.file_type` ints to
  human-readable names ("Q4_K_M", "IQ4_XS", etc.) for /props.

## Why

`/props` schema-4 wants a single authoritative "exactly what binary +
GGUF + quant + sha256 is loaded" payload so benchmarking and provenance
tooling can pin model identity across runs without re-parsing GGUF
headers in every consumer. The sidecar makes the SHA-256 free after the
first boot, which is what makes it usable as a default-on identity
field.

## Dependencies

None. This is purely additive on top of `gguf_inspect.{cpp,h}` as merged
in PR Luce-Org#305 — zero deletions, 333 insertions total. No other server
files or build rules change in this PR; consumers will be wired up
separately.

## Scope note

This PR is the extracted-and-cleaned remnant of the previously closed
PR Luce-Org#336 after a provenance audit; everything else from that branch
(c2_gate, qwen3 drafter changes, structural-defense loaders, and the
inadvertent reverts of Luce-Org#273/Luce-Org#295/Luce-Org#297) is either landing through its
canonical PR (Luce-Org#274) or being dropped entirely.
easel added a commit to easel/lucebox-hub that referenced this pull request Jun 5, 2026
@easel
feat(server): GgufMetadata reader + SHA-256 sidecar for /props schema-4
532f678 
## What

Adds a richer GGUF identity reader on top of Howard Su's existing
`gguf_inspect` (PR Luce-Org#305):

- `GgufMetadata` struct — captures `general.*` + `<arch>.*` header fields
  (architecture, name, file_type, quantization_version, block_count,
  embedding_length, context_length, vocab_size) with -1 / "" sentinels
  distinguishing "not in GGUF" from legitimate zero.
- `read_gguf_metadata(path, compute_sha256)` — best-effort header read;
  optional SHA-256 of the whole file.
- Self-contained SHA-256 mini-impl (RFC 6234) — no OpenSSL dependency
  added for one hash.
- `<path>.sha256` sidecar caching — first server start hashes the file
  (~30s for a 17 GB GGUF on NVMe), subsequent starts read the sidecar.
  Sidecar I/O failures are non-fatal.
- `llama_ftype_name` decode — maps `general.file_type` ints to
  human-readable names ("Q4_K_M", "IQ4_XS", etc.) for /props.

## Why

`/props` schema-4 wants a single authoritative "exactly what binary +
GGUF + quant + sha256 is loaded" payload so benchmarking and provenance
tooling can pin model identity across runs without re-parsing GGUF
headers in every consumer. The sidecar makes the SHA-256 free after the
first boot, which is what makes it usable as a default-on identity
field.

## Dependencies

None. This is purely additive on top of `gguf_inspect.{cpp,h}` as merged
in PR Luce-Org#305 — zero deletions, 333 insertions total. No other server
files or build rules change in this PR; consumers will be wired up
separately.

## Scope note

This PR is the extracted-and-cleaned remnant of the previously closed
PR Luce-Org#336 after a provenance audit; everything else from that branch
(c2_gate, qwen3 drafter changes, structural-defense loaders, and the
inadvertent reverts of Luce-Org#273/Luce-Org#295/Luce-Org#297) is either landing through its
canonical PR (Luce-Org#274) or being dropped entirely.
easel added a commit to easel/lucebox-hub that referenced this pull request Jun 5, 2026
@easel
feat(server): GgufMetadata reader + SHA-256 sidecar for /props schema-4
c670047 
## What

Adds a richer GGUF identity reader on top of Howard Su's existing
`gguf_inspect` (PR Luce-Org#305):

- `GgufMetadata` struct — captures `general.*` + `<arch>.*` header fields
  (architecture, name, file_type, quantization_version, block_count,
  embedding_length, context_length, vocab_size) with -1 / "" sentinels
  distinguishing "not in GGUF" from legitimate zero.
- `read_gguf_metadata(path, compute_sha256)` — best-effort header read;
  optional SHA-256 of the whole file.
- Self-contained SHA-256 mini-impl (RFC 6234) — no OpenSSL dependency
  added for one hash.
- `<path>.sha256` sidecar caching — first server start hashes the file
  (~30s for a 17 GB GGUF on NVMe), subsequent starts read the sidecar.
  Sidecar I/O failures are non-fatal.
- `llama_ftype_name` decode — maps `general.file_type` ints to
  human-readable names ("Q4_K_M", "IQ4_XS", etc.) for /props.

## Why

`/props` schema-4 wants a single authoritative "exactly what binary +
GGUF + quant + sha256 is loaded" payload so benchmarking and provenance
tooling can pin model identity across runs without re-parsing GGUF
headers in every consumer. The sidecar makes the SHA-256 free after the
first boot, which is what makes it usable as a default-on identity
field.

## Dependencies

None. This is purely additive on top of `gguf_inspect.{cpp,h}` as merged
in PR Luce-Org#305 — zero deletions, 333 insertions total. No other server
files or build rules change in this PR; consumers will be wired up
separately.

## Scope note

This PR is the extracted-and-cleaned remnant of the previously closed
PR Luce-Org#336 after a provenance audit; everything else from that branch
(c2_gate, qwen3 drafter changes, structural-defense loaders, and the
inadvertent reverts of Luce-Org#273/Luce-Org#295/Luce-Org#297) is either landing through its
canonical PR (Luce-Org#274) or being dropped entirely.
easel added a commit to easel/lucebox-hub that referenced this pull request Jun 8, 2026
@easel
feat(server): GgufMetadata reader + SHA-256 sidecar for /props schema-4
4576a44 
## What

Adds a richer GGUF identity reader on top of Howard Su's existing
`gguf_inspect` (PR Luce-Org#305):

- `GgufMetadata` struct — captures `general.*` + `<arch>.*` header fields
  (architecture, name, file_type, quantization_version, block_count,
  embedding_length, context_length, vocab_size) with -1 / "" sentinels
  distinguishing "not in GGUF" from legitimate zero.
- `read_gguf_metadata(path, compute_sha256)` — best-effort header read;
  optional SHA-256 of the whole file.
- Self-contained SHA-256 mini-impl (RFC 6234) — no OpenSSL dependency
  added for one hash.
- `<path>.sha256` sidecar caching — first server start hashes the file
  (~30s for a 17 GB GGUF on NVMe), subsequent starts read the sidecar.
  Sidecar I/O failures are non-fatal.
- `llama_ftype_name` decode — maps `general.file_type` ints to
  human-readable names ("Q4_K_M", "IQ4_XS", etc.) for /props.

## Why

`/props` schema-4 wants a single authoritative "exactly what binary +
GGUF + quant + sha256 is loaded" payload so benchmarking and provenance
tooling can pin model identity across runs without re-parsing GGUF
headers in every consumer. The sidecar makes the SHA-256 free after the
first boot, which is what makes it usable as a default-on identity
field.

## Dependencies

None. This is purely additive on top of `gguf_inspect.{cpp,h}` as merged
in PR Luce-Org#305 — zero deletions, 333 insertions total. No other server
files or build rules change in this PR; consumers will be wired up
separately.

## Scope note

This PR is the extracted-and-cleaned remnant of the previously closed
PR Luce-Org#336 after a provenance audit; everything else from that branch
(c2_gate, qwen3 drafter changes, structural-defense loaders, and the
inadvertent reverts of Luce-Org#273/Luce-Org#295/Luce-Org#297) is either landing through its
canonical PR (Luce-Org#274) or being dropped entirely.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cubic-dev-ai cubic-dev-ai[bot] cubic-dev-ai[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@weicj @davide221