fix: reject malformed scaled steering entries with 4xx#29
Merged
RhizoNymph merged 1 commit intoMar 31, 2026
Merged
Conversation
RhizoNymph
added a commit
that referenced
this pull request
Mar 31, 2026
RhizoNymph
added a commit
that referenced
this pull request
Apr 15, 2026
commit acf39ab1a8b7790be65ca3ceca623cd1b148569c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 19:35:30 2026 -0700
feat: gate global steering changes behind steering api key if set
commit 58ab6e83a55f6092470ff0b97535d03b17e9e999
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 18:50:40 2026 -0700
remove agent overview doc
commit 3d77e12e1dc2685bbe7945878b0c60c006acbb68
Merge: ada55957f 3b08fb305
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 17:09:56 2026 -0700
Merge pull request #83 from RhizoNymph/refactor/steering-mixin
refactor(steering): extract SteeringModelRunnerMixin to match LoRA pattern
commit ada55957f721d48848a5391efb2f4b0dd908e72d
Merge: 9a00f0f08 0b325140b
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 16:58:05 2026 -0700
Merge pull request #81 from RhizoNymph/docs/steering-update
docs(steering): refresh supported-scope and runtime-boundaries
commit 9a00f0f083e25058f7c66108457a78e21c590ec5
Merge: 5ffa1cd69 0eba71d29
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 16:57:15 2026 -0700
Merge pull request #82 from RhizoNymph/refactor/steering-modules-router-merge
refactor(steering): consolidate modules router under serve/steering/
commit 5ffa1cd6984fb48d04633f950e7808d31234d221
Merge: 52f85006f 548684983
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 16:49:55 2026 -0700
Merge pull request #80 from RhizoNymph/chore/rename-steering-phase-tests
chore(steering): rename phase1..phase5 test prefixes to semantic names
commit 52f85006fd34715af8154237e703a85f50ef8d73
Merge: 95ffa52f3 49649a584
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 16:48:29 2026 -0700
Merge pull request #79 from RhizoNymph/chore/steering-sampling-params-imports
chore(steering): hoist sampling_params inline imports
commit 3b08fb3059bb7042b434ec6f9ae40f136472f30f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 15:56:35 2026 -0700
refactor(steering): extract SteeringModelRunnerMixin to match LoRA pattern
Co-authored-by: Claude
commit 0eba71d2901616c54b3cdf1108a87e687a1fd535
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 15:00:50 2026 -0700
refactor(steering): fold serve/steering_modules/ into serve/steering/
Co-authored-by: Claude
commit 548684983d31bc2d192068326d8c83b8715cda63
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 14:06:57 2026 -0700
chore(steering): rename phase1..phase5 test prefixes to semantic names
Co-authored-by: Claude
commit 0b325140bbe60f00bbeee26e964ecec9ef654a23
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 14:04:14 2026 -0700
docs(steering): refresh supported-scope and runtime-boundaries sections
commit 49649a58483c49f05f7b4b9dbba53c787105e93b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 14:03:25 2026 -0700
chore(steering): hoist sampling_params inline imports to module level
commit 95ffa52f355f74adfc9dfb207f7460cf05a085e4
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 13 13:41:55 2026 -0700
perf(steering): batch per-layer table writes via index_copy_
commit 27a9fa4bc0226e98c66c6103c955dd62e63d191f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 13 13:38:14 2026 -0700
perf(steering): skip populate when state is unchanged via dirty flag
commit 6d727d4f64eedf2ed39639892876c9bbc11537a6
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 13 02:07:31 2026 -0700
perf(steering): make hash_steering_config 14x faster and cache it
commit 37a9446eea958eac8026963c441eb48f8abb02d9
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 12 16:15:54 2026 -0700
perf: short-circuit steering buffer updates when no state is active
commit b871782c147af79d7af78dc484b13d5853b47153
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 12 15:48:24 2026 -0700
fix: don't create steering manager at all when steering not enabled
commit d7db597e9b0aafec12e72c84889f1ddd3c1f3c49
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 9 01:04:34 2026 -0700
chore: remove implementation docs
commit 2e20d1b9bf897ad593aa075793317cd8431cf2f2
Merge: 897aa5e3e 346383689
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 8 21:16:40 2026 -0700
Merge pull request #77 from RhizoNymph/fix/remove-vestigial-vector-buffers
fix: remove vestigial steering_vector_{hook} GPU buffers
commit 897aa5e3ee85680be7ea0a221eda9ff33789f12f
Merge: 85024d4ca 012dcf9f3
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 8 21:14:56 2026 -0700
Merge pull request #75 from RhizoNymph/docs/remove-vector-buffer-refs
docs: remove steering_vector_{hook} buffer references
commit 85024d4ca4776645e8d0e2c2c88abfd2073c1782
Merge: 9bcf5ce42 224c95539
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 8 21:14:14 2026 -0700
Merge pull request #76 from RhizoNymph/fix/steering-transition-priority
fix: prioritize prefill→decode transitions over new-request deferrals
commit 34638368923a706bab519b7b0b0cd13d6276ca52
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 21:11:33 2026 -0700
fix: remove vestigial steering_vector_{hook} GPU buffers from Phase 1
Closes #74
Co-authored-by: Claude
commit 224c955390fff1f2098266018e6e59a95dafcd0c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 21:09:55 2026 -0700
fix: prioritize prefill→decode transitions over new-request deferrals
Co-authored-by: Claude
commit 012dcf9f3be502fda884fba20c9ce3d9feab7d7f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 21:06:54 2026 -0700
docs: remove steering_vector_{hook} buffer references
Part of #74 — the vector buffers are being removed from the code.
Co-authored-by: Claude
commit 9bcf5ce42b3e98e1b0beefe01aa2c2ffebba405a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 19:28:48 2026 -0700
phase 5 of new models
commit a8cc4349303695d48dce7a09fdeebf722a180b3c
Merge: 183425b41 ff3bf0039
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 8 17:24:56 2026 -0700
Merge pull request #71 from RhizoNymph/feat/named-steering-vectors
feat: named steering vector modules
commit ff3bf00392efa6f575ce8a8b62510561d0ea2558
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 16:37:19 2026 -0700
reject malformed json bodies instead of silently skipping them
commit 173c88b4a440e63caa2d8ecbdcefc3f3bce8085c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 16:10:29 2026 -0700
validate named steering entries
commit 183425b41a16442c123ef0a0171ea3aed0dfd098
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 15:56:13 2026 -0700
feat: more steering models
commit eba7fd616e02e8f2d79d1a56732fd1cc72c37266
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 15:44:50 2026 -0700
fix: gate named vector registry on actually enabling steering
commit b642ba6a42a12ceb4ca45046396f49714528da0b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 23:41:37 2026 -0700
feat: phase 1 of adding new models
commit 3a43d1cc1299b5a621d9b476a39112ecd1daf207
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 23:40:59 2026 -0700
add register steering buffers helper
commit 15b49e31cbf7e7e072dfbde575c8a6bf6b75db16
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 23:36:15 2026 -0700
feat: wire up named steering module registry, CLI args, and docs
Co-authored-by: Claude
commit e9bf1d7455a0a6f06231e04e75c6af6c51e60712
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 23:31:05 2026 -0700
feat: add steering modules CLI args and HTTP API endpoints
Co-authored-by: Claude
commit cb3f0d418d2553afcc320a32d9e394dd2fc076fb
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 21:18:38 2026 -0700
feat: add steering_name field and serving handler resolution
commit 48935407f5cdd564e1ab5fc24c16392b2d72c38f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 21:19:15 2026 -0700
feat: add steering module registry and merge_steering_specs helper
commit 1e50302281f4f2c55481121617cb3f308bd8f055
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 16:00:19 2026 -0700
Add steering docs
commit e27a7cced6a0a8da91c66d6045d4bbe4b24baa6e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 15:43:30 2026 -0700
fix: use httpx for async test and don't fail steering manager test
without available cuda
commit 9a1c7e0badc9dd96a875a43628f07e732ab3560c
Merge: b1b5b93b1 5575ae564
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 7 09:06:42 2026 -0700
Merge pull request #70 from RhizoNymph/feat/prefill-steering
feat: prefill/decode phase-aware steering vectors
commit 5575ae56469a4ef6b4fd0e57e226b30e97e5efce
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 17:05:35 2026 -0700
fix: APC streaming semantics
commit b10dd23f57301f01796794c6869fb9968086777c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 16:20:06 2026 -0700
fix: keep steering prefix-cache keys consistent during deferred
registration
commit 2aa6ac4a0adc16896cc1f562019989f8fef441f0
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 16:08:05 2026 -0700
chore: update docs
commit f9c5245f18e876f8e22e25b1a27258ef7bfefb3f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 16:07:51 2026 -0700
fix: remove post_mlp_post_ln hook point and rename post_mlp_pre_ln to
post_mlp
commit b72b74c0e501b6341fd823d8318d025d5393421f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 16:05:53 2026 -0700
fix: update prompt length before recomputing block hashes to fix
prefix-cache key misses
commit 03d11ebf5fceac2d9c2676e21d2907e421578cce
Merge: 650aaee77 cc205f6ff
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 18:51:00 2026 -0700
Merge pull request #69 from RhizoNymph/fix/steering-streaming-hash-mismatch
fix: remove redundant _reset call from streaming steering path
commit cc205f6ff9dab721720aaf998e306f5f08a72116
Merge: 1de87974e 650aaee77
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 18:50:50 2026 -0700
Merge branch 'feat/prefill-steering' into fix/steering-streaming-hash-mismatch
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 650aaee772d0d81ed4482b55b580533c1adf9159
Merge: e89f20aeb 4b3bb798e
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 18:50:12 2026 -0700
Merge pull request #68 from RhizoNymph/fix/steering-gpu-resident-vectors
fix: store per-request steering vectors on GPU to avoid CPU->GPU copy
commit e89f20aeb20d69ea68bf3951284542e0c46d7405
Merge: 806e39cad 96d73fba5
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 18:50:01 2026 -0700
Merge pull request #67 from RhizoNymph/fix/steering-reject-extra-keys
fix: reject extra keys in scaled steering entries
commit 1de87974e68cf455bea2ce9ab941b24040653d44
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 5 18:37:37 2026 -0700
fix: remove redundant _reset call from streaming steering path
Co-authored-by: Claude
commit 4b3bb798e8604de998cfef30a74955b8090c9f8e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 5 18:30:32 2026 -0700
fix: store per-request steering vectors on GPU to avoid CPU->GPU copy each step
commit 96d73fba5018d133ba239f2e23fd038f788b32a1
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 5 18:30:24 2026 -0700
fix: reject extra keys in scaled steering entries
commit 806e39cad4cb5ad9ff768d2d173d95312ab78bbc
Merge: 54b0ccb9b bed8e050f
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 13:28:12 2026 -0700
Merge pull request #65 from RhizoNymph/test/steering-retry-and-replay-coverage
test: cover steering transition deferral and pending globals replay
commit bed8e050fdb1b6bbe6c43f86b775c220ec247841
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 5 13:21:38 2026 -0700
test: cover steering transition deferral and pending globals replay
Closes two test-coverage gaps identified in the post-merge audit of the
steering feature:
1. TestTransitionUnderCapacity exercises the bookkeeping contract of
_handle_steering_transition when decode registration fails due to
capacity. Verifies the deferred entry is appended with phase="decode",
_req_steering_phase[req_id] is set to "decode" regardless, and
get_row_for_config falls back to row 2 until the next retry succeeds.
2. TestPendingGlobalsReplay verifies that phase-labeled entries captured
into _pending_steering_globals before SteeringManager lazy init land
in the matching global_{base,prefill,decode}_vectors dicts rather
than collapsing into a single tier, including end-to-end via
populate_steering_tables.
Also drops two unused-variable / nested-if lint issues in
TestPreemptionResumption that were blocking pre-commit on this file.
Co-authored-by: Claude
commit 54b0ccb9b1405c3feb388222388becc328fae091
Merge: 735fc4d91 416c9b388
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Apr 4 17:26:36 2026 -0700
Merge pull request #64 from RhizoNymph/fix/steering-preempt-resume-config-leak
fix: reset steering phase on preemption-resumption to prevent refcount leak
commit 416c9b38849fe2ed88b46a82345e32f7499d20b4
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Apr 4 16:33:53 2026 -0700
fix: reset steering phase on preemption-resumption to prevent refcount leak
When a request with per-request steering is preempted (e.g. via
reset_prefix_cache() which fires on every global steering update) and
later resumed, num_computed_tokens is reset to 0 but the model runner's
_req_steering_phase and the SteeringManager's registered configs are
NOT cleaned up. On resumption the request re-prefills using the global
prefill row (not its per-request row), and the prefill->decode
transition double-registers the decode config -> permanent row leak
on finish.
Add _reset_steering_for_resumption helper that releases the stale
decode config, drops any stale deferred entries, and re-registers the
prefill config. Call it from both the v1 cached-reqs resumption path
(guarded by resumed_from_preemption) and the v2 _update_streaming_request
path.
Co-authored-by: Claude
commit 735fc4d9183af796a556ca31937b9a3b7d162fef
Merge: 7d566afc7 cde991759
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Apr 4 14:42:58 2026 -0700
Merge pull request #62 from RhizoNymph/fix/steering-api-empty-hook
fix: drop empty hook entries in steering _normalize_spec
commit 7d566afc7bb163838da34957ddd8786c5766190b
Merge: 6c56b5aa7 8ed116494
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Apr 4 14:32:36 2026 -0700
Merge pull request #61 from RhizoNymph/fix/steering-manager-phase-validation
fix: validate phase in populate_steering_tables
commit 6c56b5aa7dfaab2834aae67d0cc8d284af831630
Merge: b8f783c89 729f196f7
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Apr 4 14:31:59 2026 -0700
Merge pull request #60 from RhizoNymph/fix/steering-prefill-decode-cross-validation
fix: validate prefill/decode steering dimensions without base
commit cde991759948516e5afe1be2b4dd9664a19619a8
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Apr 4 13:37:59 2026 -0700
fix: drop empty hook entries in steering _normalize_spec
Co-authored-by: Claude
commit 8ed116494e6c30dd9696e64ef8560af8dd8696b2
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Apr 4 13:36:57 2026 -0700
fix: validate phase in populate_steering_tables
Co-authored-by: Claude
commit 729f196f757e8fe88e979ae07cc211e0fdcef8dd
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Apr 4 13:35:05 2026 -0700
fix: validate overlapping prefill/decode steering dimensions without base
Co-authored-by: Claude <noreply@anthropic.com>
commit b8f783c898a9ab2b8c66807be72da8b071b8cd23
Merge: d4e78d422 546d8e14c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Apr 3 23:23:38 2026 -0700
Merge pull request #58 from RhizoNymph/fix/steering-hash-invalidation
fix: invalidate cached steering hashes on streaming sampling_params update
commit d4e78d4226c5354f61d927d23c2440221ce0adc9
Merge: 2fddb12f4 9377c7141
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Apr 3 22:38:04 2026 -0700
Merge pull request #57 from RhizoNymph/fix/steering-reject-when-disabled
fix: reject per-request steering when enable_steering is off
commit 546d8e14c92a543168a9dda37e6858a28a416a70
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 21:34:06 2026 -0700
fix: invalidate cached steering hashes on streaming sampling_params update
Co-authored-by: Claude
commit 9377c7141c95b51cf8eb92bb3d0d2633674e6eec
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 21:32:23 2026 -0700
fix: reject per-request steering when enable_steering is off
Co-authored-by: Claude
commit 2fddb12f4e86f813bdeba582973524d48fa77a76
Merge: a681974aa 5959d8fe3
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Apr 3 16:52:06 2026 -0700
Merge pull request #56 from RhizoNymph/fix/steering-streaming-hashes
fix: refresh steering hashes when updating streaming requests
commit a681974aaae738cafe54b883166b5fc063c59724
Merge: 4dc95f529 3526be8be
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Apr 3 16:51:29 2026 -0700
Merge pull request #55 from RhizoNymph/fix/steering-forbid-extra
fix: reject unknown fields in SetSteeringRequest
commit 5959d8fe325beb50a7294e123f4744f74b08ed0d
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 15:40:08 2026 -0700
fix: refresh steering hashes when updating streaming requests
Co-authored-by: Claude
commit 3526be8be85624eb4506000ac540e9c958ba9c26
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 13:19:23 2026 -0700
fix: reject unknown fields in SetSteeringRequest to catch legacy scales
Co-authored-by: Claude
commit 4dc95f52958069c4c431b7c0ebf1308d6b5ff1fe
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 01:05:58 2026 -0700
fix: fail on stale cache after steering change and relax decode-only admission
Co-authored-by: Claude
commit e2ef338ed7bfe52bcabd343f2cdcd82f7faca9c7
Merge: da2fec961 6288714c5
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 23:25:29 2026 -0700
Merge pull request #54 from RhizoNymph/fix/steering-atomic-replace
fix: make steering replace=true atomic via single worker RPC
commit da2fec96126507d48e3d5fa3b588697f6238e8de
Merge: ee488edac bb24d9947
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 23:22:06 2026 -0700
Merge pull request #53 from RhizoNymph/fix/steering-openai-plumbing
fix: expose full steering API in OpenAI protocol models
commit 6288714c5d3ff3da62d0736a766b4974d501b1f3
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 23:07:33 2026 -0700
fix: make steering replace=true atomic via single worker RPC
Co-authored-by: Claude
commit bb24d99473b12a367a962f25e5c77de36a2b5d7d
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 23:05:25 2026 -0700
fix: expose full steering API surface in OpenAI protocol models
Co-authored-by: Claude
commit ee488edacddfb8da257320656769d978a0db92d2
Merge: 56f66ad7b 442e9eba8
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:34:01 2026 -0700
Merge pull request #51 from RhizoNymph/fix/steering-request-attrs
fix: remove unused steering vector attributes from Request
commit 56f66ad7b7e31fafad1b5b4e0c9e04bfc51debf9
Merge: 46e522e0f 9b2997d1c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:29:20 2026 -0700
Merge pull request #49 from RhizoNymph/fix/steering-scheduler-capacity
fix: add decode-only capacity check and transition prediction for WAITING requests
commit 46e522e0f5fbd007b8c1b1dff6029f37481f341a
Merge: ece7238fb 7623fbe3f
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:26:51 2026 -0700
Merge pull request #48 from RhizoNymph/fix/steering-model-runner-guards
fix: add error handling for steering registration and fix backfill refcounting
commit ece7238fb087b3d498868085fa8841c47ef18292
Merge: 9ae9bcd37 6e6deadb1
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:25:48 2026 -0700
Merge pull request #47 from RhizoNymph/fix/steering-device-mismatch
fix: resolve GPU+CPU device mismatch in steering table population
commit 9ae9bcd37ac590c8f830af39bcc316fe55fe135a
Merge: 5d19f7f10 e6a9ffaf1
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:24:01 2026 -0700
Merge pull request #50 from RhizoNymph/fix/steering-replace-clear
fix: honor replace=True in set_steering_vectors when no new vectors provided
commit 5d19f7f10f2d53aad4759bfbfd24e5a0639a6a4d
Merge: 000d00f48 dd79f0844
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:22:54 2026 -0700
Merge pull request #52 from RhizoNymph/fix/steering-cache-key-phase
fix: disambiguate boundary block cache keys for prefill vs decode steering
commit 442e9eba84ba37a7e0439e43dd89509915326a17
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:31:03 2026 -0700
fix: remove unused steering vector attributes from Request to prevent memory waste
Co-authored-by: Claude
commit e6a9ffaf1031efd4b99fb8c7eac3f852f604a87b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:29:30 2026 -0700
fix: honor replace=True in set_steering_vectors when no new vectors provided
Co-authored-by: Claude
commit dd79f084480a4092c5493303bb7ebf2940ab55e5
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:29:00 2026 -0700
fix: disambiguate boundary block cache keys for prefill vs decode steering
Co-authored-by: Claude
commit 9b2997d1cb33475567c791c3a03624972e0806c5
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:27:06 2026 -0700
fix: add decode-only capacity check and transition prediction for WAITING requests
Co-authored-by: Claude
commit 7623fbe3fb789b1776e1f0c4d26e70ed143de71a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:27:05 2026 -0700
fix: add error handling for steering registration and fix backfill refcounting
Co-authored-by: Claude
commit 6e6deadb1e9fd56a13fd72ed8adb22acb1979317
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:27:04 2026 -0700
fix: resolve GPU+CPU device mismatch in steering table population
Co-authored-by: Claude
commit 000d00f48bbba901b7d4fa682c75eb6439be9db4
Merge: dfed3d2bf f9130cf07
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 1 18:34:56 2026 -0700
Merge pull request #45 from RhizoNymph/fix/steering-lazy-init-capacity
fix: gracefully handle capacity exhaustion during steering lazy init
commit f9130cf07d50a9ff0c02d89e4e5f22fb4c373d53
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 1 16:24:56 2026 -0700
fix: defer steering registrations on capacity exhaustion instead of dropping
Replace the silent fallback-to-global approach with a proper deferral
mechanism. Failed registrations during lazy init and prefill-to-decode
transitions are now appended to a unified
`_pending_steering_registrations` list that is retried with priority
before new admissions on every step.
The retry loop also filters stale entries:
- Entries for finished requests are dropped (prevents row leaks).
- Entries whose request changed phase are dropped (e.g. prefill
deferred but request already transitioned to decode).
This subsumes the decode-only `_pending_decode_registrations` list
and the row-leak fix from PR #46.
commit 251cabe638c7b33d2dcbf27e1da9754dd2e63369
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 1 01:16:28 2026 -0700
fix: gracefully handle capacity exhaustion during steering lazy init
commit dfed3d2bfd1557d7c8c3925dd5de8224c1e9e91d
Merge: 51672b8f1 41135ae57
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 1 12:53:25 2026 -0700
Merge pull request #44 from RhizoNymph/fix/steering-swap-states-hash
fix: swap steering hashes in InputBatch.swap_states()
commit 51672b8f1ef164970470580e380534f85ec3b35a
Merge: 47365775f f2638d4b7
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 1 12:36:32 2026 -0700
Merge pull request #43 from RhizoNymph/fix/steering-dimension-cross-validation
fix: validate matching dimensions between base and phase steering vectors
commit 41135ae5709c78d4639a0b0148967b491a38894e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 1 01:16:27 2026 -0700
fix: swap steering hashes in InputBatch.swap_states()
Co-authored-by: Claude
commit f2638d4b7f2f6a9136c11362f08b89b9873d42f3
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 1 01:14:35 2026 -0700
fix: validate matching dimensions between base and phase steering vectors
commit 47365775f6c7310d6381fb5495777950fd729997
Merge: 981ad5ac8 ebeff444f
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 23:59:48 2026 -0700
Merge pull request #42 from RhizoNymph/docs/steering-bugfix-round2
docs: update steering docs for deferred cleanup and pre-init status
commit 981ad5ac85d2c93c3ae1f356f148ea097350481d
Merge: 7431847f1 f06f5db30
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 23:59:35 2026 -0700
Merge pull request #41 from RhizoNymph/fix/steering-status-pending-globals
fix: report pending phase-specific globals in steering status
commit 7431847f1cd2647ba96829395b2037f2317f520f
Merge: f04aaf36d e1a6a2ef5
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 23:59:12 2026 -0700
Merge pull request #40 from RhizoNymph/fix/steering-deferred-decode-leak
fix: track req_id in deferred decode registrations to prevent leak
commit f04aaf36d143c040b2b652403de3f0449a5d3d62
Merge: 7e41715be e9d8f110c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 23:58:37 2026 -0700
Merge pull request #39 from RhizoNymph/fix/steering-boundary-block-hash
fix: include decode hash in boundary block cache keys
commit ebeff444f4bd59751824305843481a4e92178ece
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 23:34:00 2026 -0700
docs: update steering docs for deferred cleanup and pre-init status
Co-authored-by: Claude
commit e9d8f110c39446cd8c4e9bcbaddc91c82236962f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 23:29:38 2026 -0700
fix: include decode hash in boundary block cache keys
Co-authored-by: Claude
commit f06f5db3000d6fb29d21219538cc4d9c43030e04
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 23:27:52 2026 -0700
fix: report pending phase-specific globals in steering status
commit e1a6a2ef5da949d4636d98779cb146b803cb376b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 23:26:48 2026 -0700
fix: track req_id in deferred decode registrations to prevent leak
commit 7e41715be28060d2ebb7dd97bff0aef72cf04b27
Merge: 5edfb74f2 0c7e77579
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 19:44:19 2026 -0700
Merge pull request #38 from RhizoNymph/docs/steering-bugfix-updates
docs: update steering docs for bugfix PRs #35-#37
commit 5edfb74f2d3657093e316c5fe673cf58e6a1ce99
Merge: a7bdd3fe1 2f0a0c960
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 19:44:01 2026 -0700
Merge pull request #36 from RhizoNymph/fix/steering-scheduler-transition-capacity
fix: transition-aware steering capacity and decode-start admission
commit a7bdd3fe1ad5f2a3c4e25a67d4da9b1617a34992
Merge: 2830adbf0 b2d8d475c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 19:08:27 2026 -0700
Merge pull request #35 from RhizoNymph/fix/steering-worker-status-validation
fix: remove double validation and add phase-specific norms to steering status
commit 2830adbf014067736c04afdc1ddfdb11c898009c
Merge: c01e6dce4 5cb702a2a
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 19:01:21 2026 -0700
Merge pull request #37 from RhizoNymph/fix/steering-model-runner-transition
fix: graceful steering transition + phase-tracked release
commit 0c7e77579e3cab4e5cdc6987e43a376481bb647d
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 18:58:04 2026 -0700
docs: update steering docs for transition-aware capacity, deferred registration, and phase-specific norms
commit 5cb702a2aa253defaa567fbda72e8c2e26650267
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 18:30:11 2026 -0700
fix: graceful steering transition with deferred registration and phase tracking
Co-authored-by: Claude
commit b2d8d475c0010910fd2b680d9460dc9caa715708
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 17:59:07 2026 -0700
fix: remove double validation and add phase-specific norms to steering status
Co-authored-by: Claude
commit 2f0a0c96055e7330687ad1e101059eb3fa311040
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 17:58:31 2026 -0700
fix: transition-aware steering capacity counting and decode-start admission check
Co-authored-by: Claude
commit c01e6dce4032a00a962d1b257b3f907e327f5b54
Merge: 5ab47798d 84eeb8b21
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 01:05:21 2026 -0700
Merge pull request #34 from RhizoNymph/fix/steering-decode-cache-block-hash
fix: include decode steering hash in generated cache block keys
commit 5ab47798d2d6a6778e61610ddde0fcaad2f80240
Merge: c3bda6dc5 2b8a2a6ec
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 01:04:14 2026 -0700
Merge pull request #32 from RhizoNymph/fix/steering-scheduler-single-phase-capacity
fix: scheduler steering capacity counts only active phase
commit c3bda6dc5f8da7e9dcae49faba5c3b69aed6a67a
Merge: dd9e5b185 35c55f9eb
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 01:02:59 2026 -0700
Merge pull request #33 from RhizoNymph/fix/steering-lazy-init-decode-registration
fix: register decode steering configs during lazy manager init
commit 84eeb8b21d7435d91655d6a19aa7b33f7de16800
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 00:01:30 2026 -0700
fix: include decode steering hash in generated cache block keys
commit 35c55f9eb69bca17dd3f0c80830b419921bef073
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 30 23:42:06 2026 -0700
fix: register decode steering configs during lazy manager init
commit 2b8a2a6eccde4b10f8565bbc62e13c8024a85327
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 30 23:41:46 2026 -0700
fix: scheduler steering capacity counts only active phase
commit dd9e5b1852d053f31535ec75c72d973095b07424
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 30 23:09:17 2026 -0700
docs: update steering docs for PR #29, #30, #31 post-merge fixes
commit 63089a37e19e877625de97ad69e62dbf8f0f4b42
Merge: dbe8d1880 dcdf60110
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Mon Mar 30 23:00:16 2026 -0700
Merge pull request #31 from RhizoNymph/fix/steering-buffer-overwrite
fix: stop writing phase-specific globals to shared steering buffers
commit dbe8d18809e04fcfd7f959d90a4f7d2eed95adac
Merge: 5100a10d4 bd69fae0b
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Mon Mar 30 22:59:26 2026 -0700
Merge pull request #30 from RhizoNymph/fix/steering-decode-start-registration
fix: register decode steering config when request starts past prefill
commit 5100a10d4a0d95229c3b6587484b1a122bc08ae2
Merge: 8d1c1fe47 3de429a5a
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Mon Mar 30 22:57:48 2026 -0700
Merge pull request #29 from RhizoNymph/fix/steering-malformed-entry-validation
fix: reject malformed scaled steering entries with 4xx
commit dcdf60110fd6e89e8dd9a3128a8b3a71049de610
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 15:29:56 2026 -0700
fix: stop writing phase-specific globals to shared steering buffers
commit bd69fae0b51f0c2e75f323955d62265b4623db9e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 15:13:54 2026 -0700
fix: register decode steering config when request starts past prefill
commit 3de429a5a84be3952759555b49ca01f22b05e37f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 14:40:26 2026 -0700
fix: reject malformed scaled steering entries with 4xx instead of 500
commit 8d1c1fe47dac02e3b3f07ef324967a969d66790d
Merge: 9d8d7a4e9 4f7f350dd
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 14:13:33 2026 -0700
Merge pull request #27 from RhizoNymph/fix/steering-clear-pending-globals
fix: clear pending steering globals on clear_steering_vectors
commit 9d8d7a4e99566fb6c445ef46de477110d0bf5f0c
Merge: c57ff54da d6efa42ce
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 14:11:58 2026 -0700
Merge pull request #28 from RhizoNymph/fix/steering-scheduler-phase-capacity
fix: count steering capacity by (hash, phase) pairs to match worker row allocation
commit 4f7f350dd603817a7a0bbda59431220197550988
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 12:08:15 2026 -0700
fix: clear pending steering globals on clear_steering_vectors to prevent stale replay
commit d6efa42ce8806af8846600a0ec6d39dfc5609952
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 12:07:57 2026 -0700
fix: count steering capacity by (hash, phase) pairs to match worker row allocation
commit c57ff54daf3f93216b8d609b9719ed38e918339e
Merge: 849fa0577 2f3b14e09
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 11:27:02 2026 -0700
Merge pull request #25 from RhizoNymph/fix/steering-pending-globals-queue
fix: preserve phase-specific global vectors set before steering manager init
commit 2f3b14e094f4f85932b7650819b3cb0e35716583
Merge: 55a9da811 849fa0577
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 11:26:51 2026 -0700
Merge branch 'feat/prefill-steering' into fix/steering-pending-globals-queue
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 849fa0577a9ca55e201830e59d351140414c21c6
Merge: 018a7adcc 229ce88f3
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 11:24:05 2026 -0700
Merge pull request #24 from RhizoNymph/fix/steering-scheduler-phase-detection
fix: use num_prompt_tokens for steering phase detection in scheduler
commit 018a7adcc417fa3cd8647954752fbade1cdbb31c
Merge: 007de86fc 91b89b480
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 11:16:11 2026 -0700
Merge pull request #26 from RhizoNymph/fix/steering-phase-keyed-rows
fix: key steering table rows by (config_hash, phase) to prevent cross-phase aliasing
commit 91b89b4805a7a5a3464cd2d605211f4d8e2c1189
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 11:11:10 2026 -0700
fix: key steering table rows by (config_hash, phase) to prevent cross-phase aliasing
commit 55a9da811b0ae47e46816eb8588bf234d8130835
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 00:38:21 2026 -0700
fix: queue phase-specific global vectors for replay on steering manager lazy init
commit 229ce88f3392682b44cc69f7b5ca12e827acd14d
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 00:35:41 2026 -0700
fix: use num_prompt_tokens for steering phase detection in scheduler
commit 007de86fc362ed6d86dc4bbebbf5e5a41581851e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 00:13:51 2026 -0700
docs: update steering docs for prefill steering (Phase 4)
- ROADMAP.md: Mark Phase 4 as DONE with full implementation summary
- STEERING.md: Update to three-tier model, phase-aware table layout,
remove stale "decode-only" language, update key files table
- MVP_IMPLEMENTATION.md: Add Phase 4 lessons learned (prefix cache,
lazy init race condition, phase detection, additive composition,
co-located scales, scheduler dual-hash admission, test subtleties)
- API.md: Update to three-tier format, co-located scales, remove
old scales field, update examples
commit 8f055b92c5f82d1283a2b186619bcb2b128cf22a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 23:59:30 2026 -0700
fix: register steering configs on first-step manager init + fix additive test
The SteeringManager is lazily created in _update_steering_buffers, but
config registration happens in _update_states which runs before. On the
first request, registration was silently skipped because the manager
didn't exist yet. Fix: after creating the manager, scan the batch and
register any pending configs.
Also fix test_additive_composition which incorrectly assumed base(250) +
decode(250) equals base(500) — steering_vectors affects both phases, so
prefill effective vectors differed between the two approaches.
commit f15d55b8d0748a38b4e5786c09afa6430d9d28e7
Merge: 9c1dca9f5 2917406a5
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 23:21:56 2026 -0700
Merge pull request #23 from RhizoNymph/feat/prefill-steering-tests
test: add prefill steering integration tests
commit 2917406a5d1e53de9fbf4db4fbbc1adde087777a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 22:59:26 2026 -0700
test: add prefill steering integration tests and fix existing test API calls
Co-authored-by: Claude
commit 9c1dca9f547e52defa4782326a2b6777e7940f8f
Merge: 2be915751 45100595f
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:43:26 2026 -0700
Merge pull request #21 from RhizoNymph/feat/prefill-steering-manager
feat: phase-aware SteeringManager and GPU model runner
commit 45100595fc03a46d5407edab6240971e39b3d2ed
Merge: 1beb47cc2 2be915751
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:35:19 2026 -0700
Merge branch 'feat/prefill-steering' into feat/prefill-steering-manager
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 2be91575101afb56f9952093c4c40b5f5ce00174
Merge: a286bd9d6 ec9e660b3
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:34:15 2026 -0700
Merge pull request #22 from RhizoNymph/feat/prefill-steering-scheduler
feat: prefill steering scheduler, API, and worker (WS4)
commit ec9e660b307566bba43e5230cd2321e437010510
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 21:33:10 2026 -0700
feat: wire prefix cache invalidation for prefill-affecting steering changes
Replace the TODO with actual calls to engine.reset_prefix_cache() when:
- Global base or prefill vectors are set (affects prefill KV cache)
- replace=True is used (clears all tiers including prefill)
- All steering is cleared via /v1/steering/clear
commit a286bd9d660ad473232f9b60a0f04706494375a5
Merge: 67b6d9d65 89e056fb6
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:25:44 2026 -0700
Merge pull request #19 from RhizoNymph/feat/prefill-steering-cache-keys
feat: prefill steering prefix cache key integration
commit 89e056fb6bf5884135ef2a26c7e935da5c988b85
Merge: 68cce443a 67b6d9d65
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:22:09 2026 -0700
Merge branch 'feat/prefill-steering' into feat/prefill-steering-cache-keys
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 67b6d9d653da3c84fae431aad0a14e06b77ce85e
Merge: b1b5b93b1 46665f131
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:18:16 2026 -0700
Merge pull request #20 from RhizoNymph/feat/prefill-steering-types
feat: three-tier steering types with prefill/decode separation
commit b7d55a044a2fde9ae18c587711d8ddac6b03b9a6
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 21:06:52 2026 -0700
feat: implement prefill steering scheduler, API, and worker changes
- Scheduler: dual-hash admission control tracks union of active hashes
(prefill hash for prefill-phase requests, decode hash for decode-phase).
New requests must have capacity for both their prefill and decode hashes.
- API router: three-tier global steering (vectors, prefill_vectors,
decode_vectors) with co-located scale normalization. Removed old
scales field. Uses kwargs-based collective_rpc for worker calls.
- Worker: three-tier set_steering_vectors with base/prefill/decode tiers.
Extracted validation, buffer application, and manager notification into
separate helper methods. Supports replace=True across all tiers.
- Tests: comprehensive dual-hash scheduler admission tests, three-tier
API router tests with co-located scale format, and three-tier worker
tests including replace and cross-tier validation.
- Docs: updated steering.md with dual-hash admission details and
three-tier HTTP API examples.
Co-authored-by: Claude
commit 1beb47cc2be30910c88e5b2aeba266a6e6c65036
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 21:01:12 2026 -0700
feat: phase-aware SteeringManager and GPU model runner integration
Co-authored-by: Claude
commit 46665f131579bb24b50b0786d2424e9cd28a45fe
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 20:41:31 2026 -0700
feat: add three-tier steering types with prefill/decode phase separation
Co-authored-by: Claude
commit 68cce443a84085f7f591e00e9967d1a62fed5f93
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 20:38:47 2026 -0700
feat: add prefill steering config hash to prefix cache block keys
Add _gen_steering_extra_hash_keys() to include per-request prefill
steering config hashes in block hash extra keys. This ensures that
blocks computed under different prefill steering produce different
cache entries, with zero impact when steering is not used.
Co-authored-by: Claude
commit b1b5b93b15ad8bc2bcea10a786b9270398c66b3e
Merge: 03868735b 21c6acbb4
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 18:41:07 2026 -0700
Merge pull request #18 from RhizoNymph/feat/steering-hook-points
feat: Phase 3 — multiple steering intervention points
commit 21c6acbb4b55f6cb5244fb67522514660ca2abfa
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 18:36:12 2026 -0700
docs: add Phase 3 lessons learned to implementation notes
Key findings:
- Splitting ops are not required for buffer correctness — custom op
opacity is sufficient, graph partitioning is a separate concern
- Inline tensor math triggers AOT autograd pickle failures due to
Enum WeakValueDictionary in traced graph metadata
- Always-allocate beats opt-in when memory is trivial vs model size
- Hook points at same residual state exist for SAE position matching
- Breaking backward compat cleanly beats merging logic
Co-authored-by: Claude
commit 15f34a6d006cd509fe10297e007a9a658697d5c6
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 18:33:51 2026 -0700
docs: update steering docs for simplified hook point design
- Remove references to --steering-hook-points CLI flag
- Remove references to steering_hook_vectors (legacy separate field)
- Document that all 4 hook points are always allocated unconditionally
- Document that apply_steering is NOT a splitting op (zero graph partitions)
- Update all API examples to use the hook-point-aware steering_vectors format
Co-authored-by: Claude
commit 6837fabf7a76a86eebdb0546a7c03b52ec1e2204
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 18:26:07 2026 -0700
chore: remove --steering-hook-points CLI flag and config field
All 4 hook point buffers are always allocated unconditionally.
The memory cost is trivial (~11 MB at max_configs=16 for 26 layers)
and there are no graph partitions, so there is no reason to control
which hook points are active.
Co-authored-by: Claude
commit 5369eead8d2575202add671ba975ccffbc9b1be3
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 17:49:26 2026 -0700
feat: remove graph partitions, always allocate all hook point buffers
- Remove apply_steering from splitting_ops — the custom op is still
opaque to the tracer (preventing constant-folding) but no longer
partitions the compiled graph
- Always allocate all 4 hook point buffers unconditionally (~11 MB for
26 layers at max_steering_configs=16) — no hasattr guards needed
- Remove active_hook_points parameter from Gemma3DecoderLayer
Co-authored-by: Claude
commit 45dbf1fef9bc0e0a3e3d5f2a2aef033703199cfc
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 16:14:47 2026 -0700
refactor: remove backward-compat steering_vectors, require hook points
steering_vectors is now always dict[str, dict[int, list[float]]]
(hook_point → layer → vector). Removes the old flat format, the
separate steering_hook_vectors field, and all merging logic.
Co-authored-by: Claude
commit 3d4022dd3f3f795bc2ebe48c5cb618cfece31e59
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 15:05:39 2026 -0700
docs: update steering docs for Phase 3 hook points
Co-authored-by: Claude
commit cf969cb1bdf8e1e38cdfebdb6ed16e6304a73c9c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 14:57:47 2026 -0700
fix: restore hook-point-aware api_router and protocol after merge
The infrastructure agent's PR overwrote the API agent's changes to
api_router.py and protocol.py. This restores:
- hook_vectors field on SetSteeringRequest
- vectors default_factory=dict (no longer required since hook_vectors
can supply vectors)
- Hook-point-aware set_steering that wraps legacy vectors under
post_mlp_pre_ln before passing to workers
- Updated tests to match new API format
Co-authored-by: Claude
commit 11b0da7f4de47509e7ff794f90bc41616478d986
Merge: 5fd0e3929 8fbe4ddb4
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 23:29:20 2026 -0700
Merge pull request #15 from RhizoNymph/feat/steering-hook-points-manager-runner-worker
feat: multi-hook-point support for SteeringManager, ModelRunner, WorkerBase
commit 8fbe4ddb4381681807fc6d02949f5a1387ec6b74
Merge: 1c8717e30 5fd0e3929
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 23:29:05 2026 -0700
Merge branch 'feat/steering-hook-points' into feat/steering-hook-points-manager-runner-worker
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 5fd0e39291aebc935402364a942d8cacf1b87cd8
Merge: c80df1880 ee599fed1
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 23:27:46 2026 -0700
Merge pull request #16 from RhizoNymph/feat/steering-hook-points-api
feat: add steering_hook_vectors API support
commit c80df1880ea3ff2960474885a37b11480c8baa1c
Merge: 54e3357d2 119f87a2e
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 23:25:17 2026 -0700
Merge pull request #17 from RhizoNymph/feat/steering-gemma3-hook-points
feat: update Gemma3 model for multiple steering hook points
commit ee599fed1209f693691f4edf1392dfa745f4f63a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:44:19 2026 -0700
docs: update steering feature doc for hook-point API support
commit ffb372c34ae61ba81c1820951453f41a2b6455a0
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:43:20 2026 -0700
feat: add steering_hook_vectors support to API layer
Add steering_hook_vectors field across the API layer to support
multiple hook points for activation steering vectors.
Changes:
- SamplingParams: new steering_hook_vectors field with validation
that checks hook point names against VALID_HOOK_POINT_NAMES and
reuses _validate_layer_vector_dict for inner dict validation
- Request: new steering_hook_vectors field populated from
sampling_params, steering_config_hash updated to build canonical
representation merging both steering_vectors and
steering_hook_vectors
- ChatCompletionRequest/CompletionRequest: new steering_hook_vectors
field passed through to SamplingParams.from_optional
- SetSteeringRequest: new hook_vectors field, vectors field now
defaults to empty dict for backward compatibility
- api_router set_steering: builds hook-point-aware dict merging
legacy vectors (as post_mlp_pre_ln) with explicit hook_vectors,
validates hook point names, passes new format to collective_rpc
commit 1c8717e30ea7ebe65d726f384c05921b71bb7f98
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:37:33 2026 -0700
feat: update SteeringManager, ModelRunner, WorkerBase for multi-hook-point steering
Co-authored-by: Claude
commit 119f87a2e2f1894dcac27cb042ee9fa59a02d16b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:29:40 2026 -0700
feat: update Gemma3 model to support multiple steering hook points
Co-authored-by: Claude
commit 54e3357d26d440f423fe2f2a818193572b849c73
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:21:16 2026 -0700
feat: add SteeringHookPoint enum and --steering-hook-points CLI arg
Co-authored-by: Claude
commit 03868735bd343e1943f02ba7e3ace08f9e718a1f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 20:11:52 2026 -0700
docs: add per-request steering lessons learned to implementation notes
commit 2d8278f1f792ab8416b8affd70f69de280ba248a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 19:21:58 2026 -0700
fix: steering hash overflow, add concurrent CUDA graph integration test
- Mask steering_config_hash to fit np.int64 (was overflowing with
full 64-bit unsigned SHA-256 hash)
- Add test_per_request_steering_concurrent_with_cuda_graphs: sends
3 requests (unsteered, +500, -500) in same batch, verifies per-request
steering works with torch.compile + CUDA graphs active, then verifies
unsteered determinism on a second batch
commit 6bae6a4ad51baebc1d7575e320241cdd2b5bfa25
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 18:40:38 2026 -0700
docs: update steering docs for per-request implementation
commit 0265546ffca239b75ca32147f5dcb9bbb4063de5
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 18:34:12 2026 -0700
docs: add OVERVIEW.md and steering feature documentation
commit 4c57a2d31b2c647491dff7f1339cd81ad1fe9adf
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 18:12:01 2026 -0700
test: add scheduler admission, worker steering, and integration tests
commit 066839d90d3ce9c547c1fbfc9fdf9478ed4fc131
Merge: 72b7b0c68 142432b7c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 17:58:51 2026 -0700
Merge pull request #13 from RhizoNymph/feat/per-request-steering-runner
feat: integrate SteeringManager into model runner and worker_base
commit 72b7b0c6899515ce25da7846e46281f24378ea3a
Merge: b2e770b7d 2e914cfc7
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 17:58:21 2026 -0700
Merge pull request #11 from RhizoNymph/feat/per-request-steering-tests
test: SteeringManager and indexed-gather op unit tests
commit 142432b7cd911ee029ae13a3b4d29c2e713f12b7
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 17:51:50 2026 -0700
feat: integrate SteeringManager into model runner and worker_base
commit b2e770b7d38de65611808cb45ffd9c3d3c24e60f
Merge: 5d9c9f015 23198f1c0
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 17:43:22 2026 -0700
Merge pull request #12 from RhizoNymph/feat/per-request-steering-manager
feat: SteeringManager, InputBatch tracking, scheduler admission control
commit 23198f1c085115e3935edd8c4dfe6c8574c84fe7
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 17:24:10 2026 -0700
feat: add SteeringManager, InputBatch tracking, and scheduler admission control
commit 2e914cfc75f20e5b90b1801e7953aff75da585ad
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 17:02:33 2026 -0700
test: add SteeringManager and indexed-gather op unit tests
commit 5d9c9f015dd9d21c372594ac2ab237091f3e52ee
Merge: 9e535b083 2066a7e1c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 16:47:55 2026 -0700
Merge pull request #9 from RhizoNymph/feat/per-request-steering-config
feat: add SteeringConfig, SamplingParams field, and API protocol support
commit 9e535b08395e3dd40e17f6bb754a24e93432af28
Merge: 602474f3c 10c95f5d4
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 16:46:03 2026 -0700
Merge pull request #10 from RhizoNymph/feat/per-request-steering-op
feat: update steering op to indexed gather with table/index buffers
commit 2066a7e1c90fcca42816bb15e00180b9283698bb
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 16:04:51 2026 -0700
feat: add steering config, SamplingParams field, and API protocol support
commit 10c95f5d48ea45d213d2020717970467b7c93ba4
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 16:10:09 2026 -0700
feat: update steering op to indexed gather, add table/index buffers to Gemma3
commit 602474f3c947718c34b57029b45d7717c61dd2f9
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 14:08:39 2026 -0700
remove todo doc for mvp
commit 7619f8ae91273cf022892d1543a5bace29346784
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 14:08:28 2026 -0700
fix: change injection point to raw residual stream not post layernorm
commit 98408699f80226539d908b3bb690af8f6e853d9e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Mar 26 22:47:14 2026 -0700
fix: decode-only steering with torch.compile and CUDA graphs
Replace ForwardContext-based num_decode_tokens read (constant-folded by
torch.compile) with persistent per-layer mask buffers updated in-place
by the model runner before each forward pass.
- Custom op takes 3 args (hidden, vec, mask) and is a splitting op
- Each decoder layer registers a steering_decode_mask buffer
- Model runner discovers steerable layers and updates masks from
scheduler output (decode tokens = 1 scheduled token, ordered first)
- Add num_decode_tokens to TritonAttentionMetadata and
FlashAttentionMetadata
- Increase test steering magnitude for dummy-weight models
- Add prefix cache reset after clearing steering in E2E test
Co-authored-by: Claude
commit c19a78587d29270c901c4fd66453a87eef049e8b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Mar 26 13:36:55 2026 -0700
docs/update temporary steering docs
commit d51bd507c693f10372676030b3b97985c47ff561
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Mar 26 00:41:13 2026 -0700
fix: typed steering exceptions, remove GET lock, cache steerable layers
Co-authored-by: Claude
commit 6c93d113d849d6cf474a6a8c90b617c37cab609a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Mar 26 00:20:18 2026 -0700
fix: cast steering vector dtype to match hidden states
Co-authored-by: Claude
commit 531ad064402b875e1d8d11f3062e2c0165db61b2
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Mar 25 18:45:38 2026 -0700
fix: register steering as custom op for torch.compile safety
Co-authored-by: Claude
commit 5c4ec89f2fbb90e6aa4912b2d27c2684f01c609f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Mar 25 18:43:51 2026 -0700
fix: return clear error for empty steering vectors request
Co-authored-by: Claude
commit e76810451eb76e8dc5d63ada1964f9dc1f98c423
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 24 18:44:25 2026 -0700
temporary steering docs
commit 1f2c5140c5b7fde10e7cffd4ba452207a115ba4f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 24 18:15:36 2026 -0700
add steering unit tests, fix test bugs
- Add 65 unit tests across 4 files covering: SetSteeringRequest
protocol validation, get_num_decode_tokens forward context utility,
WorkerBase steering methods (set/clear/status with mock model,
validation errors, edge cases), and API router endpoints (mock
engine, all HTTP status codes, replace flag, attach/detach).
- Fix test_steering.py: use llm.llm.generate() to get RequestOutput
and compare only completion token_ids, not the prompt+output
concatenation that VllmRunner.generate() returns.
- Fix test_api_router.py: reset module-level _steering_lock between
tests to prevent cross-test lock state leakage.
- Fix test_worker_steering.py: delete no-op assertion that parsed as
`assert True` due to operator precedence.
commit 36a071e006e09890c479559880858c6dff174b97
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 23 22:12:03 2026 -0700
fix steering: unify API, remove redundant runner methods, fix test
- Remove set_steering_vectors and clear_steering_vectors from
GPUModelRunner — these had opposite semantics from the canonical
WorkerBase versions (zeroed non-specified layers vs leaving them
untouched) and were unreachable from the HTTP API path.
- Move import math/torch to module level in worker_base.py.
- Rewrite test to use string-based collective_rpc dispatching to
WorkerBase methods instead of calling GPUModelRunner directly via
lambda, making it compatible with the V2 model runner and
exercising the same code path as the HTTP API.
commit 77becfe0c9726bb5e651ee48c917bf738544bf36
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 23 22:11:52 2026 -0700
fix steering: extract decode-token utility, clean up API router
- Move _get_num_decode_tokens from Gemma3DecoderLayer to a shared
get_num_decode_tokens() utility in forward_context.py so future
steerable models can reuse it without reimplementing fragile
attn_metadata navigation.
- Collapse duplicated except (ValueError, RuntimeError) / except
Exception blocks in the steering API router into a single handler.
- Add replace field to SetSteeringRequest for atomic clear+set within
one lock acquisition, eliminating the observable empty-steering
window when swapping profiles via separate /clear + /set calls.
- Acquire _steering_lock in GET /v1/steering to prevent reading
partially-applied state during concurrent mutations.
commit 56cdb4bb97a493034caf0b9f4fe24bb84cc45d62
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 23 15:54:12 2026 -0700
steering api
commit 339a3a1bdec305d4a74a8be7be3862d8c54d569a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 21 19:57:18 2026 -0700
decode only steering
RhizoNymph
added a commit
that referenced
this pull request
Apr 16, 2026
commit 0e462d5e5c2d3c57c91980ca9cba9b8ca811154e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 15 02:16:45 2026 -0700
fix: use getattr for steering args in api_server init_app_state
commit 85594fb7dba189f9a3b8f067f5ca6df54f06f83d
Merge: d912217f8 3c2f98ec1
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 15 00:26:53 2026 -0700
Merge pull request #88 from RhizoNymph/feat/steering-more-plain-models
test: steering fixtures + layer_idx discovery fix for new models
commit 3c2f98ec1eb8d793f38a4e3e49f7f5ae7638b508
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 15 00:12:04 2026 -0700
test: add steering fixtures for new decoder families
commit d912217f8fe1c31fc2aeac617b13cbed7bb333cf
Merge: 28306b439 2ff317c9b
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 23:05:44 2026 -0700
Merge pull request #84 from RhizoNymph/feat/steering-more-plain-models
feat: wire steering hooks into seven more decoder models
commit 2ff317c9b41265d096677d2feb723bf29c8321ae
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 21:09:13 2026 -0700
feat: wire steering hooks into seven more decoder models
commit 28306b4398df9d6210acc340f1c2aa7bc15f6bf1
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 19:46:40 2026 -0700
Squashed commit of the following:
commit acf39ab1a8b7790be65ca3ceca623cd1b148569c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 19:35:30 2026 -0700
feat: gate global steering changes behind steering api key if set
commit 58ab6e83a55f6092470ff0b97535d03b17e9e999
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 18:50:40 2026 -0700
remove agent overview doc
commit 3d77e12e1dc2685bbe7945878b0c60c006acbb68
Merge: ada55957f 3b08fb305
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 17:09:56 2026 -0700
Merge pull request #83 from RhizoNymph/refactor/steering-mixin
refactor(steering): extract SteeringModelRunnerMixin to match LoRA pattern
commit ada55957f721d48848a5391efb2f4b0dd908e72d
Merge: 9a00f0f08 0b325140b
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 16:58:05 2026 -0700
Merge pull request #81 from RhizoNymph/docs/steering-update
docs(steering): refresh supported-scope and runtime-boundaries
commit 9a00f0f083e25058f7c66108457a78e21c590ec5
Merge: 5ffa1cd69 0eba71d29
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 16:57:15 2026 -0700
Merge pull request #82 from RhizoNymph/refactor/steering-modules-router-merge
refactor(steering): consolidate modules router under serve/steering/
commit 5ffa1cd6984fb48d04633f950e7808d31234d221
Merge: 52f85006f 548684983
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 16:49:55 2026 -0700
Merge pull request #80 from RhizoNymph/chore/rename-steering-phase-tests
chore(steering): rename phase1..phase5 test prefixes to semantic names
commit 52f85006fd34715af8154237e703a85f50ef8d73
Merge: 95ffa52f3 49649a584
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 14 16:48:29 2026 -0700
Merge pull request #79 from RhizoNymph/chore/steering-sampling-params-imports
chore(steering): hoist sampling_params inline imports
commit 3b08fb3059bb7042b434ec6f9ae40f136472f30f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 15:56:35 2026 -0700
refactor(steering): extract SteeringModelRunnerMixin to match LoRA pattern
Co-authored-by: Claude
commit 0eba71d2901616c54b3cdf1108a87e687a1fd535
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 15:00:50 2026 -0700
refactor(steering): fold serve/steering_modules/ into serve/steering/
Co-authored-by: Claude
commit 548684983d31bc2d192068326d8c83b8715cda63
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 14:06:57 2026 -0700
chore(steering): rename phase1..phase5 test prefixes to semantic names
Co-authored-by: Claude
commit 0b325140bbe60f00bbeee26e964ecec9ef654a23
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 14:04:14 2026 -0700
docs(steering): refresh supported-scope and runtime-boundaries sections
commit 49649a58483c49f05f7b4b9dbba53c787105e93b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 14 14:03:25 2026 -0700
chore(steering): hoist sampling_params inline imports to module level
commit 95ffa52f355f74adfc9dfb207f7460cf05a085e4
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 13 13:41:55 2026 -0700
perf(steering): batch per-layer table writes via index_copy_
commit 27a9fa4bc0226e98c66c6103c955dd62e63d191f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 13 13:38:14 2026 -0700
perf(steering): skip populate when state is unchanged via dirty flag
commit 6d727d4f64eedf2ed39639892876c9bbc11537a6
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 13 02:07:31 2026 -0700
perf(steering): make hash_steering_config 14x faster and cache it
commit 37a9446eea958eac8026963c441eb48f8abb02d9
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 12 16:15:54 2026 -0700
perf: short-circuit steering buffer updates when no state is active
commit b871782c147af79d7af78dc484b13d5853b47153
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 12 15:48:24 2026 -0700
fix: don't create steering manager at all when steering not enabled
commit d7db597e9b0aafec12e72c84889f1ddd3c1f3c49
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 9 01:04:34 2026 -0700
chore: remove implementation docs
commit 2e20d1b9bf897ad593aa075793317cd8431cf2f2
Merge: 897aa5e3e 346383689
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 8 21:16:40 2026 -0700
Merge pull request #77 from RhizoNymph/fix/remove-vestigial-vector-buffers
fix: remove vestigial steering_vector_{hook} GPU buffers
commit 897aa5e3ee85680be7ea0a221eda9ff33789f12f
Merge: 85024d4ca 012dcf9f3
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 8 21:14:56 2026 -0700
Merge pull request #75 from RhizoNymph/docs/remove-vector-buffer-refs
docs: remove steering_vector_{hook} buffer references
commit 85024d4ca4776645e8d0e2c2c88abfd2073c1782
Merge: 9bcf5ce42 224c95539
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 8 21:14:14 2026 -0700
Merge pull request #76 from RhizoNymph/fix/steering-transition-priority
fix: prioritize prefill→decode transitions over new-request deferrals
commit 34638368923a706bab519b7b0b0cd13d6276ca52
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 21:11:33 2026 -0700
fix: remove vestigial steering_vector_{hook} GPU buffers from Phase 1
Closes #74
Co-authored-by: Claude
commit 224c955390fff1f2098266018e6e59a95dafcd0c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 21:09:55 2026 -0700
fix: prioritize prefill→decode transitions over new-request deferrals
Co-authored-by: Claude
commit 012dcf9f3be502fda884fba20c9ce3d9feab7d7f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 21:06:54 2026 -0700
docs: remove steering_vector_{hook} buffer references
Part of #74 — the vector buffers are being removed from the code.
Co-authored-by: Claude
commit 9bcf5ce42b3e98e1b0beefe01aa2c2ffebba405a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 19:28:48 2026 -0700
phase 5 of new models
commit a8cc4349303695d48dce7a09fdeebf722a180b3c
Merge: 183425b41 ff3bf0039
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 8 17:24:56 2026 -0700
Merge pull request #71 from RhizoNymph/feat/named-steering-vectors
feat: named steering vector modules
commit ff3bf00392efa6f575ce8a8b62510561d0ea2558
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 16:37:19 2026 -0700
reject malformed json bodies instead of silently skipping them
commit 173c88b4a440e63caa2d8ecbdcefc3f3bce8085c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 16:10:29 2026 -0700
validate named steering entries
commit 183425b41a16442c123ef0a0171ea3aed0dfd098
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 15:56:13 2026 -0700
feat: more steering models
commit eba7fd616e02e8f2d79d1a56732fd1cc72c37266
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 8 15:44:50 2026 -0700
fix: gate named vector registry on actually enabling steering
commit b642ba6a42a12ceb4ca45046396f49714528da0b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 23:41:37 2026 -0700
feat: phase 1 of adding new models
commit 3a43d1cc1299b5a621d9b476a39112ecd1daf207
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 23:40:59 2026 -0700
add register steering buffers helper
commit 15b49e31cbf7e7e072dfbde575c8a6bf6b75db16
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 23:36:15 2026 -0700
feat: wire up named steering module registry, CLI args, and docs
Co-authored-by: Claude
commit e9bf1d7455a0a6f06231e04e75c6af6c51e60712
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 23:31:05 2026 -0700
feat: add steering modules CLI args and HTTP API endpoints
Co-authored-by: Claude
commit cb3f0d418d2553afcc320a32d9e394dd2fc076fb
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 21:18:38 2026 -0700
feat: add steering_name field and serving handler resolution
commit 48935407f5cdd564e1ab5fc24c16392b2d72c38f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 21:19:15 2026 -0700
feat: add steering module registry and merge_steering_specs helper
commit 1e50302281f4f2c55481121617cb3f308bd8f055
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 16:00:19 2026 -0700
Add steering docs
commit e27a7cced6a0a8da91c66d6045d4bbe4b24baa6e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Apr 7 15:43:30 2026 -0700
fix: use httpx for async test and don't fail steering manager test
without available cuda
commit 9a1c7e0badc9dd96a875a43628f07e732ab3560c
Merge: b1b5b93b1 5575ae564
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Apr 7 09:06:42 2026 -0700
Merge pull request #70 from RhizoNymph/feat/prefill-steering
feat: prefill/decode phase-aware steering vectors
commit 5575ae56469a4ef6b4fd0e57e226b30e97e5efce
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 17:05:35 2026 -0700
fix: APC streaming semantics
commit b10dd23f57301f01796794c6869fb9968086777c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 16:20:06 2026 -0700
fix: keep steering prefix-cache keys consistent during deferred
registration
commit 2aa6ac4a0adc16896cc1f562019989f8fef441f0
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 16:08:05 2026 -0700
chore: update docs
commit f9c5245f18e876f8e22e25b1a27258ef7bfefb3f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 16:07:51 2026 -0700
fix: remove post_mlp_post_ln hook point and rename post_mlp_pre_ln to
post_mlp
commit b72b74c0e501b6341fd823d8318d025d5393421f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Apr 6 16:05:53 2026 -0700
fix: update prompt length before recomputing block hashes to fix
prefix-cache key misses
commit 03d11ebf5fceac2d9c2676e21d2907e421578cce
Merge: 650aaee77 cc205f6ff
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 18:51:00 2026 -0700
Merge pull request #69 from RhizoNymph/fix/steering-streaming-hash-mismatch
fix: remove redundant _reset call from streaming steering path
commit cc205f6ff9dab721720aaf998e306f5f08a72116
Merge: 1de87974e 650aaee77
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 18:50:50 2026 -0700
Merge branch 'feat/prefill-steering' into fix/steering-streaming-hash-mismatch
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 650aaee772d0d81ed4482b55b580533c1adf9159
Merge: e89f20aeb 4b3bb798e
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 18:50:12 2026 -0700
Merge pull request #68 from RhizoNymph/fix/steering-gpu-resident-vectors
fix: store per-request steering vectors on GPU to avoid CPU->GPU copy
commit e89f20aeb20d69ea68bf3951284542e0c46d7405
Merge: 806e39cad 96d73fba5
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 18:50:01 2026 -0700
Merge pull request #67 from RhizoNymph/fix/steering-reject-extra-keys
fix: reject extra keys in scaled steering entries
commit 1de87974e68cf455bea2ce9ab941b24040653d44
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 5 18:37:37 2026 -0700
fix: remove redundant _reset call from streaming steering path
Co-authored-by: Claude
commit 4b3bb798e8604de998cfef30a74955b8090c9f8e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 5 18:30:32 2026 -0700
fix: store per-request steering vectors on GPU to avoid CPU->GPU copy each step
commit 96d73fba5018d133ba239f2e23fd038f788b32a1
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 5 18:30:24 2026 -0700
fix: reject extra keys in scaled steering entries
commit 806e39cad4cb5ad9ff768d2d173d95312ab78bbc
Merge: 54b0ccb9b bed8e050f
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Apr 5 13:28:12 2026 -0700
Merge pull request #65 from RhizoNymph/test/steering-retry-and-replay-coverage
test: cover steering transition deferral and pending globals replay
commit bed8e050fdb1b6bbe6c43f86b775c220ec247841
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Apr 5 13:21:38 2026 -0700
test: cover steering transition deferral and pending globals replay
Closes two test-coverage gaps identified in the post-merge audit of the
steering feature:
1. TestTransitionUnderCapacity exercises the bookkeeping contract of
_handle_steering_transition when decode registration fails due to
capacity. Verifies the deferred entry is appended with phase="decode",
_req_steering_phase[req_id] is set to "decode" regardless, and
get_row_for_config falls back to row 2 until the next retry succeeds.
2. TestPendingGlobalsReplay verifies that phase-labeled entries captured
into _pending_steering_globals before SteeringManager lazy init land
in the matching global_{base,prefill,decode}_vectors dicts rather
than collapsing into a single tier, including end-to-end via
populate_steering_tables.
Also drops two unused-variable / nested-if lint issues in
TestPreemptionResumption that were blocking pre-commit on this file.
Co-authored-by: Claude
commit 54b0ccb9b1405c3feb388222388becc328fae091
Merge: 735fc4d91 416c9b388
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Apr 4 17:26:36 2026 -0700
Merge pull request #64 from RhizoNymph/fix/steering-preempt-resume-config-leak
fix: reset steering phase on preemption-resumption to prevent refcount leak
commit 416c9b38849fe2ed88b46a82345e32f7499d20b4
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Apr 4 16:33:53 2026 -0700
fix: reset steering phase on preemption-resumption to prevent refcount leak
When a request with per-request steering is preempted (e.g. via
reset_prefix_cache() which fires on every global steering update) and
later resumed, num_computed_tokens is reset to 0 but the model runner's
_req_steering_phase and the SteeringManager's registered configs are
NOT cleaned up. On resumption the request re-prefills using the global
prefill row (not its per-request row), and the prefill->decode
transition double-registers the decode config -> permanent row leak
on finish.
Add _reset_steering_for_resumption helper that releases the stale
decode config, drops any stale deferred entries, and re-registers the
prefill config. Call it from both the v1 cached-reqs resumption path
(guarded by resumed_from_preemption) and the v2 _update_streaming_request
path.
Co-authored-by: Claude
commit 735fc4d9183af796a556ca31937b9a3b7d162fef
Merge: 7d566afc7 cde991759
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Apr 4 14:42:58 2026 -0700
Merge pull request #62 from RhizoNymph/fix/steering-api-empty-hook
fix: drop empty hook entries in steering _normalize_spec
commit 7d566afc7bb163838da34957ddd8786c5766190b
Merge: 6c56b5aa7 8ed116494
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Apr 4 14:32:36 2026 -0700
Merge pull request #61 from RhizoNymph/fix/steering-manager-phase-validation
fix: validate phase in populate_steering_tables
commit 6c56b5aa7dfaab2834aae67d0cc8d284af831630
Merge: b8f783c89 729f196f7
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Apr 4 14:31:59 2026 -0700
Merge pull request #60 from RhizoNymph/fix/steering-prefill-decode-cross-validation
fix: validate prefill/decode steering dimensions without base
commit cde991759948516e5afe1be2b4dd9664a19619a8
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Apr 4 13:37:59 2026 -0700
fix: drop empty hook entries in steering _normalize_spec
Co-authored-by: Claude
commit 8ed116494e6c30dd9696e64ef8560af8dd8696b2
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Apr 4 13:36:57 2026 -0700
fix: validate phase in populate_steering_tables
Co-authored-by: Claude
commit 729f196f757e8fe88e979ae07cc211e0fdcef8dd
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Apr 4 13:35:05 2026 -0700
fix: validate overlapping prefill/decode steering dimensions without base
Co-authored-by: Claude <noreply@anthropic.com>
commit b8f783c898a9ab2b8c66807be72da8b071b8cd23
Merge: d4e78d422 546d8e14c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Apr 3 23:23:38 2026 -0700
Merge pull request #58 from RhizoNymph/fix/steering-hash-invalidation
fix: invalidate cached steering hashes on streaming sampling_params update
commit d4e78d4226c5354f61d927d23c2440221ce0adc9
Merge: 2fddb12f4 9377c7141
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Apr 3 22:38:04 2026 -0700
Merge pull request #57 from RhizoNymph/fix/steering-reject-when-disabled
fix: reject per-request steering when enable_steering is off
commit 546d8e14c92a543168a9dda37e6858a28a416a70
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 21:34:06 2026 -0700
fix: invalidate cached steering hashes on streaming sampling_params update
Co-authored-by: Claude
commit 9377c7141c95b51cf8eb92bb3d0d2633674e6eec
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 21:32:23 2026 -0700
fix: reject per-request steering when enable_steering is off
Co-authored-by: Claude
commit 2fddb12f4e86f813bdeba582973524d48fa77a76
Merge: a681974aa 5959d8fe3
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Apr 3 16:52:06 2026 -0700
Merge pull request #56 from RhizoNymph/fix/steering-streaming-hashes
fix: refresh steering hashes when updating streaming requests
commit a681974aaae738cafe54b883166b5fc063c59724
Merge: 4dc95f529 3526be8be
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Apr 3 16:51:29 2026 -0700
Merge pull request #55 from RhizoNymph/fix/steering-forbid-extra
fix: reject unknown fields in SetSteeringRequest
commit 5959d8fe325beb50a7294e123f4744f74b08ed0d
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 15:40:08 2026 -0700
fix: refresh steering hashes when updating streaming requests
Co-authored-by: Claude
commit 3526be8be85624eb4506000ac540e9c958ba9c26
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 13:19:23 2026 -0700
fix: reject unknown fields in SetSteeringRequest to catch legacy scales
Co-authored-by: Claude
commit 4dc95f52958069c4c431b7c0ebf1308d6b5ff1fe
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Apr 3 01:05:58 2026 -0700
fix: fail on stale cache after steering change and relax decode-only admission
Co-authored-by: Claude
commit e2ef338ed7bfe52bcabd343f2cdcd82f7faca9c7
Merge: da2fec961 6288714c5
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 23:25:29 2026 -0700
Merge pull request #54 from RhizoNymph/fix/steering-atomic-replace
fix: make steering replace=true atomic via single worker RPC
commit da2fec96126507d48e3d5fa3b588697f6238e8de
Merge: ee488edac bb24d9947
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 23:22:06 2026 -0700
Merge pull request #53 from RhizoNymph/fix/steering-openai-plumbing
fix: expose full steering API in OpenAI protocol models
commit 6288714c5d3ff3da62d0736a766b4974d501b1f3
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 23:07:33 2026 -0700
fix: make steering replace=true atomic via single worker RPC
Co-authored-by: Claude
commit bb24d99473b12a367a962f25e5c77de36a2b5d7d
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 23:05:25 2026 -0700
fix: expose full steering API surface in OpenAI protocol models
Co-authored-by: Claude
commit ee488edacddfb8da257320656769d978a0db92d2
Merge: 56f66ad7b 442e9eba8
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:34:01 2026 -0700
Merge pull request #51 from RhizoNymph/fix/steering-request-attrs
fix: remove unused steering vector attributes from Request
commit 56f66ad7b7e31fafad1b5b4e0c9e04bfc51debf9
Merge: 46e522e0f 9b2997d1c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:29:20 2026 -0700
Merge pull request #49 from RhizoNymph/fix/steering-scheduler-capacity
fix: add decode-only capacity check and transition prediction for WAITING requests
commit 46e522e0f5fbd007b8c1b1dff6029f37481f341a
Merge: ece7238fb 7623fbe3f
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:26:51 2026 -0700
Merge pull request #48 from RhizoNymph/fix/steering-model-runner-guards
fix: add error handling for steering registration and fix backfill refcounting
commit ece7238fb087b3d498868085fa8841c47ef18292
Merge: 9ae9bcd37 6e6deadb1
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:25:48 2026 -0700
Merge pull request #47 from RhizoNymph/fix/steering-device-mismatch
fix: resolve GPU+CPU device mismatch in steering table population
commit 9ae9bcd37ac590c8f830af39bcc316fe55fe135a
Merge: 5d19f7f10 e6a9ffaf1
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:24:01 2026 -0700
Merge pull request #50 from RhizoNymph/fix/steering-replace-clear
fix: honor replace=True in set_steering_vectors when no new vectors provided
commit 5d19f7f10f2d53aad4759bfbfd24e5a0639a6a4d
Merge: 000d00f48 dd79f0844
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Thu Apr 2 18:22:54 2026 -0700
Merge pull request #52 from RhizoNymph/fix/steering-cache-key-phase
fix: disambiguate boundary block cache keys for prefill vs decode steering
commit 442e9eba84ba37a7e0439e43dd89509915326a17
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:31:03 2026 -0700
fix: remove unused steering vector attributes from Request to prevent memory waste
Co-authored-by: Claude
commit e6a9ffaf1031efd4b99fb8c7eac3f852f604a87b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:29:30 2026 -0700
fix: honor replace=True in set_steering_vectors when no new vectors provided
Co-authored-by: Claude
commit dd79f084480a4092c5493303bb7ebf2940ab55e5
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:29:00 2026 -0700
fix: disambiguate boundary block cache keys for prefill vs decode steering
Co-authored-by: Claude
commit 9b2997d1cb33475567c791c3a03624972e0806c5
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:27:06 2026 -0700
fix: add decode-only capacity check and transition prediction for WAITING requests
Co-authored-by: Claude
commit 7623fbe3fb789b1776e1f0c4d26e70ed143de71a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:27:05 2026 -0700
fix: add error handling for steering registration and fix backfill refcounting
Co-authored-by: Claude
commit 6e6deadb1e9fd56a13fd72ed8adb22acb1979317
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Apr 2 00:27:04 2026 -0700
fix: resolve GPU+CPU device mismatch in steering table population
Co-authored-by: Claude
commit 000d00f48bbba901b7d4fa682c75eb6439be9db4
Merge: dfed3d2bf f9130cf07
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 1 18:34:56 2026 -0700
Merge pull request #45 from RhizoNymph/fix/steering-lazy-init-capacity
fix: gracefully handle capacity exhaustion during steering lazy init
commit f9130cf07d50a9ff0c02d89e4e5f22fb4c373d53
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 1 16:24:56 2026 -0700
fix: defer steering registrations on capacity exhaustion instead of dropping
Replace the silent fallback-to-global approach with a proper deferral
mechanism. Failed registrations during lazy init and prefill-to-decode
transitions are now appended to a unified
`_pending_steering_registrations` list that is retried with priority
before new admissions on every step.
The retry loop also filters stale entries:
- Entries for finished requests are dropped (prevents row leaks).
- Entries whose request changed phase are dropped (e.g. prefill
deferred but request already transitioned to decode).
This subsumes the decode-only `_pending_decode_registrations` list
and the row-leak fix from PR #46.
commit 251cabe638c7b33d2dcbf27e1da9754dd2e63369
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 1 01:16:28 2026 -0700
fix: gracefully handle capacity exhaustion during steering lazy init
commit dfed3d2bfd1557d7c8c3925dd5de8224c1e9e91d
Merge: 51672b8f1 41135ae57
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 1 12:53:25 2026 -0700
Merge pull request #44 from RhizoNymph/fix/steering-swap-states-hash
fix: swap steering hashes in InputBatch.swap_states()
commit 51672b8f1ef164970470580e380534f85ec3b35a
Merge: 47365775f f2638d4b7
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Wed Apr 1 12:36:32 2026 -0700
Merge pull request #43 from RhizoNymph/fix/steering-dimension-cross-validation
fix: validate matching dimensions between base and phase steering vectors
commit 41135ae5709c78d4639a0b0148967b491a38894e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 1 01:16:27 2026 -0700
fix: swap steering hashes in InputBatch.swap_states()
Co-authored-by: Claude
commit f2638d4b7f2f6a9136c11362f08b89b9873d42f3
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Apr 1 01:14:35 2026 -0700
fix: validate matching dimensions between base and phase steering vectors
commit 47365775f6c7310d6381fb5495777950fd729997
Merge: 981ad5ac8 ebeff444f
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 23:59:48 2026 -0700
Merge pull request #42 from RhizoNymph/docs/steering-bugfix-round2
docs: update steering docs for deferred cleanup and pre-init status
commit 981ad5ac85d2c93c3ae1f356f148ea097350481d
Merge: 7431847f1 f06f5db30
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 23:59:35 2026 -0700
Merge pull request #41 from RhizoNymph/fix/steering-status-pending-globals
fix: report pending phase-specific globals in steering status
commit 7431847f1cd2647ba96829395b2037f2317f520f
Merge: f04aaf36d e1a6a2ef5
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 23:59:12 2026 -0700
Merge pull request #40 from RhizoNymph/fix/steering-deferred-decode-leak
fix: track req_id in deferred decode registrations to prevent leak
commit f04aaf36d143c040b2b652403de3f0449a5d3d62
Merge: 7e41715be e9d8f110c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 23:58:37 2026 -0700
Merge pull request #39 from RhizoNymph/fix/steering-boundary-block-hash
fix: include decode hash in boundary block cache keys
commit ebeff444f4bd59751824305843481a4e92178ece
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 23:34:00 2026 -0700
docs: update steering docs for deferred cleanup and pre-init status
Co-authored-by: Claude
commit e9d8f110c39446cd8c4e9bcbaddc91c82236962f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 23:29:38 2026 -0700
fix: include decode hash in boundary block cache keys
Co-authored-by: Claude
commit f06f5db3000d6fb29d21219538cc4d9c43030e04
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 23:27:52 2026 -0700
fix: report pending phase-specific globals in steering status
commit e1a6a2ef5da949d4636d98779cb146b803cb376b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 23:26:48 2026 -0700
fix: track req_id in deferred decode registrations to prevent leak
commit 7e41715be28060d2ebb7dd97bff0aef72cf04b27
Merge: 5edfb74f2 0c7e77579
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 19:44:19 2026 -0700
Merge pull request #38 from RhizoNymph/docs/steering-bugfix-updates
docs: update steering docs for bugfix PRs #35-#37
commit 5edfb74f2d3657093e316c5fe673cf58e6a1ce99
Merge: a7bdd3fe1 2f0a0c960
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 19:44:01 2026 -0700
Merge pull request #36 from RhizoNymph/fix/steering-scheduler-transition-capacity
fix: transition-aware steering capacity and decode-start admission
commit a7bdd3fe1ad5f2a3c4e25a67d4da9b1617a34992
Merge: 2830adbf0 b2d8d475c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 19:08:27 2026 -0700
Merge pull request #35 from RhizoNymph/fix/steering-worker-status-validation
fix: remove double validation and add phase-specific norms to steering status
commit 2830adbf014067736c04afdc1ddfdb11c898009c
Merge: c01e6dce4 5cb702a2a
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 19:01:21 2026 -0700
Merge pull request #37 from RhizoNymph/fix/steering-model-runner-transition
fix: graceful steering transition + phase-tracked release
commit 0c7e77579e3cab4e5cdc6987e43a376481bb647d
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 18:58:04 2026 -0700
docs: update steering docs for transition-aware capacity, deferred registration, and phase-specific norms
commit 5cb702a2aa253defaa567fbda72e8c2e26650267
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 18:30:11 2026 -0700
fix: graceful steering transition with deferred registration and phase tracking
Co-authored-by: Claude
commit b2d8d475c0010910fd2b680d9460dc9caa715708
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 17:59:07 2026 -0700
fix: remove double validation and add phase-specific norms to steering status
Co-authored-by: Claude
commit 2f0a0c96055e7330687ad1e101059eb3fa311040
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 17:58:31 2026 -0700
fix: transition-aware steering capacity counting and decode-start admission check
Co-authored-by: Claude
commit c01e6dce4032a00a962d1b257b3f907e327f5b54
Merge: 5ab47798d 84eeb8b21
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 01:05:21 2026 -0700
Merge pull request #34 from RhizoNymph/fix/steering-decode-cache-block-hash
fix: include decode steering hash in generated cache block keys
commit 5ab47798d2d6a6778e61610ddde0fcaad2f80240
Merge: c3bda6dc5 2b8a2a6ec
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 01:04:14 2026 -0700
Merge pull request #32 from RhizoNymph/fix/steering-scheduler-single-phase-capacity
fix: scheduler steering capacity counts only active phase
commit c3bda6dc5f8da7e9dcae49faba5c3b69aed6a67a
Merge: dd9e5b185 35c55f9eb
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Tue Mar 31 01:02:59 2026 -0700
Merge pull request #33 from RhizoNymph/fix/steering-lazy-init-decode-registration
fix: register decode steering configs during lazy manager init
commit 84eeb8b21d7435d91655d6a19aa7b33f7de16800
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 31 00:01:30 2026 -0700
fix: include decode steering hash in generated cache block keys
commit 35c55f9eb69bca17dd3f0c80830b419921bef073
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 30 23:42:06 2026 -0700
fix: register decode steering configs during lazy manager init
commit 2b8a2a6eccde4b10f8565bbc62e13c8024a85327
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 30 23:41:46 2026 -0700
fix: scheduler steering capacity counts only active phase
commit dd9e5b1852d053f31535ec75c72d973095b07424
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 30 23:09:17 2026 -0700
docs: update steering docs for PR #29, #30, #31 post-merge fixes
commit 63089a37e19e877625de97ad69e62dbf8f0f4b42
Merge: dbe8d1880 dcdf60110
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Mon Mar 30 23:00:16 2026 -0700
Merge pull request #31 from RhizoNymph/fix/steering-buffer-overwrite
fix: stop writing phase-specific globals to shared steering buffers
commit dbe8d18809e04fcfd7f959d90a4f7d2eed95adac
Merge: 5100a10d4 bd69fae0b
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Mon Mar 30 22:59:26 2026 -0700
Merge pull request #30 from RhizoNymph/fix/steering-decode-start-registration
fix: register decode steering config when request starts past prefill
commit 5100a10d4a0d95229c3b6587484b1a122bc08ae2
Merge: 8d1c1fe47 3de429a5a
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Mon Mar 30 22:57:48 2026 -0700
Merge pull request #29 from RhizoNymph/fix/steering-malformed-entry-validation
fix: reject malformed scaled steering entries with 4xx
commit dcdf60110fd6e89e8dd9a3128a8b3a71049de610
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 15:29:56 2026 -0700
fix: stop writing phase-specific globals to shared steering buffers
commit bd69fae0b51f0c2e75f323955d62265b4623db9e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 15:13:54 2026 -0700
fix: register decode steering config when request starts past prefill
commit 3de429a5a84be3952759555b49ca01f22b05e37f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 14:40:26 2026 -0700
fix: reject malformed scaled steering entries with 4xx instead of 500
commit 8d1c1fe47dac02e3b3f07ef324967a969d66790d
Merge: 9d8d7a4e9 4f7f350dd
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 14:13:33 2026 -0700
Merge pull request #27 from RhizoNymph/fix/steering-clear-pending-globals
fix: clear pending steering globals on clear_steering_vectors
commit 9d8d7a4e99566fb6c445ef46de477110d0bf5f0c
Merge: c57ff54da d6efa42ce
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 14:11:58 2026 -0700
Merge pull request #28 from RhizoNymph/fix/steering-scheduler-phase-capacity
fix: count steering capacity by (hash, phase) pairs to match worker row allocation
commit 4f7f350dd603817a7a0bbda59431220197550988
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 12:08:15 2026 -0700
fix: clear pending steering globals on clear_steering_vectors to prevent stale replay
commit d6efa42ce8806af8846600a0ec6d39dfc5609952
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 12:07:57 2026 -0700
fix: count steering capacity by (hash, phase) pairs to match worker row allocation
commit c57ff54daf3f93216b8d609b9719ed38e918339e
Merge: 849fa0577 2f3b14e09
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 11:27:02 2026 -0700
Merge pull request #25 from RhizoNymph/fix/steering-pending-globals-queue
fix: preserve phase-specific global vectors set before steering manager init
commit 2f3b14e094f4f85932b7650819b3cb0e35716583
Merge: 55a9da811 849fa0577
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 11:26:51 2026 -0700
Merge branch 'feat/prefill-steering' into fix/steering-pending-globals-queue
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 849fa0577a9ca55e201830e59d351140414c21c6
Merge: 018a7adcc 229ce88f3
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 11:24:05 2026 -0700
Merge pull request #24 from RhizoNymph/fix/steering-scheduler-phase-detection
fix: use num_prompt_tokens for steering phase detection in scheduler
commit 018a7adcc417fa3cd8647954752fbade1cdbb31c
Merge: 007de86fc 91b89b480
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sun Mar 29 11:16:11 2026 -0700
Merge pull request #26 from RhizoNymph/fix/steering-phase-keyed-rows
fix: key steering table rows by (config_hash, phase) to prevent cross-phase aliasing
commit 91b89b4805a7a5a3464cd2d605211f4d8e2c1189
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 11:11:10 2026 -0700
fix: key steering table rows by (config_hash, phase) to prevent cross-phase aliasing
commit 55a9da811b0ae47e46816eb8588bf234d8130835
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 00:38:21 2026 -0700
fix: queue phase-specific global vectors for replay on steering manager lazy init
commit 229ce88f3392682b44cc69f7b5ca12e827acd14d
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 00:35:41 2026 -0700
fix: use num_prompt_tokens for steering phase detection in scheduler
commit 007de86fc362ed6d86dc4bbebbf5e5a41581851e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sun Mar 29 00:13:51 2026 -0700
docs: update steering docs for prefill steering (Phase 4)
- ROADMAP.md: Mark Phase 4 as DONE with full implementation summary
- STEERING.md: Update to three-tier model, phase-aware table layout,
remove stale "decode-only" language, update key files table
- MVP_IMPLEMENTATION.md: Add Phase 4 lessons learned (prefix cache,
lazy init race condition, phase detection, additive composition,
co-located scales, scheduler dual-hash admission, test subtleties)
- API.md: Update to three-tier format, co-located scales, remove
old scales field, update examples
commit 8f055b92c5f82d1283a2b186619bcb2b128cf22a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 23:59:30 2026 -0700
fix: register steering configs on first-step manager init + fix additive test
The SteeringManager is lazily created in _update_steering_buffers, but
config registration happens in _update_states which runs before. On the
first request, registration was silently skipped because the manager
didn't exist yet. Fix: after creating the manager, scan the batch and
register any pending configs.
Also fix test_additive_composition which incorrectly assumed base(250) +
decode(250) equals base(500) — steering_vectors affects both phases, so
prefill effective vectors differed between the two approaches.
commit f15d55b8d0748a38b4e5786c09afa6430d9d28e7
Merge: 9c1dca9f5 2917406a5
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 23:21:56 2026 -0700
Merge pull request #23 from RhizoNymph/feat/prefill-steering-tests
test: add prefill steering integration tests
commit 2917406a5d1e53de9fbf4db4fbbc1adde087777a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 22:59:26 2026 -0700
test: add prefill steering integration tests and fix existing test API calls
Co-authored-by: Claude
commit 9c1dca9f547e52defa4782326a2b6777e7940f8f
Merge: 2be915751 45100595f
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:43:26 2026 -0700
Merge pull request #21 from RhizoNymph/feat/prefill-steering-manager
feat: phase-aware SteeringManager and GPU model runner
commit 45100595fc03a46d5407edab6240971e39b3d2ed
Merge: 1beb47cc2 2be915751
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:35:19 2026 -0700
Merge branch 'feat/prefill-steering' into feat/prefill-steering-manager
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 2be91575101afb56f9952093c4c40b5f5ce00174
Merge: a286bd9d6 ec9e660b3
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:34:15 2026 -0700
Merge pull request #22 from RhizoNymph/feat/prefill-steering-scheduler
feat: prefill steering scheduler, API, and worker (WS4)
commit ec9e660b307566bba43e5230cd2321e437010510
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 21:33:10 2026 -0700
feat: wire prefix cache invalidation for prefill-affecting steering changes
Replace the TODO with actual calls to engine.reset_prefix_cache() when:
- Global base or prefill vectors are set (affects prefill KV cache)
- replace=True is used (clears all tiers including prefill)
- All steering is cleared via /v1/steering/clear
commit a286bd9d660ad473232f9b60a0f04706494375a5
Merge: 67b6d9d65 89e056fb6
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:25:44 2026 -0700
Merge pull request #19 from RhizoNymph/feat/prefill-steering-cache-keys
feat: prefill steering prefix cache key integration
commit 89e056fb6bf5884135ef2a26c7e935da5c988b85
Merge: 68cce443a 67b6d9d65
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:22:09 2026 -0700
Merge branch 'feat/prefill-steering' into feat/prefill-steering-cache-keys
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 67b6d9d653da3c84fae431aad0a14e06b77ce85e
Merge: b1b5b93b1 46665f131
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 21:18:16 2026 -0700
Merge pull request #20 from RhizoNymph/feat/prefill-steering-types
feat: three-tier steering types with prefill/decode separation
commit b7d55a044a2fde9ae18c587711d8ddac6b03b9a6
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 21:06:52 2026 -0700
feat: implement prefill steering scheduler, API, and worker changes
- Scheduler: dual-hash admission control tracks union of active hashes
(prefill hash for prefill-phase requests, decode hash for decode-phase).
New requests must have capacity for both their prefill and decode hashes.
- API router: three-tier global steering (vectors, prefill_vectors,
decode_vectors) with co-located scale normalization. Removed old
scales field. Uses kwargs-based collective_rpc for worker calls.
- Worker: three-tier set_steering_vectors with base/prefill/decode tiers.
Extracted validation, buffer application, and manager notification into
separate helper methods. Supports replace=True across all tiers.
- Tests: comprehensive dual-hash scheduler admission tests, three-tier
API router tests with co-located scale format, and three-tier worker
tests including replace and cross-tier validation.
- Docs: updated steering.md with dual-hash admission details and
three-tier HTTP API examples.
Co-authored-by: Claude
commit 1beb47cc2be30910c88e5b2aeba266a6e6c65036
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 21:01:12 2026 -0700
feat: phase-aware SteeringManager and GPU model runner integration
Co-authored-by: Claude
commit 46665f131579bb24b50b0786d2424e9cd28a45fe
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 20:41:31 2026 -0700
feat: add three-tier steering types with prefill/decode phase separation
Co-authored-by: Claude
commit 68cce443a84085f7f591e00e9967d1a62fed5f93
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 20:38:47 2026 -0700
feat: add prefill steering config hash to prefix cache block keys
Add _gen_steering_extra_hash_keys() to include per-request prefill
steering config hashes in block hash extra keys. This ensures that
blocks computed under different prefill steering produce different
cache entries, with zero impact when steering is not used.
Co-authored-by: Claude
commit b1b5b93b15ad8bc2bcea10a786b9270398c66b3e
Merge: 03868735b 21c6acbb4
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Sat Mar 28 18:41:07 2026 -0700
Merge pull request #18 from RhizoNymph/feat/steering-hook-points
feat: Phase 3 — multiple steering intervention points
commit 21c6acbb4b55f6cb5244fb67522514660ca2abfa
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 18:36:12 2026 -0700
docs: add Phase 3 lessons learned to implementation notes
Key findings:
- Splitting ops are not required for buffer correctness — custom op
opacity is sufficient, graph partitioning is a separate concern
- Inline tensor math triggers AOT autograd pickle failures due to
Enum WeakValueDictionary in traced graph metadata
- Always-allocate beats opt-in when memory is trivial vs model size
- Hook points at same residual state exist for SAE position matching
- Breaking backward compat cleanly beats merging logic
Co-authored-by: Claude
commit 15f34a6d006cd509fe10297e007a9a658697d5c6
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 18:33:51 2026 -0700
docs: update steering docs for simplified hook point design
- Remove references to --steering-hook-points CLI flag
- Remove references to steering_hook_vectors (legacy separate field)
- Document that all 4 hook points are always allocated unconditionally
- Document that apply_steering is NOT a splitting op (zero graph partitions)
- Update all API examples to use the hook-point-aware steering_vectors format
Co-authored-by: Claude
commit 6837fabf7a76a86eebdb0546a7c03b52ec1e2204
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 18:26:07 2026 -0700
chore: remove --steering-hook-points CLI flag and config field
All 4 hook point buffers are always allocated unconditionally.
The memory cost is trivial (~11 MB at max_configs=16 for 26 layers)
and there are no graph partitions, so there is no reason to control
which hook points are active.
Co-authored-by: Claude
commit 5369eead8d2575202add671ba975ccffbc9b1be3
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 17:49:26 2026 -0700
feat: remove graph partitions, always allocate all hook point buffers
- Remove apply_steering from splitting_ops — the custom op is still
opaque to the tracer (preventing constant-folding) but no longer
partitions the compiled graph
- Always allocate all 4 hook point buffers unconditionally (~11 MB for
26 layers at max_steering_configs=16) — no hasattr guards needed
- Remove active_hook_points parameter from Gemma3DecoderLayer
Co-authored-by: Claude
commit 45dbf1fef9bc0e0a3e3d5f2a2aef033703199cfc
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 16:14:47 2026 -0700
refactor: remove backward-compat steering_vectors, require hook points
steering_vectors is now always dict[str, dict[int, list[float]]]
(hook_point → layer → vector). Removes the old flat format, the
separate steering_hook_vectors field, and all merging logic.
Co-authored-by: Claude
commit 3d4022dd3f3f795bc2ebe48c5cb618cfece31e59
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 15:05:39 2026 -0700
docs: update steering docs for Phase 3 hook points
Co-authored-by: Claude
commit cf969cb1bdf8e1e38cdfebdb6ed16e6304a73c9c
Author: RhizoNymph <quantnymph@gmail.com>
Date: Sat Mar 28 14:57:47 2026 -0700
fix: restore hook-point-aware api_router and protocol after merge
The infrastructure agent's PR overwrote the API agent's changes to
api_router.py and protocol.py. This restores:
- hook_vectors field on SetSteeringRequest
- vectors default_factory=dict (no longer required since hook_vectors
can supply vectors)
- Hook-point-aware set_steering that wraps legacy vectors under
post_mlp_pre_ln before passing to workers
- Updated tests to match new API format
Co-authored-by: Claude
commit 11b0da7f4de47509e7ff794f90bc41616478d986
Merge: 5fd0e3929 8fbe4ddb4
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 23:29:20 2026 -0700
Merge pull request #15 from RhizoNymph/feat/steering-hook-points-manager-runner-worker
feat: multi-hook-point support for SteeringManager, ModelRunner, WorkerBase
commit 8fbe4ddb4381681807fc6d02949f5a1387ec6b74
Merge: 1c8717e30 5fd0e3929
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 23:29:05 2026 -0700
Merge branch 'feat/steering-hook-points' into feat/steering-hook-points-manager-runner-worker
Signed-off-by: Nymph <82485126+RhizoNymph@users.noreply.github.com>
commit 5fd0e39291aebc935402364a942d8cacf1b87cd8
Merge: c80df1880 ee599fed1
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 23:27:46 2026 -0700
Merge pull request #16 from RhizoNymph/feat/steering-hook-points-api
feat: add steering_hook_vectors API support
commit c80df1880ea3ff2960474885a37b11480c8baa1c
Merge: 54e3357d2 119f87a2e
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 23:25:17 2026 -0700
Merge pull request #17 from RhizoNymph/feat/steering-gemma3-hook-points
feat: update Gemma3 model for multiple steering hook points
commit ee599fed1209f693691f4edf1392dfa745f4f63a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:44:19 2026 -0700
docs: update steering feature doc for hook-point API support
commit ffb372c34ae61ba81c1820951453f41a2b6455a0
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:43:20 2026 -0700
feat: add steering_hook_vectors support to API layer
Add steering_hook_vectors field across the API layer to support
multiple hook points for activation steering vectors.
Changes:
- SamplingParams: new steering_hook_vectors field with validation
that checks hook point names against VALID_HOOK_POINT_NAMES and
reuses _validate_layer_vector_dict for inner dict validation
- Request: new steering_hook_vectors field populated from
sampling_params, steering_config_hash updated to build canonical
representation merging both steering_vectors and
steering_hook_vectors
- ChatCompletionRequest/CompletionRequest: new steering_hook_vectors
field passed through to SamplingParams.from_optional
- SetSteeringRequest: new hook_vectors field, vectors field now
defaults to empty dict for backward compatibility
- api_router set_steering: builds hook-point-aware dict merging
legacy vectors (as post_mlp_pre_ln) with explicit hook_vectors,
validates hook point names, passes new format to collective_rpc
commit 1c8717e30ea7ebe65d726f384c05921b71bb7f98
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:37:33 2026 -0700
feat: update SteeringManager, ModelRunner, WorkerBase for multi-hook-point steering
Co-authored-by: Claude
commit 119f87a2e2f1894dcac27cb042ee9fa59a02d16b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:29:40 2026 -0700
feat: update Gemma3 model to support multiple steering hook points
Co-authored-by: Claude
commit 54e3357d26d440f423fe2f2a818193572b849c73
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 21:21:16 2026 -0700
feat: add SteeringHookPoint enum and --steering-hook-points CLI arg
Co-authored-by: Claude
commit 03868735bd343e1943f02ba7e3ace08f9e718a1f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 20:11:52 2026 -0700
docs: add per-request steering lessons learned to implementation notes
commit 2d8278f1f792ab8416b8affd70f69de280ba248a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 19:21:58 2026 -0700
fix: steering hash overflow, add concurrent CUDA graph integration test
- Mask steering_config_hash to fit np.int64 (was overflowing with
full 64-bit unsigned SHA-256 hash)
- Add test_per_request_steering_concurrent_with_cuda_graphs: sends
3 requests (unsteered, +500, -500) in same batch, verifies per-request
steering works with torch.compile + CUDA graphs active, then verifies
unsteered determinism on a second batch
commit 6bae6a4ad51baebc1d7575e320241cdd2b5bfa25
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 18:40:38 2026 -0700
docs: update steering docs for per-request implementation
commit 0265546ffca239b75ca32147f5dcb9bbb4063de5
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 18:34:12 2026 -0700
docs: add OVERVIEW.md and steering feature documentation
commit 4c57a2d31b2c647491dff7f1339cd81ad1fe9adf
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 18:12:01 2026 -0700
test: add scheduler admission, worker steering, and integration tests
commit 066839d90d3ce9c547c1fbfc9fdf9478ed4fc131
Merge: 72b7b0c68 142432b7c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 17:58:51 2026 -0700
Merge pull request #13 from RhizoNymph/feat/per-request-steering-runner
feat: integrate SteeringManager into model runner and worker_base
commit 72b7b0c6899515ce25da7846e46281f24378ea3a
Merge: b2e770b7d 2e914cfc7
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 17:58:21 2026 -0700
Merge pull request #11 from RhizoNymph/feat/per-request-steering-tests
test: SteeringManager and indexed-gather op unit tests
commit 142432b7cd911ee029ae13a3b4d29c2e713f12b7
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 17:51:50 2026 -0700
feat: integrate SteeringManager into model runner and worker_base
commit b2e770b7d38de65611808cb45ffd9c3d3c24e60f
Merge: 5d9c9f015 23198f1c0
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 17:43:22 2026 -0700
Merge pull request #12 from RhizoNymph/feat/per-request-steering-manager
feat: SteeringManager, InputBatch tracking, scheduler admission control
commit 23198f1c085115e3935edd8c4dfe6c8574c84fe7
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 17:24:10 2026 -0700
feat: add SteeringManager, InputBatch tracking, and scheduler admission control
commit 2e914cfc75f20e5b90b1801e7953aff75da585ad
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 17:02:33 2026 -0700
test: add SteeringManager and indexed-gather op unit tests
commit 5d9c9f015dd9d21c372594ac2ab237091f3e52ee
Merge: 9e535b083 2066a7e1c
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 16:47:55 2026 -0700
Merge pull request #9 from RhizoNymph/feat/per-request-steering-config
feat: add SteeringConfig, SamplingParams field, and API protocol support
commit 9e535b08395e3dd40e17f6bb754a24e93432af28
Merge: 602474f3c 10c95f5d4
Author: Nymph <82485126+RhizoNymph@users.noreply.github.com>
Date: Fri Mar 27 16:46:03 2026 -0700
Merge pull request #10 from RhizoNymph/feat/per-request-steering-op
feat: update steering op to indexed gather with table/index buffers
commit 2066a7e1c90fcca42816bb15e00180b9283698bb
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 16:04:51 2026 -0700
feat: add steering config, SamplingParams field, and API protocol support
commit 10c95f5d48ea45d213d2020717970467b7c93ba4
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 16:10:09 2026 -0700
feat: update steering op to indexed gather, add table/index buffers to Gemma3
commit 602474f3c947718c34b57029b45d7717c61dd2f9
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 14:08:39 2026 -0700
remove todo doc for mvp
commit 7619f8ae91273cf022892d1543a5bace29346784
Author: RhizoNymph <quantnymph@gmail.com>
Date: Fri Mar 27 14:08:28 2026 -0700
fix: change injection point to raw residual stream not post layernorm
commit 98408699f80226539d908b3bb690af8f6e853d9e
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Mar 26 22:47:14 2026 -0700
fix: decode-only steering with torch.compile and CUDA graphs
Replace ForwardContext-based num_decode_tokens read (constant-folded by
torch.compile) with persistent per-layer mask buffers updated in-place
by the model runner before each forward pass.
- Custom op takes 3 args (hidden, vec, mask) and is a splitting op
- Each decoder layer registers a steering_decode_mask buffer
- Model runner discovers steerable layers and updates masks from
scheduler output (decode tokens = 1 scheduled token, ordered first)
- Add num_decode_tokens to TritonAttentionMetadata and
FlashAttentionMetadata
- Increase test steering magnitude for dummy-weight models
- Add prefix cache reset after clearing steering in E2E test
Co-authored-by: Claude
commit c19a78587d29270c901c4fd66453a87eef049e8b
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Mar 26 13:36:55 2026 -0700
docs/update temporary steering docs
commit d51bd507c693f10372676030b3b97985c47ff561
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Mar 26 00:41:13 2026 -0700
fix: typed steering exceptions, remove GET lock, cache steerable layers
Co-authored-by: Claude
commit 6c93d113d849d6cf474a6a8c90b617c37cab609a
Author: RhizoNymph <quantnymph@gmail.com>
Date: Thu Mar 26 00:20:18 2026 -0700
fix: cast steering vector dtype to match hidden states
Co-authored-by: Claude
commit 531ad064402b875e1d8d11f3062e2c0165db61b2
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Mar 25 18:45:38 2026 -0700
fix: register steering as custom op for torch.compile safety
Co-authored-by: Claude
commit 5c4ec89f2fbb90e6aa4912b2d27c2684f01c609f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Wed Mar 25 18:43:51 2026 -0700
fix: return clear error for empty steering vectors request
Co-authored-by: Claude
commit e76810451eb76e8dc5d63ada1964f9dc1f98c423
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 24 18:44:25 2026 -0700
temporary steering docs
commit 1f2c5140c5b7fde10e7cffd4ba452207a115ba4f
Author: RhizoNymph <quantnymph@gmail.com>
Date: Tue Mar 24 18:15:36 2026 -0700
add steering unit tests, fix test bugs
- Add 65 unit tests across 4 files covering: SetSteeringRequest
protocol validation, get_num_decode_tokens forward context utility,
WorkerBase steering methods (set/clear/status with mock model,
validation errors, edge cases), and API router endpoints (mock
engine, all HTTP status codes, replace flag, attach/detach).
- Fix test_steering.py: use llm.llm.generate() to get RequestOutput
and compare only completion token_ids, not the prompt+output
concatenation that VllmRunner.generate() returns.
- Fix test_api_router.py: reset module-level _steering_lock between
tests to prevent cross-test lock state leakage.
- Fix test_worker_steering.py: delete no-op assertion that parsed as
`assert True` due to operator precedence.
commit 36a071e006e09890c479559880858c6dff174b97
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 23 22:12:03 2026 -0700
fix steering: unify API, remove redundant runner methods, fix test
- Remove set_steering_vectors and clear_steering_vectors from
GPUModelRunner — these had opposite semantics from the canonical
WorkerBase versions (zeroed non-specified layers vs leaving them
untouched) and were unreachable from the HTTP API path.
- Move import math/torch to module level in worker_base.py.
- Rewrite test to use string-based collective_rpc dispatching to
WorkerBase methods instead of calling GPUModelRunner directly via
lambda, making it compatible with the V2 model runner and
exercising the same code path as the HTTP API.
commit 77becfe0c9726bb5e651ee48c917bf738544bf36
Author: RhizoNymph <quantnymph@gmail.com>
Date: Mon Mar 23 22:11:52 2026 -0700
fix steering: extract decode-token utility, clean up API router
- Move _get_num_decode_tokens from Gemma3DecoderLayer to a shared
get_num_decode_tokens() utility in forward_context.py so future
steerable models can reuse it without reimplementing fragile
attn_metadata navigation.
- Collapse duplicated except (ValueError, RuntimeError) / except
…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Malformed dict-style steering layer entries (e.g.
{"vector": [...]}without"scale", or{"scale": 2.0}without"vector") caused aKeyErrorthat bubbled up as a 500 Internal Server Error. This PR fixes the issue in two places:vllm/config/steering_types.py:normalize_layer_entry()now validates that both"vector"and"scale"keys are present in dict entries before accessing them. Missing keys raise aValueErrorwith a clear message listing exactly which keys are absent.vllm/entrypoints/serve/steering/api_router.py: The three_normalize_spec()calls that happened before the maintryblock are now wrapped in their owntry/exceptcatchingKeyError,TypeError, andValueError, returning a 400 Bad Request with a descriptive error message.Four new unit tests in
tests/v1/test_steering_types.pycover the missing-key scenarios (missing"scale", missing"vector", missing both, and verifying the error message content).What changed and why
The root cause was that
normalize_layer_entryblindly didentry["vector"]andentry["scale"]on dict entries without checking the keys existed first. Additionally, the normalization step in the API router happened outside any error handling, so even if the function raised a proper error it would still surface as a 500.Both issues are fixed: the function now validates up front and raises
ValueError(notKeyError), and the router catches normalization errors and returns 400.