Repository navigation
b12170
Pre-release
moe-cache : LRFU/MQ-like hybrid cache, ClassLRU (#31233)
-
moe-cache : LRFU/MQ-inspired hybrid cache
-
make the clock advance by N tokens: use the real token count (ne[1] is always 1, ids have n_expert_used entries per token)
-
sel is a RESHAPE node, the orig input is in src[0], so sel->src[0]->ne[1] is the count
-
c for the class index, last_use for last use time
-
explain the roles of these
-
compute the key once
-
use the key directly instead of re-reading
-
show how the score is computed from the age
-
remove the branch and clarify exp2 and log2 relationship
-
reword the header comment and elaborate on unused slots and sorting
-
not a copy, not just skip
-
std::array maybe
-
Revert "std::array maybe"
This reverts commit 7649b1f.
-
half life it is
-
add key and iteration helpers
-
extract decayed_crf
-
bump classes to 8, document the formula, display the cache size in token worth count
-
assert the computed number of classes matches the half life
Website:
Attestations:
macOS/iOS:
- macOS Apple Silicon (arm64)
- macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
- macOS Intel (x64)
- iOS XCFramework
Linux:
- Ubuntu x64 (CPU)
- Ubuntu arm64 (CPU)
- Ubuntu s390x (CPU)
- Ubuntu 26.04 s390x (CPU)
- Ubuntu 26.04 s390x (zDNN)
- Ubuntu x64 (Vulkan)
- Ubuntu arm64 (Vulkan)
- Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries
- Ubuntu x64 (CUDA 13) - CUDA 13.5 libraries
- Ubuntu arm64 (CUDA 13) - CUDA 13.5 libraries
- Ubuntu x64 (ROCm 10.0)
- Ubuntu x64 (OpenVINO)
- Ubuntu x64 (SYCL FP32)
- Ubuntu x64 (SYCL FP16)
- Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide
Android:
Windows:
- Windows x64 (CPU)
- Windows arm64 (CPU)
- Windows arm64 (OpenCL Adreno)
- Windows x64 (CUDA 12) - CUDA 12.4 DLLs
- Windows x64 (CUDA 13) - CUDA 13.5 DLLs
- Windows arm64 (CUDA 13) - CUDA 13.5 DLLs
- Windows x64 (Vulkan)
- Windows arm64 (Vulkan)
- Windows x64 (OpenVINO)
- Windows x64 (SYCL)
- Windows x64 (ROCm 10.0)
openEuler:
- DISABLED
- openEuler x86 (310p)
- openEuler x86 (910b, ACL Graph)
- openEuler aarch64 (310p)
- openEuler aarch64 (910b, ACL Graph)
UI: