llama : cache llama_token_to_piece #7587

ggerganov · 2024-05-28T10:16:50Z

Build llama_token_to_piece caches to speed-up sampling with grammar

github-actions · 2024-05-28T10:45:47Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 526 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8887.35ms p(95)=21250.86ms fails=, finish reason: stop=465 truncated=61
Prompt processing (pp): avg=116.81tk/s p(95)=574.35tk/s
Token generation (tg): avg=31.64tk/s p(95)=44.82tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/cache-token-to-piece commit=8a8f8b953f6d21c2be62fb0e8f8c509d58b8c6ca

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 526 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717011448 --> 1717012072
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 308.81, 308.81, 308.81, 308.81, 308.81, 474.11, 474.11, 474.11, 474.11, 474.11, 520.65, 520.65, 520.65, 520.65, 520.65, 584.02, 584.02, 584.02, 584.02, 584.02, 600.53, 600.53, 600.53, 600.53, 600.53, 601.72, 601.72, 601.72, 601.72, 601.72, 606.06, 606.06, 606.06, 606.06, 606.06, 620.71, 620.71, 620.71, 620.71, 620.71, 642.66, 642.66, 642.66, 642.66, 642.66, 661.73, 661.73, 661.73, 661.73, 661.73, 666.58, 666.58, 666.58, 666.58, 666.58, 688.9, 688.9, 688.9, 688.9, 688.9, 694.91, 694.91, 694.91, 694.91, 694.91, 700.79, 700.79, 700.79, 700.79, 700.79, 717.39, 717.39, 717.39, 717.39, 717.39, 718.05, 718.05, 718.05, 718.05, 718.05, 726.48, 726.48, 726.48, 726.48, 726.48, 723.71, 723.71, 723.71, 723.71, 723.71, 723.5, 723.5, 723.5, 723.5, 723.5, 750.32, 750.32, 750.32, 750.32, 750.32, 754.76, 754.76, 754.76, 754.76, 754.76, 764.77, 764.77, 764.77, 764.77, 764.77, 766.44, 766.44, 766.44, 766.44, 766.44, 771.42, 771.42, 771.42, 771.42, 771.42, 790.47, 790.47, 790.47, 790.47, 790.47, 791.88, 791.88, 791.88, 791.88, 791.88, 793.05, 793.05, 793.05, 793.05, 793.05, 809.38, 809.38, 809.38, 809.38, 809.38, 806.43, 806.43, 806.43, 806.43, 806.43, 805.76, 805.76, 805.76, 805.76, 805.76, 806.06, 806.06, 806.06, 806.06, 806.06, 807.65, 807.65, 807.65, 807.65, 807.65, 811.81, 811.81, 811.81, 811.81, 811.81, 809.56, 809.56, 809.56, 809.56, 809.56, 812.36, 812.36, 812.36, 812.36, 812.36, 821.55, 821.55, 821.55, 821.55, 821.55, 828.62, 828.62, 828.62, 828.62, 828.62, 825.23, 825.23, 825.23, 825.23, 825.23, 823.15, 823.15, 823.15, 823.15, 823.15, 823.26, 823.26, 823.26, 823.26, 823.26, 828.27, 828.27, 828.27, 828.27, 828.27, 829.4, 829.4, 829.4, 829.4, 829.4, 834.66, 834.66, 834.66, 834.66, 834.66, 802.54, 802.54, 802.54, 802.54, 802.54, 790.74, 790.74, 790.74, 790.74, 790.74, 790.49, 790.49, 790.49, 790.49, 790.49, 789.05, 789.05, 789.05, 789.05, 789.05, 793.62, 793.62, 793.62, 793.62, 793.62, 790.86, 790.86, 790.86, 790.86, 790.86, 790.52, 790.52, 790.52, 790.52, 790.52, 794.02, 794.02, 794.02, 794.02, 794.02, 796.05, 796.05, 796.05, 796.05, 796.05, 800.51, 800.51, 800.51, 800.51, 800.51, 803.24, 803.24, 803.24, 803.24, 803.24, 804.0, 804.0, 804.0, 804.0, 804.0, 808.72, 808.72, 808.72, 808.72, 808.72, 808.05, 808.05, 808.05, 808.05, 808.05, 809.55, 809.55, 809.55, 809.55, 809.55, 809.59, 809.59, 809.59, 809.59, 809.59, 810.89, 810.89, 810.89, 810.89, 810.89, 810.96, 810.96, 810.96]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 526 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717011448 --> 1717012072
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 41.16, 41.16, 41.16, 41.16, 41.16, 42.15, 42.15, 42.15, 42.15, 42.15, 27.0, 27.0, 27.0, 27.0, 27.0, 28.43, 28.43, 28.43, 28.43, 28.43, 29.25, 29.25, 29.25, 29.25, 29.25, 30.05, 30.05, 30.05, 30.05, 30.05, 31.2, 31.2, 31.2, 31.2, 31.2, 31.43, 31.43, 31.43, 31.43, 31.43, 32.37, 32.37, 32.37, 32.37, 32.37, 32.62, 32.62, 32.62, 32.62, 32.62, 32.75, 32.75, 32.75, 32.75, 32.75, 32.65, 32.65, 32.65, 32.65, 32.65, 32.49, 32.49, 32.49, 32.49, 32.49, 30.99, 30.99, 30.99, 30.99, 30.99, 30.03, 30.03, 30.03, 30.03, 30.03, 29.68, 29.68, 29.68, 29.68, 29.68, 28.47, 28.47, 28.47, 28.47, 28.47, 28.15, 28.15, 28.15, 28.15, 28.15, 28.26, 28.26, 28.26, 28.26, 28.26, 28.4, 28.4, 28.4, 28.4, 28.4, 28.51, 28.51, 28.51, 28.51, 28.51, 28.81, 28.81, 28.81, 28.81, 28.81, 29.01, 29.01, 29.01, 29.01, 29.01, 29.3, 29.3, 29.3, 29.3, 29.3, 29.25, 29.25, 29.25, 29.25, 29.25, 29.23, 29.23, 29.23, 29.23, 29.23, 29.47, 29.47, 29.47, 29.47, 29.47, 29.63, 29.63, 29.63, 29.63, 29.63, 29.37, 29.37, 29.37, 29.37, 29.37, 29.38, 29.38, 29.38, 29.38, 29.38, 29.64, 29.64, 29.64, 29.64, 29.64, 29.73, 29.73, 29.73, 29.73, 29.73, 29.84, 29.84, 29.84, 29.84, 29.84, 30.01, 30.01, 30.01, 30.01, 30.01, 30.14, 30.14, 30.14, 30.14, 30.14, 30.1, 30.1, 30.1, 30.1, 30.1, 29.98, 29.98, 29.98, 29.98, 29.98, 29.96, 29.96, 29.96, 29.96, 29.96, 29.7, 29.7, 29.7, 29.7, 29.7, 29.87, 29.87, 29.87, 29.87, 29.87, 29.93, 29.93, 29.93, 29.93, 29.93, 30.05, 30.05, 30.05, 30.05, 30.05, 30.23, 30.23, 30.23, 30.23, 30.23, 30.14, 30.14, 30.14, 30.14, 30.14, 30.1, 30.1, 30.1, 30.1, 30.1, 29.48, 29.48, 29.48, 29.48, 29.48, 28.91, 28.91, 28.91, 28.91, 28.91, 28.88, 28.88, 28.88, 28.88, 28.88, 28.89, 28.89, 28.89, 28.89, 28.89, 28.81, 28.81, 28.81, 28.81, 28.81, 28.81, 28.81, 28.81, 28.81, 28.81, 28.85, 28.85, 28.85, 28.85, 28.85, 28.91, 28.91, 28.91, 28.91, 28.91, 28.99, 28.99, 28.99, 28.99, 28.99, 28.9, 28.9, 28.9, 28.9, 28.9, 28.88, 28.88, 28.88, 28.88, 28.88, 28.84, 28.84, 28.84, 28.84, 28.84, 28.94, 28.94, 28.94, 28.94, 28.94, 29.05, 29.05, 29.05, 29.05, 29.05, 29.07, 29.07, 29.07, 29.07, 29.07, 29.18, 29.18, 29.18]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 526 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717011448 --> 1717012072
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.12, 0.12, 0.12, 0.12, 0.12, 0.41, 0.41, 0.41, 0.41, 0.41, 0.29, 0.29, 0.29, 0.29, 0.29, 0.11, 0.11, 0.11, 0.11, 0.11, 0.19, 0.19, 0.19, 0.19, 0.19, 0.22, 0.22, 0.22, 0.22, 0.22, 0.16, 0.16, 0.16, 0.16, 0.16, 0.15, 0.15, 0.15, 0.15, 0.15, 0.13, 0.13, 0.13, 0.13, 0.13, 0.17, 0.17, 0.17, 0.17, 0.17, 0.22, 0.22, 0.22, 0.22, 0.22, 0.24, 0.24, 0.24, 0.24, 0.24, 0.36, 0.36, 0.36, 0.36, 0.36, 0.34, 0.34, 0.34, 0.34, 0.34, 0.37, 0.37, 0.37, 0.37, 0.37, 0.39, 0.39, 0.39, 0.39, 0.39, 0.32, 0.32, 0.32, 0.32, 0.32, 0.26, 0.26, 0.26, 0.26, 0.26, 0.11, 0.11, 0.11, 0.11, 0.11, 0.24, 0.24, 0.24, 0.24, 0.24, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.18, 0.18, 0.18, 0.18, 0.18, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.32, 0.32, 0.32, 0.32, 0.32, 0.09, 0.09, 0.09, 0.09, 0.09, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.14, 0.14, 0.14, 0.14, 0.14, 0.11, 0.11, 0.11, 0.11, 0.11, 0.24, 0.24, 0.24, 0.24, 0.24, 0.19, 0.19, 0.19, 0.19, 0.19, 0.43, 0.43, 0.43, 0.43, 0.43, 0.18, 0.18, 0.18, 0.18, 0.18, 0.08, 0.08, 0.08, 0.08, 0.08, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.24, 0.24, 0.24, 0.24, 0.24, 0.5, 0.5, 0.5, 0.5, 0.5, 0.64, 0.64, 0.64, 0.64, 0.64, 0.36, 0.36, 0.36, 0.36, 0.36, 0.16, 0.16, 0.16, 0.16, 0.16, 0.21, 0.21, 0.21, 0.21, 0.21, 0.26, 0.26, 0.26, 0.26, 0.26, 0.3, 0.3, 0.3, 0.3, 0.3, 0.13, 0.13, 0.13, 0.13, 0.13, 0.08, 0.08, 0.08, 0.08, 0.08, 0.18, 0.18, 0.18, 0.18, 0.18, 0.28, 0.28, 0.28, 0.28, 0.28, 0.26, 0.26, 0.26, 0.26, 0.26, 0.25, 0.25, 0.25, 0.25, 0.25, 0.1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.2, 0.2, 0.2, 0.2, 0.09, 0.09, 0.09, 0.09, 0.09, 0.17, 0.17, 0.17, 0.17, 0.17, 0.2, 0.2, 0.2]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 526 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717011448 --> 1717012072
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0]

skoulik · 2024-05-28T11:27:50Z

#7554 (comment)

ggml-ci

llama.cpp

HanClinto · 2024-05-29T17:31:29Z

llama.cpp

@@ -2162,7 +2163,11 @@ struct llama_vocab {
    std::unordered_map<token, id> token_to_id;
    std::vector<token_data>       id_to_token;

-    std::vector<id> special_tokens_cache;
+    bool has_cache = false;


Is there a mechanism by which the vocab can be loaded without having a cache in place? If not, I'm wondering if has_cache is useful right now...?

There was a way to exit early before creating the cache if the tokenizer was unknown. I've removed this path by throwing an exception: 1494a18

There is another path where the GGUF explicitly does not contain a vocabulary: "no_vocab". In that case calling any of the functions that rely on a cache would throw exception due to accessing the caches via cache.at(). I think this makes sense

Removed has_cache and replaced the unordered maps with vectors

Perfect, thank you!

I like all your changes here -- this all feels really good. The only other thing that I'll note is the caveat that I noted on @ochafik 's similar PR in #6811 :

#6811 (comment)

I think it'd be simpler to leave it as is and keep it as an area where to potentially squeeze a couple of MB when times are scarce. wdyt?

This also sounds not unreasonable, but I don't know how to weigh such things. I know I really like grammar-constrained sampling, but I don't know how popular the feature is overall, and is it worth negatively impacting hyper-resource-constrained usages (such as Raspberry Pis or whatnot) vs. grammars? That's what I'm unable to weigh -- I feel like that's a strategic decision that's a bit above my level.

In short, we don't need the cache for situations that don't use grammars, and we're adding a bit of memory usage (n_vocab*2) to every context that we're creating. On most systems this isn't a problem, but on highly-constrained systems (such as Raspberry Pi and whatnot) then this is wasted memory.

How do we weigh the interests of memory-constrained users vs. grammar-enabled users? That's something that I'm not able to make, but overall I think that improving speed performance on grammar-enabled sampling is going to benefit the largest number of people, and the ultra-constrained users are going to be pretty small. We might want to make a note somewhere in a comment that if one is looking for a way to decrease memory usage that they could disable the caching, but beyond that we're probably fine with kicking that can down the road.

I added a log for the memory usage of the "token to piece" caches:

# llama 3 llm_load_vocab: token to piece cache size = 1.5928 MB

I think this is completely fine and no need to worry about it for now

Excellent, thank you! That was the one reservation that held me back from fully approving #6811 (I felt that choice required someone with a larger project scope than I have), so I'm very happy to have you weigh in on that.

llama.cpp

ggml-ci

llama.cpp

HanClinto · 2024-05-29T18:34:07Z

Additional ref: #6811

HanClinto

@ochafik may have some insights from his very similar PR, but overall this looks good to me!

HanClinto · 2024-05-29T20:04:53Z

Part of this may be touched in future server refactorings as well.

It would be nice to flag this and not build the cache if there isn't a grammar. However, the server complicates this -- at the time of server initialization, it's not yet known if the user's request is going to include a grammar or not.

So we need to take the time and memory to initialize the cache no matter what.

All that to say, there is room for additional improvement on this, but ultimately it feels like this is enough of a net win (and the grammar feature is so powerful) that we should include token caching in all cases for now.

slaren · 2024-05-29T20:06:35Z

Can't you initialize the cache the first time llama_sample_grammar is called?

ggerganov · 2024-05-30T07:28:30Z

We can lazy initialize the cache, but it's not just the grammar that benefits from it. All calls to llama_token_to_piece would be faster too if the cache is pre-computed at the start

ochafik · 2024-05-30T15:42:07Z

llama.cpp

@@ -18292,69 +18313,83 @@ static std::string llama_decode_text(const std::string & text) {

 // does not write null-terminator to buf
 int32_t llama_token_to_piece(const struct llama_model * model, llama_token token, char * buf, int32_t length, bool special) {
+    // if we have a cache - use it
+    {
+        const auto & cache = special ? model->vocab.cache_token_to_piece_special : model->vocab.cache_token_to_piece;


nit: Maybe we could get away w/ a single cache (built w/ special=true) and early-exit in special case at the top of the function?

int32_t llama_token_to_piece(const struct llama_model * model, llama_token token, char * buf, int32_t length, bool special) { if (!special && llama_is_control_token(model->vocab, token)) { return 0; } // if we have a cache - use it if (!model->vocab.cache_token_to_piece.empty()) { .... } ...

ochafik · 2024-05-30T15:50:39Z

@slaren re/ lazy init on first call of grammar usage, I did this in #6811 w/ an awkward mutex to guard against concurrent calls in the server case. It's prettier in this PR w/o that wart.

All calls to llama_token_to_piece would be faster too if the cache is pre-computed at the start

+1

ggerganov mentioned this pull request May 28, 2024

Bug: sample time becomes very long when using Llama-3 #7554

Closed

ggerganov force-pushed the gg/cache-token-to-piece branch from 92b88a0 to 3e5d281 Compare May 28, 2024 10:56

mofosyne added the Review Complexity : Low Trivial changes to code that most beginner devs (or those who want a break) can tackle. e.g. UI fix label May 28, 2024

ggerganov requested review from ochafik and HanClinto May 28, 2024 18:04

llama : cache llama_token_to_piece

9964cd0

ggml-ci

ggerganov force-pushed the gg/cache-token-to-piece branch from 3e5d281 to 9964cd0 Compare May 29, 2024 17:22

ggerganov added the merge ready indicates that this may be ready to merge soon and is just holding out in case of objections label May 29, 2024