Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) #7425

jaime-m-p · 2024-05-21T00:35:13Z

Bugfixing #7375 : Need rtrim after the pre-inserted BOS. More details here.

github-actions · 2024-05-21T01:04:10Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 531 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8827.26ms p(95)=23186.6ms fails=, finish reason: stop=468 truncated=63
Prompt processing (pp): avg=104.01tk/s p(95)=435.03tk/s
Token generation (tg): avg=32.11tk/s p(95)=46.81tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=tokenizer-spm-fixes commit=0594bfdd5e22e5df6ea44583088ca08609a70e00

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 531 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1716282012 --> 1716282640
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 626.54, 626.54, 626.54, 626.54, 626.54, 750.3, 750.3, 750.3, 750.3, 750.3, 745.22, 745.22, 745.22, 745.22, 745.22, 768.62, 768.62, 768.62, 768.62, 768.62, 856.96, 856.96, 856.96, 856.96, 856.96, 847.71, 847.71, 847.71, 847.71, 847.71, 849.26, 849.26, 849.26, 849.26, 849.26, 872.34, 872.34, 872.34, 872.34, 872.34, 866.74, 866.74, 866.74, 866.74, 866.74, 876.91, 876.91, 876.91, 876.91, 876.91, 895.11, 895.11, 895.11, 895.11, 895.11, 909.16, 909.16, 909.16, 909.16, 909.16, 902.35, 902.35, 902.35, 902.35, 902.35, 889.67, 889.67, 889.67, 889.67, 889.67, 894.49, 894.49, 894.49, 894.49, 894.49, 870.24, 870.24, 870.24, 870.24, 870.24, 861.4, 861.4, 861.4, 861.4, 861.4, 863.69, 863.69, 863.69, 863.69, 863.69, 870.1, 870.1, 870.1, 870.1, 870.1, 874.86, 874.86, 874.86, 874.86, 874.86, 874.59, 874.59, 874.59, 874.59, 874.59, 878.46, 878.46, 878.46, 878.46, 878.46, 878.67, 878.67, 878.67, 878.67, 878.67, 881.0, 881.0, 881.0, 881.0, 881.0, 871.94, 871.94, 871.94, 871.94, 871.94, 864.04, 864.04, 864.04, 864.04, 864.04, 878.75, 878.75, 878.75, 878.75, 878.75, 877.1, 877.1, 877.1, 877.1, 877.1, 875.2, 875.2, 875.2, 875.2, 875.2, 875.04, 875.04, 875.04, 875.04, 875.04, 878.79, 878.79, 878.79, 878.79, 878.79, 877.34, 877.34, 877.34, 877.34, 877.34, 875.68, 875.68, 875.68, 875.68, 875.68, 877.99, 877.99, 877.99, 877.99, 877.99, 887.73, 887.73, 887.73, 887.73, 887.73, 886.08, 886.08, 886.08, 886.08, 886.08, 888.57, 888.57, 888.57, 888.57, 888.57, 886.98, 886.98, 886.98, 886.98, 886.98, 882.31, 882.31, 882.31, 882.31, 882.31, 881.79, 881.79, 881.79, 881.79, 881.79, 881.18, 881.18, 881.18, 881.18, 881.18, 880.64, 880.64, 880.64, 880.64, 880.64, 871.01, 871.01, 871.01, 871.01, 871.01, 853.28, 853.28, 853.28, 853.28, 853.28, 853.14, 853.14, 853.14, 853.14, 853.14, 850.75, 850.75, 850.75, 850.75, 850.75, 849.35, 849.35, 849.35, 849.35, 849.35, 847.18, 847.18, 847.18, 847.18, 847.18, 847.58, 847.58, 847.58, 847.58, 847.58, 846.04, 846.04, 846.04, 846.04, 846.04, 849.15, 849.15, 849.15, 849.15, 849.15, 848.22, 848.22, 848.22, 848.22, 848.22, 851.14, 851.14, 851.14, 851.14, 851.14, 854.63, 854.63, 854.63, 854.63, 854.63, 851.73, 851.73, 851.73, 851.73, 851.73, 856.85, 856.85, 856.85, 856.85, 856.85, 854.16, 854.16, 854.16, 854.16, 854.16, 854.53, 854.53, 854.53, 854.53, 854.53, 855.7, 855.7, 855.7, 855.7, 855.7, 855.44, 855.44, 855.44, 855.44, 855.44]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 531 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1716282012 --> 1716282640
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 40.43, 40.43, 40.43, 40.43, 40.43, 34.38, 34.38, 34.38, 34.38, 34.38, 28.96, 28.96, 28.96, 28.96, 28.96, 31.05, 31.05, 31.05, 31.05, 31.05, 31.26, 31.26, 31.26, 31.26, 31.26, 31.37, 31.37, 31.37, 31.37, 31.37, 32.69, 32.69, 32.69, 32.69, 32.69, 33.49, 33.49, 33.49, 33.49, 33.49, 33.57, 33.57, 33.57, 33.57, 33.57, 33.62, 33.62, 33.62, 33.62, 33.62, 33.55, 33.55, 33.55, 33.55, 33.55, 33.04, 33.04, 33.04, 33.04, 33.04, 32.4, 32.4, 32.4, 32.4, 32.4, 32.39, 32.39, 32.39, 32.39, 32.39, 31.8, 31.8, 31.8, 31.8, 31.8, 30.53, 30.53, 30.53, 30.53, 30.53, 28.91, 28.91, 28.91, 28.91, 28.91, 28.92, 28.92, 28.92, 28.92, 28.92, 29.22, 29.22, 29.22, 29.22, 29.22, 29.01, 29.01, 29.01, 29.01, 29.01, 28.86, 28.86, 28.86, 28.86, 28.86, 28.81, 28.81, 28.81, 28.81, 28.81, 28.86, 28.86, 28.86, 28.86, 28.86, 29.16, 29.16, 29.16, 29.16, 29.16, 29.21, 29.21, 29.21, 29.21, 29.21, 29.41, 29.41, 29.41, 29.41, 29.41, 29.72, 29.72, 29.72, 29.72, 29.72, 29.66, 29.66, 29.66, 29.66, 29.66, 29.74, 29.74, 29.74, 29.74, 29.74, 29.83, 29.83, 29.83, 29.83, 29.83, 29.99, 29.99, 29.99, 29.99, 29.99, 30.07, 30.07, 30.07, 30.07, 30.07, 30.18, 30.18, 30.18, 30.18, 30.18, 30.29, 30.29, 30.29, 30.29, 30.29, 30.11, 30.11, 30.11, 30.11, 30.11, 30.03, 30.03, 30.03, 30.03, 30.03, 30.01, 30.01, 30.01, 30.01, 30.01, 29.74, 29.74, 29.74, 29.74, 29.74, 29.52, 29.52, 29.52, 29.52, 29.52, 29.49, 29.49, 29.49, 29.49, 29.49, 29.68, 29.68, 29.68, 29.68, 29.68, 29.9, 29.9, 29.9, 29.9, 29.9, 30.03, 30.03, 30.03, 30.03, 30.03, 29.87, 29.87, 29.87, 29.87, 29.87, 29.75, 29.75, 29.75, 29.75, 29.75, 29.44, 29.44, 29.44, 29.44, 29.44, 28.39, 28.39, 28.39, 28.39, 28.39, 28.43, 28.43, 28.43, 28.43, 28.43, 28.48, 28.48, 28.48, 28.48, 28.48, 28.35, 28.35, 28.35, 28.35, 28.35, 28.37, 28.37, 28.37, 28.37, 28.37, 28.4, 28.4, 28.4, 28.4, 28.4, 28.42, 28.42, 28.42, 28.42, 28.42, 28.49, 28.49, 28.49, 28.49, 28.49, 28.41, 28.41, 28.41, 28.41, 28.41, 28.41, 28.41, 28.41, 28.41, 28.41, 28.4, 28.4, 28.4, 28.4, 28.4, 28.48, 28.48, 28.48, 28.48, 28.48, 28.63, 28.63, 28.63, 28.63, 28.63, 28.71, 28.71, 28.71, 28.71, 28.71]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 531 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1716282012 --> 1716282640
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.2, 0.2, 0.2, 0.2, 0.2, 0.38, 0.38, 0.38, 0.38, 0.38, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.22, 0.22, 0.22, 0.22, 0.22, 0.17, 0.17, 0.17, 0.17, 0.17, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.19, 0.19, 0.19, 0.19, 0.19, 0.23, 0.23, 0.23, 0.23, 0.23, 0.28, 0.28, 0.28, 0.28, 0.28, 0.26, 0.26, 0.26, 0.26, 0.26, 0.27, 0.27, 0.27, 0.27, 0.27, 0.39, 0.39, 0.39, 0.39, 0.39, 0.42, 0.42, 0.42, 0.42, 0.42, 0.38, 0.38, 0.38, 0.38, 0.38, 0.24, 0.24, 0.24, 0.24, 0.24, 0.14, 0.14, 0.14, 0.14, 0.14, 0.2, 0.2, 0.2, 0.2, 0.2, 0.29, 0.29, 0.29, 0.29, 0.29, 0.33, 0.33, 0.33, 0.33, 0.33, 0.23, 0.23, 0.23, 0.23, 0.23, 0.13, 0.13, 0.13, 0.13, 0.13, 0.16, 0.16, 0.16, 0.16, 0.16, 0.27, 0.27, 0.27, 0.27, 0.27, 0.12, 0.12, 0.12, 0.12, 0.12, 0.28, 0.28, 0.28, 0.28, 0.28, 0.22, 0.22, 0.22, 0.22, 0.22, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.14, 0.14, 0.14, 0.14, 0.14, 0.15, 0.15, 0.15, 0.15, 0.15, 0.29, 0.29, 0.29, 0.29, 0.29, 0.28, 0.28, 0.28, 0.28, 0.28, 0.29, 0.29, 0.29, 0.29, 0.29, 0.28, 0.28, 0.28, 0.28, 0.28, 0.14, 0.14, 0.14, 0.14, 0.14, 0.09, 0.09, 0.09, 0.09, 0.09, 0.12, 0.12, 0.12, 0.12, 0.12, 0.11, 0.11, 0.11, 0.11, 0.11, 0.39, 0.39, 0.39, 0.39, 0.39, 0.56, 0.56, 0.56, 0.56, 0.56, 0.53, 0.53, 0.53, 0.53, 0.53, 0.52, 0.52, 0.52, 0.52, 0.52, 0.1, 0.1, 0.1, 0.1, 0.1, 0.22, 0.22, 0.22, 0.22, 0.22, 0.28, 0.28, 0.28, 0.28, 0.28, 0.18, 0.18, 0.18, 0.18, 0.18, 0.22, 0.22, 0.22, 0.22, 0.22, 0.13, 0.13, 0.13, 0.13, 0.13, 0.21, 0.21, 0.21, 0.21, 0.21, 0.33, 0.33, 0.33, 0.33, 0.33, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.23, 0.23, 0.23, 0.23, 0.23, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 531 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1716282012 --> 1716282640
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0]

This reverts commit 7fb66eb.

ggerganov · 2024-05-21T08:36:07Z

I reverted the changes that I made to the server tests - these should no longer be necessary

* Update brute force test: add_special * Update brute force test: default values for add_bos_token and add_eos_token * Enable rtrim when pre-inserting BOS Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Revert "server : fix test regexes"

mofosyne · 2024-05-22T07:51:40Z

On main branch this caused a failure under 'CI / macOS-latest-cmake-arm64 (push) Failing after 2m' for this test

20 - test-backend-ops

ggerganov · 2024-05-22T08:13:47Z

It's not related - some of the CPY tests occasionally produce numerical differences that slightly exceed the epsilon threshold:

20:   CPY(type_src=f32,type_dst=q5_1,ne=[256,4,4,4]): [CPY] NMSE = 0.000000730 > 0.000000100 FAIL

The difference is due to different rounding mode between the CPU / GPU: #4698

jaime-m-p added 4 commits May 21, 2024 02:07

Style

c78a53a

Update brute force test: add_special

3d490e8

Enable rtrim when pre-inserting BOS

9b21dc3

Default values for add_bos_token and add_eos_token

f2340b4

github-actions bot added testing Everything test related python python script changes labels May 21, 2024

Revert "server : fix test regexes"

0594bfd

This reverts commit 7fb66eb.

github-actions bot added examples server labels May 21, 2024

ggerganov approved these changes May 21, 2024

View reviewed changes

jaime-m-p merged commit d7e852c into ggml-org:master May 21, 2024

This was referenced May 25, 2024

Old GGUF have broken tokenization and there is no warning #7476

Closed

Bug: Inconsistent ggml-4-x86-cuda-v100 ci failures on master #7613

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) #7425

Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) #7425

Uh oh!

jaime-m-p commented May 21, 2024

Uh oh!

github-actions bot commented May 21, 2024 •

edited

Loading

Uh oh!

ggerganov commented May 21, 2024

Uh oh!

mofosyne commented May 22, 2024

Uh oh!

ggerganov commented May 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) #7425

Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) #7425

Uh oh!

Conversation

jaime-m-p commented May 21, 2024

Uh oh!

github-actions bot commented May 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented May 21, 2024

Uh oh!

mofosyne commented May 22, 2024

Uh oh!

ggerganov commented May 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented May 21, 2024 •

edited

Loading