Add BPE pre-tokenization for Qwen2.

[jklj077]

Attempting to add BPE pre-tokenization support to Qwen2 (including Qwen/Qwen1.5/Qwen1.5-MoE). The same as cl100k or Llama3 except individual digits.

[github-actions]

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 551 iterations 🚀

Expand details for performance related PR only

• Concurrent users: 8, duration: 10m
• HTTP request : avg=8477.62ms p(95)=21689.25ms fails=, finish reason: stop=488 truncated=63
• Prompt processing (pp): avg=102.14tk/s p(95)=455.92tk/s
• Token generation (tg): avg=33.04tk/s p(95)=49.01tk/s
• ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=qwen2-pretok commit=08a436a89026c49dcee96aced59d94da0c202438

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 551 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1715173771 --> 1715174401
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 715.35, 715.35, 715.35, 715.35, 715.35, 581.23, 581.23, 581.23, 581.23, 581.23, 612.21, 612.21, 612.21, 612.21, 612.21, 647.28, 647.28, 647.28, 647.28, 647.28, 713.7, 713.7, 713.7, 713.7, 713.7, 719.17, 719.17, 719.17, 719.17, 719.17, 739.88, 739.88, 739.88, 739.88, 739.88, 757.39, 757.39, 757.39, 757.39, 757.39, 773.51, 773.51, 773.51, 773.51, 773.51, 779.52, 779.52, 779.52, 779.52, 779.52, 812.32, 812.32, 812.32, 812.32, 812.32, 813.47, 813.47, 813.47, 813.47, 813.47, 858.53, 858.53, 858.53, 858.53, 858.53, 880.79, 880.79, 880.79, 880.79, 880.79, 890.94, 890.94, 890.94, 890.94, 890.94, 889.3, 889.3, 889.3, 889.3, 889.3, 902.29, 902.29, 902.29, 902.29, 902.29, 904.85, 904.85, 904.85, 904.85, 904.85, 902.35, 902.35, 902.35, 902.35, 902.35, 907.05, 907.05, 907.05, 907.05, 907.05, 905.37, 905.37, 905.37, 905.37, 905.37, 904.92, 904.92, 904.92, 904.92, 904.92, 921.03, 921.03, 921.03, 921.03, 921.03, 916.87, 916.87, 916.87, 916.87, 916.87, 917.8, 917.8, 917.8, 917.8, 917.8, 927.75, 927.75, 927.75, 927.75, 927.75, 923.03, 923.03, 923.03, 923.03, 923.03, 920.82, 920.82, 920.82, 920.82, 920.82, 919.95, 919.95, 919.95, 919.95, 919.95, 917.46, 917.46, 917.46, 917.46, 917.46, 915.57, 915.57, 915.57, 915.57, 915.57, 917.7, 917.7, 917.7, 917.7, 917.7, 922.76, 922.76, 922.76, 922.76, 922.76, 934.34, 934.34, 934.34, 934.34, 934.34, 942.34, 942.34, 942.34, 942.34, 942.34, 940.09, 940.09, 940.09, 940.09, 940.09, 937.05, 937.05, 937.05, 937.05, 937.05, 934.11, 934.11, 934.11, 934.11, 934.11, 937.34, 937.34, 937.34, 937.34, 937.34, 937.09, 937.09, 937.09, 937.09, 937.09, 903.42, 903.42, 903.42, 903.42, 903.42, 906.6, 906.6, 906.6, 906.6, 906.6, 905.13, 905.13, 905.13, 905.13, 905.13, 904.15, 904.15, 904.15, 904.15, 904.15, 900.73, 900.73, 900.73, 900.73, 900.73, 903.4, 903.4, 903.4, 903.4, 903.4, 905.8, 905.8, 905.8, 905.8, 905.8, 905.16, 905.16, 905.16, 905.16, 905.16, 908.87, 908.87, 908.87, 908.87, 908.87, 911.72, 911.72, 911.72, 911.72, 911.72, 915.38, 915.38, 915.38, 915.38, 915.38, 913.31, 913.31, 913.31, 913.31, 913.31, 912.54, 912.54, 912.54, 912.54, 912.54, 915.08, 915.08, 915.08, 915.08, 915.08, 915.43, 915.43, 915.43, 915.43, 915.43, 914.11, 914.11, 914.11, 914.11, 914.11, 914.98, 914.98, 914.98, 914.98, 914.98, 915.75, 915.75, 915.75, 915.75, 915.75, 917.92, 917.92, 917.92, 917.92, 917.92, 917.49, 917.49, 917.49, 917.49, 917.49, 917.49]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 551 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1715173771 --> 1715174401
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 34.78, 34.78, 34.78, 34.78, 34.78, 30.25, 30.25, 30.25, 30.25, 30.25, 29.83, 29.83, 29.83, 29.83, 29.83, 31.34, 31.34, 31.34, 31.34, 31.34, 32.26, 32.26, 32.26, 32.26, 32.26, 33.89, 33.89, 33.89, 33.89, 33.89, 34.86, 34.86, 34.86, 34.86, 34.86, 35.04, 35.04, 35.04, 35.04, 35.04, 35.13, 35.13, 35.13, 35.13, 35.13, 34.96, 34.96, 34.96, 34.96, 34.96, 34.99, 34.99, 34.99, 34.99, 34.99, 34.46, 34.46, 34.46, 34.46, 34.46, 34.34, 34.34, 34.34, 34.34, 34.34, 32.97, 32.97, 32.97, 32.97, 32.97, 32.66, 32.66, 32.66, 32.66, 32.66, 32.53, 32.53, 32.53, 32.53, 32.53, 32.56, 32.56, 32.56, 32.56, 32.56, 31.95, 31.95, 31.95, 31.95, 31.95, 31.85, 31.85, 31.85, 31.85, 31.85, 31.53, 31.53, 31.53, 31.53, 31.53, 31.35, 31.35, 31.35, 31.35, 31.35, 31.37, 31.37, 31.37, 31.37, 31.37, 31.16, 31.16, 31.16, 31.16, 31.16, 31.27, 31.27, 31.27, 31.27, 31.27, 31.42, 31.42, 31.42, 31.42, 31.42, 31.47, 31.47, 31.47, 31.47, 31.47, 31.05, 31.05, 31.05, 31.05, 31.05, 30.91, 30.91, 30.91, 30.91, 30.91, 31.14, 31.14, 31.14, 31.14, 31.14, 31.29, 31.29, 31.29, 31.29, 31.29, 31.32, 31.32, 31.32, 31.32, 31.32, 31.57, 31.57, 31.57, 31.57, 31.57, 31.68, 31.68, 31.68, 31.68, 31.68, 31.61, 31.61, 31.61, 31.61, 31.61, 31.5, 31.5, 31.5, 31.5, 31.5, 31.3, 31.3, 31.3, 31.3, 31.3, 31.23, 31.23, 31.23, 31.23, 31.23, 31.23, 31.23, 31.23, 31.23, 31.23, 31.44, 31.44, 31.44, 31.44, 31.44, 31.54, 31.54, 31.54, 31.54, 31.54, 31.67, 31.67, 31.67, 31.67, 31.67, 31.61, 31.61, 31.61, 31.61, 31.61, 30.94, 30.94, 30.94, 30.94, 30.94, 30.89, 30.89, 30.89, 30.89, 30.89, 29.64, 29.64, 29.64, 29.64, 29.64, 29.46, 29.46, 29.46, 29.46, 29.46, 29.45, 29.45, 29.45, 29.45, 29.45, 29.56, 29.56, 29.56, 29.56, 29.56, 29.69, 29.69, 29.69, 29.69, 29.69, 29.81, 29.81, 29.81, 29.81, 29.81, 29.82, 29.82, 29.82, 29.82, 29.82, 29.79, 29.79, 29.79, 29.79, 29.79, 29.67, 29.67, 29.67, 29.67, 29.67, 29.63, 29.63, 29.63, 29.63, 29.63, 29.67, 29.67, 29.67, 29.67, 29.67, 29.84, 29.84, 29.84, 29.84, 29.84, 29.93, 29.93, 29.93, 29.93, 29.93, 30.03, 30.03, 30.03, 30.03, 30.03, 30.09, 30.09, 30.09, 30.09, 30.09, 30.1, 30.1, 30.1, 30.1, 30.1, 30.16]
                    

Loading

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 551 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1715173771 --> 1715174401
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.24, 0.24, 0.24, 0.24, 0.24, 0.25, 0.25, 0.25, 0.25, 0.25, 0.19, 0.19, 0.19, 0.19, 0.19, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.15, 0.15, 0.15, 0.15, 0.15, 0.22, 0.22, 0.22, 0.22, 0.22, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.13, 0.13, 0.13, 0.13, 0.13, 0.28, 0.28, 0.28, 0.28, 0.28, 0.17, 0.17, 0.17, 0.17, 0.17, 0.18, 0.18, 0.18, 0.18, 0.18, 0.21, 0.21, 0.21, 0.21, 0.21, 0.22, 0.22, 0.22, 0.22, 0.22, 0.29, 0.29, 0.29, 0.29, 0.29, 0.27, 0.27, 0.27, 0.27, 0.27, 0.22, 0.22, 0.22, 0.22, 0.22, 0.18, 0.18, 0.18, 0.18, 0.18, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.08, 0.08, 0.08, 0.08, 0.08, 0.17, 0.17, 0.17, 0.17, 0.17, 0.28, 0.28, 0.28, 0.28, 0.28, 0.22, 0.22, 0.22, 0.22, 0.22, 0.11, 0.11, 0.11, 0.11, 0.11, 0.14, 0.14, 0.14, 0.14, 0.14, 0.11, 0.11, 0.11, 0.11, 0.11, 0.1, 0.1, 0.1, 0.1, 0.1, 0.17, 0.17, 0.17, 0.17, 0.17, 0.15, 0.15, 0.15, 0.15, 0.15, 0.18, 0.18, 0.18, 0.18, 0.18, 0.35, 0.35, 0.35, 0.35, 0.35, 0.16, 0.16, 0.16, 0.16, 0.16, 0.23, 0.23, 0.23, 0.23, 0.23, 0.08, 0.08, 0.08, 0.08, 0.08, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.36, 0.36, 0.36, 0.36, 0.36, 0.5, 0.5, 0.5, 0.5, 0.5, 0.55, 0.55, 0.55, 0.55, 0.55, 0.61, 0.61, 0.61, 0.61, 0.61, 0.4, 0.4, 0.4, 0.4, 0.4, 0.19, 0.19, 0.19, 0.19, 0.19, 0.18, 0.18, 0.18, 0.18, 0.18, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.32, 0.32, 0.32, 0.32, 0.32, 0.22, 0.22, 0.22, 0.22, 0.22, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.11, 0.11, 0.11, 0.11, 0.11, 0.06, 0.06, 0.06, 0.06, 0.06, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.21, 0.21, 0.21, 0.21, 0.21, 0.16, 0.16, 0.16, 0.16, 0.16, 0.22]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 551 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1715173771 --> 1715174401
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0]
                    

Loading

[teleprint-me]

Should I not include Qwen in #7117?

[ggerganov]

Should I not include Qwen in #7117?

Yes, let's merge this PR first because it also has tests