Skip to content

Add measured CLI timings and richer benchmark variants#1130

Merged
frankslin merged 3 commits intoBYVoid:masterfrom
frankslin:upstream-master
Apr 20, 2026
Merged

Add measured CLI timings and richer benchmark variants#1130
frankslin merged 3 commits intoBYVoid:masterfrom
frankslin:upstream-master

Conversation

@frankslin
Copy link
Copy Markdown
Collaborator

Extend the benchmark binary to compare compiled ocd2 configs against temporary text-json variants without duplicating committed config files. The benchmark runner now rewrites config dictionaries to text inputs on the fly, registers grouped in-process and command-line benchmark families, and prints grouped console output with explicit throughput plus command-line stage breakdowns.

Add --measured_result to the opencc command-line tool so end-to-end benchmark runs can record structured load/convert/write/total timing data as JSON. Cover the new option with a command-line test and make the benchmark target depend on the opencc executable so command-line benchmarks run reliably in the build tree.

Extend the benchmark binary to compare compiled ocd2 configs against
temporary text-json variants without duplicating committed config files.
The benchmark runner now rewrites config dictionaries to text inputs on
the fly, registers grouped in-process and command-line benchmark families,
and prints grouped console output with explicit throughput plus
command-line stage breakdowns.

Add --measured_result to the opencc command-line tool so end-to-end
benchmark runs can record structured load/convert/write/total timing
data as JSON. Cover the new option with a command-line test and make
the benchmark target depend on the opencc executable so command-line
benchmarks run reliably in the build tree.
@frankslin
Copy link
Copy Markdown
Collaborator Author

Example result:

1: Run on (8 X 24.121 MHz CPU s)
1: CPU Caches:
1:   L1 Data 64 KiB
1:   L1 Instruction 128 KiB
1:   L2 Unified 4096 KiB (x8)
1: Load Average: 1.57, 1.66, 2.14
1: [Initialization]
1: ------------------------------------------------------------------------------------------------------
1: Benchmark                                                            Time             CPU   Iterations
1: ------------------------------------------------------------------------------------------------------
1: BM_Initialization/hk2s/ocd2                                       1406   us       1403   us        493
1: BM_Initialization/hk2s/text_json                                  3114   us       3113   us        223
1: BM_Initialization/hk2t/ocd2                                        218   us        218   us       3212
1: BM_Initialization/hk2t/text_json                                   378   us        377   us       1838
1: BM_Initialization/jp2t/ocd2                                        324   us        323   us       2168
1: BM_Initialization/jp2t/text_json                                   617   us        615   us       1108
1: BM_Initialization/s2hk/ocd2                                      41904   us      41896   us         17
1: BM_Initialization/s2hk/text_json                                112368   us     112314   us          6
1: BM_Initialization/s2t/ocd2                                       42986   us      42919   us         16
1: BM_Initialization/s2t/text_json                                 112874   us     112874   us          6
1: BM_Initialization/s2tw/ocd2                                      42570   us      42546   us         17
1: BM_Initialization/s2tw/text_json                                113160   us     113159   us          6
1: BM_Initialization/s2twp/ocd2                                     43496   us      43496   us         16
1: BM_Initialization/s2twp/text_json                               114006   us     114006   us          6
1: BM_Initialization/t2hk/ocd2                                        106   us        106   us       6607
1: BM_Initialization/t2hk/text_json                                   142   us        142   us       4934
1: BM_Initialization/t2jp/ocd2                                        249   us        249   us       2816
1: BM_Initialization/t2jp/text_json                                   471   us        471   us       1488
1: BM_Initialization/t2s/ocd2                                        1296   us       1296   us        540
1: BM_Initialization/t2s/text_json                                   2856   us       2856   us        244
1: BM_Initialization/t2tw/ocd2                                       91.3   us       91.3   us       7623
1: BM_Initialization/t2tw/text_json                                   115   us        115   us       6076
1: BM_Initialization/tw2s/ocd2                                       1376   us       1376   us        510
1: BM_Initialization/tw2s/text_json                                  3006   us       3006   us        231
1: BM_Initialization/tw2sp/ocd2                                      1623   us       1623   us        430
1: BM_Initialization/tw2sp/text_json                                 3843   us       3842   us        184
1: BM_Initialization/tw2t/ocd2                                        148   us        148   us       4727
1: BM_Initialization/tw2t/text_json                                   214   us        214   us       3277
1: BM_Initialization/s2twp_jieba/ocd2                              168424   us     168384   us          4
1: BM_Initialization/s2twp_jieba/text_json                         206202   us     206111   us          3
1:
1: [Convert Long Text]
1: -----------------------------------------------------------------------------------------------------------------------
1: Benchmark                                                            Time             CPU   Iterations       Throughput
1: -----------------------------------------------------------------------------------------------------------------------
1: BM_ConvertLongText/s2t/ocd2                                        588   ms        588   ms          1        3.17 MB/s
1: BM_ConvertLongText/s2t/text_json                                   579   ms        579   ms          1        3.22 MB/s
1: BM_ConvertLongText/s2twp/ocd2                                      932   ms        932   ms          1        2.00 MB/s
1: BM_ConvertLongText/s2twp/text_json                                 922   ms        922   ms          1        2.02 MB/s
1: BM_ConvertLongText/s2twp_jieba/ocd2                                990   ms        990   ms          1        1.89 MB/s
1: BM_ConvertLongText/s2twp_jieba/text_json                           975   ms        975   ms          1        1.91 MB/s
1:
1: [Convert]
1: -----------------------------------------------------------------------------------------------------------------------
1: Benchmark                                                            Time             CPU   Iterations       Throughput
1: -----------------------------------------------------------------------------------------------------------------------
1: BM_Convert/s2t/ocd2/100                                           1.06   ms       1.06   ms        662        3.95 MB/s
1: BM_Convert/s2t/ocd2/1000                                          10.9   ms       10.9   ms         63        3.95 MB/s
1: BM_Convert/s2t/ocd2/10000                                          112   ms        112   ms          6        3.93 MB/s
1: BM_Convert/s2t/ocd2/100000                                        1150   ms       1150   ms          1        3.90 MB/s
1: BM_Convert/s2t/text_json/100                                      1.06   ms       1.06   ms        665        3.97 MB/s
1: BM_Convert/s2t/text_json/1000                                     10.8   ms       10.8   ms         65        3.95 MB/s
1: BM_Convert/s2t/text_json/10000                                     111   ms        111   ms          6        3.96 MB/s
1: BM_Convert/s2t/text_json/100000                                   1138   ms       1138   ms          1        3.95 MB/s
1: BM_Convert/s2twp/ocd2/100                                         1.97   ms       1.97   ms        352        2.13 MB/s
1: BM_Convert/s2twp/ocd2/1000                                        20.2   ms       20.2   ms         35        2.12 MB/s
1: BM_Convert/s2twp/ocd2/10000                                        206   ms        206   ms          3        2.13 MB/s
1: BM_Convert/s2twp/ocd2/100000                                      2124   ms       2124   ms          1        2.11 MB/s
1: BM_Convert/s2twp/text_json/100                                    1.96   ms       1.96   ms        356        2.13 MB/s
1: BM_Convert/s2twp/text_json/1000                                   20.3   ms       20.3   ms         35        2.11 MB/s
1: BM_Convert/s2twp/text_json/10000                                   208   ms        208   ms          3        2.11 MB/s
1: BM_Convert/s2twp/text_json/100000                                 2114   ms       2114   ms          1        2.12 MB/s
1: BM_Convert/s2twp_jieba/ocd2/100                                   2.35   ms       2.35   ms        299        1.79 MB/s
1: BM_Convert/s2twp_jieba/ocd2/1000                                  23.9   ms       23.9   ms         29        1.80 MB/s
1: BM_Convert/s2twp_jieba/ocd2/10000                                  249   ms        249   ms          3        1.76 MB/s
1: BM_Convert/s2twp_jieba/ocd2/100000                                2505   ms       2505   ms          1        1.79 MB/s
1: BM_Convert/s2twp_jieba/text_json/100                              2.33   ms       2.33   ms        297        1.80 MB/s
1: BM_Convert/s2twp_jieba/text_json/1000                             24.4   ms       24.4   ms         30        1.76 MB/s
1: BM_Convert/s2twp_jieba/text_json/10000                             249   ms        249   ms          3        1.76 MB/s
1: BM_Convert/s2twp_jieba/text_json/100000                           2527   ms       2527   ms          1        1.78 MB/s
1:
1: [Command Line Long Text]
1: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1: Benchmark                                                            Time             CPU   Iterations       Throughput       LoadMs    ConvertMs      WriteMs      TotalMs
1: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1: BM_CommandLineLongText/s2t/ocd2/min_time:0.010                     597   ms       1.78   ms          1        3.22 MB/s     43.84 ms    533.11 ms      0.37 ms    579.84 ms
1: BM_CommandLineLongText/s2t/text_json/min_time:0.010                653   ms      0.428   ms          1        2.92 MB/s    113.70 ms    523.54 ms      0.33 ms    639.42 ms
1: BM_CommandLineLongText/s2twp/ocd2/min_time:0.010                   928   ms      0.420   ms          1        2.04 MB/s     43.04 ms    870.34 ms      0.51 ms    915.72 ms
1: BM_CommandLineLongText/s2twp/text_json/min_time:0.010              997   ms       1.70   ms          1        1.90 MB/s    114.26 ms    865.50 ms      0.51 ms    982.58 ms
1: BM_CommandLineLongText/s2twp_jieba/ocd2/min_time:0.010            1410   ms       2.17   ms          1        1.39 MB/s    335.22 ms    999.65 ms      0.43 ms   1338.14 ms
1: BM_CommandLineLongText/s2twp_jieba/text_json/min_time:0.010       1468   ms      0.536   ms          1        1.34 MB/s    374.55 ms   1021.25 ms      0.45 ms   1397.99 ms
1:
1: [Command Line]
1: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1: Benchmark                                                            Time             CPU   Iterations       Throughput       LoadMs    ConvertMs      WriteMs      TotalMs
1: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1: BM_CommandLine/s2t/ocd2/100/min_time:0.010                        57.6   ms       1.06   ms          1        0.09 MB/s     43.10 ms      1.08 ms      0.02 ms     44.52 ms
1: BM_CommandLine/s2t/ocd2/1000/min_time:0.010                       64.2   ms      0.275   ms          1        0.80 MB/s     42.66 ms     10.75 ms      0.02 ms     53.69 ms
1: BM_CommandLine/s2t/ocd2/10000/min_time:0.010                       163   ms      0.244   ms          1        2.87 MB/s     42.69 ms    109.53 ms      0.08 ms    152.67 ms
1: BM_CommandLine/s2t/ocd2/100000/min_time:0.010                     1196   ms      0.544   ms          1        3.80 MB/s     42.50 ms   1128.32 ms      1.02 ms   1182.47 ms
1: BM_CommandLine/s2t/text_json/100/min_time:0.010                    128   ms      0.995   ms          1        0.04 MB/s    113.82 ms      1.07 ms      0.02 ms    115.21 ms
1: BM_CommandLine/s2t/text_json/1000/min_time:0.010                   135   ms      0.267   ms          1        0.35 MB/s    113.19 ms     10.74 ms      0.03 ms    124.19 ms
1: BM_CommandLine/s2t/text_json/10000/min_time:0.010                  240   ms      0.285   ms          1        1.93 MB/s    113.13 ms    112.83 ms      0.09 ms    227.31 ms
1: BM_CommandLine/s2t/text_json/100000/min_time:0.010                1321   ms      0.382   ms          1        3.44 MB/s    120.23 ms   1172.16 ms      1.28 ms   1304.91 ms
1: BM_CommandLine/s2twp/ocd2/100/min_time:0.010                      59.6   ms      0.881   ms          1        0.09 MB/s     42.97 ms      1.96 ms      0.02 ms     45.24 ms
1: BM_CommandLine/s2twp/ocd2/1000/min_time:0.010                     73.6   ms      0.260   ms          1        0.68 MB/s     42.92 ms     19.95 ms      0.02 ms     63.09 ms
1: BM_CommandLine/s2twp/ocd2/10000/min_time:0.010                     260   ms      0.251   ms          1        1.77 MB/s     43.02 ms    203.40 ms      0.12 ms    248.15 ms
1: BM_CommandLine/s2twp/ocd2/100000/min_time:0.010                   2193   ms      0.535   ms          1        2.06 MB/s     43.73 ms   2122.20 ms      1.23 ms   2179.16 ms
1: BM_CommandLine/s2twp/text_json/100/min_time:0.010                  131   ms       1.13   ms          1        0.04 MB/s    115.57 ms      2.03 ms      0.03 ms    117.85 ms
1: BM_CommandLine/s2twp/text_json/1000/min_time:0.010                 147   ms      0.287   ms          1        0.32 MB/s    115.33 ms     19.78 ms      0.03 ms    135.48 ms
1: BM_CommandLine/s2twp/text_json/10000/min_time:0.010                333   ms      0.304   ms          1        1.37 MB/s    114.66 ms    204.75 ms      0.08 ms    321.19 ms
1: BM_CommandLine/s2twp/text_json/100000/min_time:0.010              2241   ms      0.526   ms          1        2.02 MB/s    114.04 ms   2100.33 ms      0.95 ms   2226.28 ms
1: BM_CommandLine/s2twp_jieba/ocd2/100/min_time:0.010                 402   ms       1.11   ms          1        0.01 MB/s    343.19 ms      2.45 ms      0.04 ms    346.01 ms
1: BM_CommandLine/s2twp_jieba/ocd2/1000/min_time:0.010                425   ms      0.363   ms          1        0.12 MB/s    342.19 ms     23.92 ms      0.04 ms    366.49 ms
1: BM_CommandLine/s2twp_jieba/ocd2/10000/min_time:0.010               664   ms      0.449   ms          1        0.75 MB/s    339.41 ms    247.21 ms      0.18 ms    588.29 ms
1: BM_CommandLine/s2twp_jieba/ocd2/100000/min_time:0.010             3078   ms      0.441   ms          1        1.49 MB/s    336.79 ms   2659.14 ms      1.10 ms   3009.20 ms
1: BM_CommandLine/s2twp_jieba/text_json/100/min_time:0.010            431   ms      0.377   ms          1        0.01 MB/s    373.01 ms      2.38 ms      0.04 ms    375.75 ms
1: BM_CommandLine/s2twp_jieba/text_json/1000/min_time:0.010           455   ms      0.360   ms          1        0.11 MB/s    376.23 ms     24.18 ms      0.03 ms    400.80 ms
1: BM_CommandLine/s2twp_jieba/text_json/10000/min_time:0.010          681   ms      1.000   ms          1        0.71 MB/s    374.50 ms    245.90 ms      0.10 ms    621.66 ms
1: BM_CommandLine/s2twp_jieba/text_json/100000/min_time:0.010        2994   ms      0.590   ms          1        1.53 MB/s    372.98 ms   2538.95 ms      1.01 ms   2925.15 ms
1/1 Test #1: BenchmarkTest ....................   Passed   97.67 sec

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 28dfb5e98a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchmark/Performance.cpp Outdated
Comment thread src/benchmark/Performance.cpp Outdated
Use CMake's resolved opencc target path for command-line benchmarks so Windows
multi-config generators launch the correct executable instead of hardcoding
src/tools/opencc(.exe).

Write generated benchmark config variants under the benchmark build directory
instead of the source config directories, which keeps the source tree clean and
allows read-only source checkouts to run the benchmark suite.
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ff79da9658

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchmark/Performance.cpp
Comment thread src/benchmark/Performance.cpp
Rewrite benchmark-generated segmentation resource paths so jieba text_json cases
keep resolving cppjieba dictionary assets after configs move into the build tree.

Wrap BM_CommandLine system commands with an outer quoted string on Windows,
matching the command-line tests and avoiding cmd.exe path parsing failures when
build paths contain spaces.
@frankslin frankslin merged commit 642a86e into BYVoid:master Apr 20, 2026
27 checks passed
@frankslin frankslin deleted the upstream-master branch April 20, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant