Add generated-Ok() code-size + execution-speed benchmark harness#270
Open
AaronWebster wants to merge 2 commits into
Open
Add generated-Ok() code-size + execution-speed benchmark harness#270AaronWebster wants to merge 2 commits into
AaronWebster wants to merge 2 commits into
Conversation
Adopt the size-benchmark tooling from the #265->#267 branch stack (scripts/size_bench.py, scripts/size_comment.py, .github/workflows/code-size.yml, testdata/benchmark.emb) and extend it into the committed size+speed harness the Ok() optimization work needs, superseding the #252 embedded-bench scripts while folding their coverage back in. Size measurement (size_bench.py): - Add MicroBlaze little-endian targets: the AMD/Xilinx Vitis v11 toolchain (the actually-shipped target, authoritative for LE size; local-only, since it is proprietary), plus an open-source Bootlin microblazeel proxy for CI where Vitis is absent. Rename the existing MicroBlaze target to MicroBlaze BE. Per-compiler env support threads Vitis' LD_LIBRARY_PATH through every compiler/binutils call. - Fold back #252's dual-Ok coverage: measure DisjunctionConditionals::Ok() alongside LargeConditionals::Ok(). - Add four Ok() driver TUs the later optimization phases consume: reader-only, both-instantiated, writer-only, and residual-heavy (stubbed on DisjunctionConditionals until an OuterGuardedUnion schema lands). Reader vs writer Ok() symbols are split on the `const` in the demangled storage arg. Execution speed (opt-in --speed): - Deterministic retired-instruction count under qemu-user + a bundled TCG plugin (scripts/emboss_insncount.c; QEMU 10.x dropped contrib libinsn) for ARM/MicroBlaze BE/LE, plus native host wall-clock. The plugin-enabled qemu is built from source by scripts/build_qemu_plugin.sh and located via EMBOSS_QEMU_DIR/EMBOSS_QEMU_PLUGIN; when absent, speed degrades to host wall-clock and the size report is unaffected. CI (code-size.yml): provision the Bootlin microblazeel LE toolchain and, best-effort and cached, the plugin-enabled qemu so the per-PR comment can carry LE size and (when the build succeeds) retired-instruction speed. size_comment.py gains a Disj Ok() column and a speed table. Checkpoint 0: on origin/master this reproduces the verified LargeConditionals:: Ok() baseline exactly -- x86 3048, ARM 4628, MicroBlaze BE 7704, Vitis LE 8156 -- and emits a retired-instruction speed number per target.
Two measurement-only corrections to the Ok() size/speed harness that were derived while running later phases and are owed back to this branch: - Repoint the `residual_heavy` size driver and the `--speed` hot loop from the placeholder `DisjunctionConditionals` to `OuterGuardedUnion`, the residual-guarded schema that the residual-read-elimination work adds to testdata/many_conditionals.emb. Until that schema lands the driver simply fails to compile and is skipped (compile_obj returns False), so the size report is unaffected; once it lands, residual_heavy measures Lever A. - Build every Ok()/benchmark driver view from `unsigned char*` / `const unsigned char*` instead of `char*`. `char*` deduces ContiguousBuffer<char, ...>, which is not the canonical ReadWriteContiguousBuffer/ReadOnlyContiguousBuffer (unsigned char) that Emboss's own View/Writer aliases and real uint8_t-buffer users hit. Any codegen conditioned on the exact storage type is otherwise invisible to the harness.
This was referenced Jul 19, 2026
📐 Generated code size & instructions
Code size (compiled
|
| Target · Compiler | Code size | Instructions | Large Ok() |
Disj Ok() |
|---|---|---|---|---|
| x86-64 · gcc | 12842 B | 2364 | 2384 B | 407 B |
| x86-64 · clang | 6258 B | 873 | 1460 B | 290 B |
| ARM Cortex-M4 · gcc | 7146 B | 1799 | 1948 B | 212 B |
| MicroBlaze BE · gcc | 10957 B | 2452 | 7704 B | 540 B |
| MicroBlaze LE (Bootlin) · gcc | 11229 B | 2520 | 7704 B | 540 B |
-O2 / -O0
-O2
| Target · Compiler | Code size | Instructions | Large Ok() |
Disj Ok() |
|---|---|---|---|---|
| x86-64 · gcc | 5125 B | 868 | 9048 B | 343 B |
| x86-64 · clang | 3086 B | 451 | 4608 B | 189 B |
| ARM Cortex-M4 · gcc | 3322 B | 851 | 4768 B | 180 B |
| MicroBlaze BE · gcc | 6449 B | 1325 | 10200 B | — |
| MicroBlaze LE (Bootlin) · gcc | 5461 B | 1078 | 15608 B | 548 B |
-O0
| Target · Compiler | Code size | Instructions | Large Ok() |
Disj Ok() |
|---|---|---|---|---|
| x86-64 · gcc | 100054 B | 19618 | 6442 B | 900 B |
| x86-64 · clang | 76530 B | 13895 | 7278 B | 785 B |
| ARM Cortex-M4 · gcc | 59564 B | 19289 | 6788 B | 712 B |
| MicroBlaze BE · gcc | 107420 B | 23868 | 8276 B | 1280 B |
| MicroBlaze LE (Bootlin) · gcc | 105340 B | 23348 | 8276 B | 1280 B |
Compilers: x86-64 gcc 13.3.0 · x86-64 clang 18.1.3 · ARM Cortex-M4 gcc 13.2.1 · MicroBlaze BE gcc 14.3.0 · MicroBlaze LE (Bootlin) gcc 14.3.0. benchmark.emb is a fixed fixture (Ok()+CopyFrom over every top-level view); it is pulled forward from head, so only the code generator under test varies between base and head.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a committed, CI-integrated harness for measuring the static code size and
execution speed of Emboss-generated C++
Ok()validation code across theembedded targets the ongoing
Ok()/switch codegen work cares about.scripts/size_bench.py— compiles a fixed benchmark schema (testdata/benchmark.emb)plus the
many_conditionalsOk()highlight across a target × compiler × opt-levelmatrix, reporting per-TU
.text(size) and per-symbolOk()size (nm -S, nested->demangle regex).
--revisions BASE HEADholds the schema fixed (pulled forward fromhead) so only the generator/runtime under test varies;
--speedadds a dynamic run.scripts/size_comment.py— renders the JSON into a sticky Markdown PR comment..github/workflows/code-size.yml— provisions the toolchains, baselines against themerge-base (not the base tip, so codegen that lands on the base after a PR branches
isn't misattributed), and posts/updates one sticky size comment per PR.
scripts/emboss_insncount.c+scripts/build_qemu_plugin.sh— a plugin-enabledqemu-user build + TCG plugin for the deterministic retired-instruction speed metric.
Targets
arm-none-eabi-g++, size) +arm-linux-gnueabihf(speed).microblazebe).AMD/Xilinx Vitis 2025.2 LE v11.0 (
mb-g++), which is proprietary andlocally-licensed, so
have()skips it in public CI; the open-source Bootlinmicroblazeelbuild is the CI LE proxy and provides the LE speed binary.g++/clang++) as a reference.Speed metric
Deterministic guest retired-instruction count under a plugin-enabled qemu-user (both
MicroBlaze endiannesses + ARM), plus host wall-clock on x86. The distro qemu-user has no
plugin support and qemu 10 dropped
contrib/plugins/libinsn.c, so the harness buildsqemu 10.0.11
--enable-pluginsand ships its own instruction-count plugin; when absent,the speed run degrades to wall-clock and the size report is unaffected.
Checkpoint 0 baseline (verified against master)
LargeConditionals::Ok()reader symbol @-Os:Ok()bytesSpeed @
-Os, retired instructions: ARM 9.32B · MB-BE 12.75B · MB-LE 12.17B; x86 hostwall-clock 0.42 s.
Supersedes
Replaces the earlier code-size tooling PRs, folding their coverage into one CI-integrated
harness:
embedded_bench.sh/embedded_compare.sh) — its dual-Ok()-symbol coverage(
LargeConditionals+DisjunctionConditionals) is preserved here.embedded_bench.shrewrite) — superseded bysize_bench.py.profile_tool.pyPR-comment workflow, stacked on Add embedded code-size benchmarking scripts #265) — superseded bysize_comment.py+code-size.yml.New here beyond all three: MicroBlaze LE (Vitis authoritative + Bootlin proxy), the
dynamic execution-speed metric, and four per-instantiation
Ok()driver TUs(reader-only / both / writer-only / residual-heavy) consumed by the follow-on
optimization PRs.