Skip to content

scripts/run_benchmarks.sh BUILD_DIR drifted from Makefile output path #551

@mvillmow

Description

@mvillmow

Context

Surfaced by #550 (forbid-advisory-warnings).

When make benchmark.native removed its ::warning:: suppression, CI failed with:

Error: Build directory not found
Run: mkdir build && cd build && cmake -DCMAKE_BUILD_TYPE=Release -G Ninja .. && ninja
make[1]: *** [Makefile:171: benchmark] Error 1

Root cause

scripts/run_benchmarks.sh defaulted BUILD_DIR to $PROJECT_ROOT/build/release/bin, but the Makefile produces binaries at $(BUILD_DIR)/$(BUILD_SUBDIR) which expands to build/x86.release/ (BUILD_DIR=build, BUILD_SUBDIR=x86, suffixed with .release from CMAKE_BUILD_TYPE).

The script's path never existed. Under the prior if ! make benchmark.native; then echo "::warning::..."; fi invocation, the failure was swallowed and benchmarks were never actually exercised in CI.

Short-term fix (in #550)

Pointed the script's default BUILD_DIR at build/x86.release and let callers override BUILD_DIR or BUILD_SUBDIR.

Follow-up to consider

  1. Have the Makefile benchmark target export BUILD_DIR and BUILD_SUBDIR to the subshell so there is no second source of truth.
  2. Add a CI assertion that at least N benchmark binaries actually ran (the script today silently continues past any missing binary and only fails when ALL are missing — a partial regression could go unnoticed).
  3. Reconcile benchmarks/README.md (which documents ./build/hierarchy_benchmarks) with the actual output path.

Acceptance

  • Makefile and script agree on a single source of truth for the build output path.
  • CI fails if any individual benchmark binary is missing (not just all of them).
  • Docs in benchmarks/README.md reflect the real path.

Linked: #550

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdxDeveloper experience findingtestingTesting finding

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions