[CI][Bench] Refactor bench workflow #20742

lukaszstolarczuk · 2025-11-24T13:44:51Z

No description provided.

devops/actions/run-tests/benchmark/action.yml

sarnex · 2025-12-04T20:36:00Z

devops/actions/run-tests/benchmark/action.yml


      echo "::endgroup::"
      echo "::group::compare_results"
+
      python3 ./devops/scripts/benchmarks/compare.py to_hist \
        --avg-type EWMA \
        --cutoff "$(date -u -d '7 days ago' +'%Y%m%d_%H%M%S')" \
        --name "$SAVE_NAME" \
-        --compare-file "./llvm-ci-perf-results/results/${SAVE_NAME}_${SAVE_TIMESTAMP}.json" \
-        --results-dir "./llvm-ci-perf-results/results/" \
+        --compare-file "${{ steps.results_repo.outputs.BENCHMARK_RESULTS_REPO_PATH }}/results/${SAVE_NAME}_${SAVE_TIMESTAMP}.json" \
+        --results-dir "${{ steps.results_repo.outputs.BENCHMARK_RESULTS_REPO_PATH }}/results/" \
        --regression-filter '^[a-z_]+_sycl .* CPU count' \
        --regression-filter-type 'SYCL benchmark (measured using CPU cycle count)' \
        --verbose \
        --produce-github-summary \
        ${{ inputs.dry_run == 'true' && '--dry-run' || '' }} \

      echo "::endgroup::"
-
+  - name: Run benchmarks integration tests
+    shell: bash
+    if: ${{ github.event_name == 'pull_request' }}


@lukaszstolarczuk Do we need to do something about this (github security issue)

I think I fixed it, but I'm not sure if zizmor correctly showed me the issue.

Anyway, I will take another look at the zizmor alerts later on (I can see a few other alerts in the benchmark/action.yml file which are not displayed here - will take care of them in another PR).

devops/actions/run-tests/benchmark/action.yml

lukaszstolarczuk · 2025-11-28T12:34:50Z

dispatch run: https://github.com/intel/llvm/actions/runs/19759131571/job/56617416122#step:19:1007
and results saved into dashboard: https://intel.github.io/llvm/benchmarks/?runs=test_refactor_BMG_L0v2

lukaszstolarczuk · 2025-12-02T09:38:56Z

@sarnex , can you please re-review?

sarnex · 2025-12-04T20:33:10Z

Sorry I don't know how I missed so many pings

sarnex

no major flags besides this one thing, in depth review left to benchmark team

sarnex · 2025-12-04T20:36:00Z

devops/actions/run-tests/benchmark/action.yml


      echo "::endgroup::"
      echo "::group::compare_results"
+
      python3 ./devops/scripts/benchmarks/compare.py to_hist \
        --avg-type EWMA \
        --cutoff "$(date -u -d '7 days ago' +'%Y%m%d_%H%M%S')" \
        --name "$SAVE_NAME" \
-        --compare-file "./llvm-ci-perf-results/results/${SAVE_NAME}_${SAVE_TIMESTAMP}.json" \
-        --results-dir "./llvm-ci-perf-results/results/" \
+        --compare-file "${{ steps.results_repo.outputs.BENCHMARK_RESULTS_REPO_PATH }}/results/${SAVE_NAME}_${SAVE_TIMESTAMP}.json" \
+        --results-dir "${{ steps.results_repo.outputs.BENCHMARK_RESULTS_REPO_PATH }}/results/" \
        --regression-filter '^[a-z_]+_sycl .* CPU count' \
        --regression-filter-type 'SYCL benchmark (measured using CPU cycle count)' \
        --verbose \
        --produce-github-summary \
        ${{ inputs.dry_run == 'true' && '--dry-run' || '' }} \

      echo "::endgroup::"
-
+  - name: Run benchmarks integration tests
+    shell: bash
+    if: ${{ github.event_name == 'pull_request' }}


@lukaszstolarczuk Do we need to do something about this (github security issue)

lukaszstolarczuk · 2025-12-11T16:39:19Z

I started another dispatch to be sure I didn't break anything: https://github.com/intel/llvm/actions/runs/20140113209

PatKamin · 2025-12-12T08:08:35Z

devops/actions/run-tests/benchmark/action.yml

+      export SAVE_SUFFIX=$SAVE_SUFFIX
+      echo "SAVE_SUFFIX=$SAVE_SUFFIX" >> $GITHUB_ENV
+      export MACHINE_TYPE=$MACHINE_TYPE
+      echo "MACHINE_TYPE=$MACHINE_TYPE" >> $GITHUB_ENV


nit: GPU_TYPE would be more accurate here

PatKamin · 2025-12-12T08:18:22Z

devops/actions/run-tests/benchmark/action.yml

-      fi
-  - name: Cache changes and upload github summary
+      export LLVM_BENCHMARKS_UNIT_TESTING=1
+      export COMPUTE_BENCHMARKS_BUILD_PATH=$BENCH_WORKDIR/compute-benchmarks-build


Why not putting these two in env section?

lukaszstolarczuk · 2025-12-12T10:07:28Z

@intel/llvm-gatekeepers, changes are only in benchmarks' workflow - issues in tests are not related. Please merge

devops/actions/run-tests/benchmark/action.yml

kswiecicki · 2025-12-12T12:02:28Z

Failing e2e tests are unrelated, this patch contains only workflow changes. I'll merge it.

This reverts commit 666693d.

lukaszstolarczuk had a problem deploying to WindowsCILock November 24, 2025 13:44 — with GitHub Actions Failure

lukaszstolarczuk temporarily deployed to WindowsCILock November 24, 2025 14:16 — with GitHub Actions Inactive

lukaszstolarczuk force-pushed the workflow-bench branch from 777ea28 to 144bedd Compare November 24, 2025 15:18

lukaszstolarczuk had a problem deploying to WindowsCILock November 24, 2025 15:18 — with GitHub Actions Error

github-advanced-security bot found potential problems Nov 24, 2025

View reviewed changes

devops/actions/run-tests/benchmark/action.yml Fixed Show fixed Hide fixed

devops/actions/run-tests/benchmark/action.yml Fixed Show fixed Hide fixed

lukaszstolarczuk force-pushed the workflow-bench branch from 144bedd to 107e1b3 Compare November 24, 2025 15:33

lukaszstolarczuk temporarily deployed to WindowsCILock November 24, 2025 15:33 — with GitHub Actions Inactive

github-advanced-security bot found potential problems Nov 24, 2025

View reviewed changes

devops/actions/run-tests/benchmark/action.yml Fixed Show fixed Hide fixed

lukaszstolarczuk temporarily deployed to WindowsCILock November 24, 2025 16:08 — with GitHub Actions Inactive

lukaszstolarczuk had a problem deploying to WindowsCILock November 24, 2025 19:07 — with GitHub Actions Error

github-advanced-security bot found potential problems Nov 24, 2025

View reviewed changes

devops/actions/run-tests/benchmark/action.yml Fixed Show fixed Hide fixed

lukaszstolarczuk force-pushed the workflow-bench branch from 582bf20 to 2d0ae0b Compare November 24, 2025 19:26

lukaszstolarczuk temporarily deployed to WindowsCILock November 24, 2025 19:27 — with GitHub Actions Inactive

lukaszstolarczuk had a problem deploying to WindowsCILock November 24, 2025 19:27 — with GitHub Actions Error

lukaszstolarczuk temporarily deployed to WindowsCILock November 24, 2025 19:46 — with GitHub Actions Inactive

lukaszstolarczuk had a problem deploying to WindowsCILock November 24, 2025 19:46 — with GitHub Actions Error

lukaszstolarczuk force-pushed the workflow-bench branch 3 times, most recently from 3c623ab to e43b8c7 Compare November 25, 2025 11:03

github-advanced-security bot found potential problems Nov 25, 2025

View reviewed changes

lukaszstolarczuk force-pushed the workflow-bench branch from e43b8c7 to 82b2f04 Compare November 25, 2025 11:19

lukaszstolarczuk temporarily deployed to WindowsCILock November 25, 2025 11:20 — with GitHub Actions Inactive

lukaszstolarczuk temporarily deployed to WindowsCILock November 25, 2025 12:04 — with GitHub Actions Inactive

lukaszstolarczuk had a problem deploying to WindowsCILock November 25, 2025 14:20 — with GitHub Actions Error

lukaszstolarczuk force-pushed the workflow-bench branch from 8eb3fe7 to 02b8752 Compare November 25, 2025 14:21

lukaszstolarczuk changed the title ~~WIP: [CI][Bench] Refactor bench workflow~~ [CI][Bench] Refactor bench workflow Nov 25, 2025

lukaszstolarczuk temporarily deployed to WindowsCILock November 25, 2025 14:22 — with GitHub Actions Inactive

lukaszstolarczuk temporarily deployed to WindowsCILock November 25, 2025 14:56 — with GitHub Actions Inactive

lukaszstolarczuk force-pushed the workflow-bench branch from 02b8752 to 50b91ca Compare November 25, 2025 17:11

lukaszstolarczuk requested a review from sarnex November 28, 2025 12:26

sarnex approved these changes Dec 4, 2025

View reviewed changes

lukaszstolarczuk force-pushed the workflow-bench branch from 8f57263 to 02ae04b Compare December 11, 2025 16:26

lukaszstolarczuk temporarily deployed to WindowsCILock December 11, 2025 16:26 — with GitHub Actions Inactive

lukaszstolarczuk had a problem deploying to WindowsCILock December 11, 2025 16:53 — with GitHub Actions Failure

PatKamin reviewed Dec 12, 2025

View reviewed changes

lukaszstolarczuk added 6 commits December 12, 2025 10:04

[CI][Bench] Refactor bench workflow

80decee

[CI][Bench] Move bench integrations tests (on PR) to a sep. step

baefd92

[CI][Bench] Use output to store sensitive info instead of env

ad29f0c

fix zizmor issues

34b491e

fixes

4c961ee

Fix zizmor issue - workdir

057f684

lukaszstolarczuk force-pushed the workflow-bench branch from 02ae04b to 0077156 Compare December 12, 2025 09:04

lukaszstolarczuk temporarily deployed to WindowsCILock December 12, 2025 09:04 — with GitHub Actions Inactive

lukaszstolarczuk temporarily deployed to WindowsCILock December 12, 2025 09:33 — with GitHub Actions Inactive

PatKamin approved these changes Dec 12, 2025

View reviewed changes

kswiecicki approved these changes Dec 12, 2025

View reviewed changes

devops/actions/run-tests/benchmark/action.yml Show resolved Hide resolved

fixes for review

fc3931b

lukaszstolarczuk force-pushed the workflow-bench branch from 0077156 to fc3931b Compare December 12, 2025 10:33

lukaszstolarczuk temporarily deployed to WindowsCILock December 12, 2025 10:33 — with GitHub Actions Inactive

lukaszstolarczuk temporarily deployed to WindowsCILock December 12, 2025 10:59 — with GitHub Actions Inactive

kswiecicki merged commit 666693d into intel:sycl Dec 12, 2025
39 of 41 checks passed

lukaszstolarczuk deleted the workflow-bench branch December 12, 2025 12:02

PatKamin added a commit to PatKamin/llvm that referenced this pull request Dec 16, 2025

Revert "[CI][Bench] Refactor bench workflow (intel#20742)"

82994de

This reverts commit 666693d.

[CI][Bench] Refactor bench workflow #20742

[CI][Bench] Refactor bench workflow #20742

Uh oh!

Conversation

lukaszstolarczuk commented Nov 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check failure

sarnex Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukaszstolarczuk Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lukaszstolarczuk commented Nov 28, 2025

Uh oh!

lukaszstolarczuk commented Dec 2, 2025

Uh oh!

sarnex commented Dec 4, 2025

Uh oh!

sarnex left a comment

Choose a reason for hiding this comment

Uh oh!

sarnex Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukaszstolarczuk commented Dec 11, 2025

Uh oh!

PatKamin Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

lukaszstolarczuk Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

PatKamin Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

lukaszstolarczuk Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

lukaszstolarczuk commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kswiecicki commented Dec 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sarnex Dec 4, 2025 •

edited

Loading

sarnex Dec 4, 2025 •

edited

Loading

lukaszstolarczuk commented Dec 12, 2025 •

edited

Loading