Skip to content

[CI] Remove dependency on bot user for uploading benchmark CI results #20333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Nov 3, 2025
Merged

[CI] Remove dependency on bot user for uploading benchmark CI results #20333

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
df8990c
test pushing to branch
ianayl Oct 3, 2025
7ce4420
give write permissions
ianayl Oct 8, 2025
866b64e
amend perms
ianayl Oct 8, 2025
b518e18
Revert "amend perms"
ianayl Oct 8, 2025
2109b73
Revert "give write permissions"
ianayl Oct 8, 2025
6e73cf7
test perms
ianayl Oct 8, 2025
e20975f
typo
ianayl Oct 8, 2025
3bea805
more perms
ianayl Oct 8, 2025
418bd1d
test fail
ianayl Oct 9, 2025
1539867
set to correct repo
ianayl Oct 9, 2025
8e1f1b8
switch branch
ianayl Oct 9, 2025
66c4b5c
try removing manually passing secrets
ianayl Oct 10, 2025
baf64ef
fix typo
ianayl Oct 14, 2025
ac56a43
make write permissions job-level
ianayl Oct 14, 2025
de538b6
Merge branch 'sycl' of https://github.com/intel/llvm into ianayl/benc…
ianayl Oct 30, 2025
a589c14
fix perms
ianayl Oct 30, 2025
3ffa758
fix perms 2
ianayl Oct 31, 2025
9b92cfd
bump
ianayl Oct 31, 2025
25976a1
shuffle perms for sycl-linux-run-tests
ianayl Oct 31, 2025
0669771
permisisons changed, need to update
ianayl Oct 31, 2025
983fa54
this is so stupid
ianayl Oct 31, 2025
fad9bae
fix const
ianayl Oct 31, 2025
0c4e7e2
fix perms
ianayl Oct 31, 2025
3b260f6
Merge branch 'sycl' of https://github.com/intel/llvm into ianayl/benc…
ianayl Oct 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/sycl-docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ jobs:
touch .nojekyll
# Update benchmarking dashboard configuration
cat << EOF > benchmarks/config.js
remoteDataUrl = 'https://raw.githubusercontent.com/intel/llvm-ci-perf-results/refs/heads/unify-ci/';
remoteDataUrl = 'https://raw.githubusercontent.com/intel/llvm/refs/heads/sycl-benchmark-ci-results/';
Copy link
Contributor

@lukaszstolarczuk lukaszstolarczuk Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my note on this change - while it seems convenient, I can see on my PC, that repo intel/llvm-ci-perf-results is ~600MB - this will be extra MBs added to intel/llvm repo, which is already quite big - just to consider

Copy link
Contributor

@uditagarwal97 uditagarwal97 Oct 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ianayl would it be possible to compress the benchmarking data before pushing it to intel/llvm ?

Copy link
Contributor Author

@ianayl ianayl Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory yes, but only the archived data. I think we'll need to have a conversation on this: it might even be reasonable to delete horribly outdated data.

We could also try moving archived data to intel/llvm-ci-perf-results, but alas that'll mean we need to keep an updated bot user token again for the repository.

Do github repositories have a max storage quota?

Copy link
Contributor

@uditagarwal97 uditagarwal97 Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do github repositories have a max storage quota?

https://docs.github.com/en/repositories/creating-and-managing-repositories/repository-limits#repository-size

we'll need to have a conversation on this

I think we should have that conversation before merging this PR. Currently, the size of intel/llvm-ci-perf-results is ~600MB, but, if unchecked the size will increase further.

Copy link
Contributor

@lslusarczyk lslusarczyk Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory yes, but only the archived data. I think we'll need to have a conversation on this: it might even be reasonable to delete horribly outdated data.

Compressing results files gives ~190MB out of 320M, so I think it is better to just remove outdated data. We have data starting at March'25.

How many months of data do wee need to keep. 6? In this case we will have data staring from 15th April today. The rest will be in git history, hopefully well compressed.

Copy link
Contributor

@lukaszstolarczuk lukaszstolarczuk Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also try moving archived data to intel/llvm-ci-perf-results, but alas that'll mean we need to keep an updated bot user token again for the repository.

If we go this way, we can archive e.g. once every second week, or something, and make a PR instead of pushing

Copy link
Contributor Author

@ianayl ianayl Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, today is the last day of my internship, I will not have time to implement this.

I will aim to get this PR to be merge-ready today, but someone else will need to make a new PR that deletes old data

defaultCompareNames = ["Baseline_PVC_L0"];
EOF
# Upload the generated docs as an artifact and deploy to GitHub Pages.
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/sycl-linux-precommit-aws.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ jobs:
e2e-cuda:
needs: [aws-start]
uses: ./.github/workflows/sycl-linux-run-tests.yml
permissions:
contents: write
packages: read
with:
name: CUDA E2E
runner: '["aws_cuda-${{ github.event.workflow_run.id }}-${{ github.event.workflow_run.run_attempt }}"]'
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/sycl-linux-precommit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,9 @@ jobs:
E2E:
needs: [build, detect_changes, compat_read_exclude]
if: ${{ always() && !cancelled() && needs.build.outputs.build_conclusion == 'success' }}
permissions:
contents: write
packages: read
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -231,6 +234,9 @@ jobs:

test-perf:
needs: [build, detect_changes]
permissions:
contents: write
packages: read
if: |
always() && !cancelled()
&& needs.build.outputs.build_conclusion == 'success'
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/sycl-linux-run-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,9 @@ jobs:
if: github.event_name == 'workflow_dispatch' || inputs.skip_run == 'false'
name: ${{ inputs.name }}
runs-on: ${{ fromJSON(inputs.runner) }}
permissions:
contents: write
packages: read
container:
image: ${{ inputs.image || 'ghcr.io/intel/llvm/ubuntu2404_intel_drivers:alldeps'}}
options: ${{ inputs.image_options }}
Expand Down Expand Up @@ -376,4 +379,3 @@ jobs:
build_ref: ${{ inputs.repo_ref }}
env:
RUNNER_TAG: ${{ inputs.runner }}
GITHUB_TOKEN: ${{ secrets.LLVM_SYCL_BENCHMARK_TOKEN }}
15 changes: 15 additions & 0 deletions .github/workflows/sycl-nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,9 @@ jobs:

ubuntu2204_test:
needs: [ubuntu2204_build]
permissions:
contents: write
packages: read
if: ${{ always() && !cancelled() && needs.ubuntu2204_build.outputs.build_conclusion == 'success' }}
strategy:
fail-fast: false
Expand Down Expand Up @@ -169,6 +172,9 @@ jobs:

ubuntu2404_oneapi_test:
needs: [ubuntu2404_oneapi_build]
permissions:
contents: write
packages: read
if: ${{ always() && !cancelled() && needs.ubuntu2404_oneapi_build.outputs.build_conclusion == 'success' }}
uses: ./.github/workflows/sycl-linux-run-tests.yml
with:
Expand Down Expand Up @@ -230,6 +236,9 @@ jobs:

cuda-run-tests:
needs: [ubuntu2204_build, cuda-aws-start]
permissions:
contents: write
packages: read
if: ${{ always() && !cancelled() && needs.ubuntu2204_build.outputs.build_conclusion == 'success' }}
uses: ./.github/workflows/sycl-linux-run-tests.yml
with:
Expand All @@ -254,6 +263,9 @@ jobs:

build-sycl-cts-linux:
needs: ubuntu2204_build
permissions:
contents: write
packages: read
if: ${{ always() && !cancelled() && needs.ubuntu2204_build.outputs.build_conclusion == 'success' }}
uses: ./.github/workflows/sycl-linux-run-tests.yml
with:
Expand All @@ -270,6 +282,9 @@ jobs:

run-sycl-cts-linux:
needs: [ubuntu2204_build, build-sycl-cts-linux]
permissions:
contents: write
packages: read
if: ${{ always() && !cancelled() && needs.ubuntu2204_build.outputs.build_conclusion == 'success' }}
strategy:
fail-fast: false
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/sycl-post-commit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ jobs:
e2e-lin:
needs: [detect_changes, build-lin]
if: ${{ always() && !cancelled() && needs.build-lin.outputs.build_conclusion == 'success' }}
permissions:
contents: write
packages: read
strategy:
fail-fast: false
matrix:
Expand Down
9 changes: 9 additions & 0 deletions .github/workflows/sycl-ur-perf-benchmarking.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,9 @@ jobs:
benchmark_dispatch:
name: '[Dispatch] Benchmarks'
needs: [ build_sycl_dispatch, sanitize_inputs_dispatch ]
permissions:
contents: write
packages: read
if: always() && !cancelled() && needs.build_sycl_dispatch.outputs.build_conclusion == 'success'
strategy:
matrix:
Expand Down Expand Up @@ -193,6 +196,9 @@ jobs:
benchmark_nightly:
name: '[Nightly] Benchmarks'
needs: [build_nightly]
permissions:
contents: write
packages: read
if: always() && !cancelled() && needs.build_nightly.outputs.build_conclusion == 'success'
strategy:
fail-fast: false
Expand Down Expand Up @@ -241,6 +247,9 @@ jobs:
test_benchmark_framework:
name: '[PR] Benchmark suite testing'
needs: [build_pr]
permissions:
contents: write
packages: read
if: always() && !cancelled() && needs.build_pr.outputs.build_conclusion == 'success'
uses: ./.github/workflows/sycl-linux-run-tests.yml
with:
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/sycl-weekly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ jobs:
# This job builds SYCL-CTS with -fsycl-use-spirv-backend-for-spirv-gen.
build-sycl-cts:
needs: ubuntu2204_build
permissions:
contents: write
packages: read
if: ${{ always() && !cancelled() && needs.ubuntu2204_build.outputs.build_conclusion == 'success' }}
uses: ./.github/workflows/sycl-linux-run-tests.yml
with:
Expand All @@ -39,6 +42,9 @@ jobs:

run-sycl-cts:
needs: [ubuntu2204_build, build-sycl-cts]
permissions:
contents: write
packages: read
if: ${{ always() && !cancelled() && needs.ubuntu2204_build.outputs.build_conclusion == 'success' }}
strategy:
fail-fast: false
Expand Down
22 changes: 12 additions & 10 deletions devops/actions/run-tests/benchmark/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ inputs:
type: string
required: False

env:
BENCHMARK_RESULTS_BRANCH: "sycl-benchmark-ci-results"

runs:
# composite actions don't make use of 'name', so copy-paste names as a comment in the first line of each step
using: "composite"
Expand Down Expand Up @@ -141,10 +144,10 @@ runs:

cd -
- name: Checkout results repo
shell: bash
run: |
# Checkout results repo
git clone -b unify-ci https://github.com/intel/llvm-ci-perf-results
uses: actions/checkout@v5
with:
ref: ${{ env.BENCHMARK_RESULTS_BRANCH }}
path: llvm-ci-perf-results
- name: Build and run benchmarks
env:
# Need to append "_<device>_<backend>" to save name in order to follow
Expand Down Expand Up @@ -252,9 +255,8 @@ runs:
run: |
# Push benchmarks results
cd "./llvm-ci-perf-results"
git config user.name "SYCL Benchmarking Bot"
git config user.email "sys_sycl_benchmarks@intel.com"
results_branch="unify-ci"
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"

if git diff --quiet && git diff --cached --quiet; then
echo "No new results added, skipping push."
Expand All @@ -267,7 +269,7 @@ runs:
git commit -m "[GHA] Upload compute-benchmarks results from https://github.com/intel/llvm/actions/runs/${{ github.run_id }}"
results_file="$(git diff HEAD~1 --name-only -- results/ | head -n 1)"

if git push "https://$GITHUB_TOKEN@github.com/intel/llvm-ci-perf-results.git" "$results_branch"; then
if git push; then
echo "Push succeeded"
break
fi
Expand All @@ -277,8 +279,8 @@ runs:
cached_result="$(mktemp -d)/$(basename $results_file)"
mv "$results_file" "$cached_result"

git reset --hard "origin/$results_branch"
git pull origin "$results_branch"
git reset --hard "origin/$BENCHMARK_RESULTS_BRANCH"
git pull

mv "$cached_result" "$results_file"
fi
Expand Down
Loading