Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-14976: [Dev][Archery] Fail early if no benchmark found #11865

Closed
wants to merge 2 commits into from

Conversation

cyb70289
Copy link
Contributor

@cyb70289 cyb70289 commented Dec 6, 2021

If no benchmark matches the suite/benchmark filter, current benchmark
tool will build Arrow twice, run no benchmark, and fail with confusing
error message.

This patch is to fail after the first build when the benchmark tool
finds no benchmark to run.

If no benchmark matches the suite/benchmark filter, current benchmark
tool will build Arrow twice, run no benchmark, and fail with confusing
error message.

This patch is to fail after the first build when the benchmark tool
finds no benchmark to run.
@github-actions
Copy link

github-actions bot commented Dec 6, 2021

@github-actions
Copy link

github-actions bot commented Dec 6, 2021

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@kszucs
Copy link
Member

kszucs commented Dec 6, 2021

@cyb70289 can we cover this change with a unittest in test_benchmarks.py?

dev/archery/archery/benchmark/runner.py Outdated Show resolved Hide resolved
@cyb70289
Copy link
Contributor Author

cyb70289 commented Dec 7, 2021

@cyb70289 can we cover this change with a unittest in test_benchmarks.py?

@kszucs, not sure if there's simple way to test it.

Current workflow is to first build all the benchmarks, then scan *-benchmark files and select wanted binaries per suite_filter, then list benchmarks by running the binary xxx-benchmark --benchmark_list_tests and apply benchmark_filter to select wanted benchmarks. Only after that we can judge if the suite/benchmark filter is valid.

@cyb70289 cyb70289 closed this in f0110cf Dec 9, 2021
@cyb70289 cyb70289 deleted the 14976-bench-fail-early branch December 9, 2021 01:51
@ursabot
Copy link

ursabot commented Dec 9, 2021

Benchmark runs are scheduled for baseline = 9d200f5 and contender = f0110cf. f0110cf is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.45% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.49% ⬆️0.04%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

AlenkaF pushed a commit to AlenkaF/arrow that referenced this pull request Dec 9, 2021
If no benchmark matches the suite/benchmark filter, current benchmark
tool will build Arrow twice, run no benchmark, and fail with confusing
error message.

This patch is to fail after the first build when the benchmark tool
finds no benchmark to run.

Closes apache#11865 from cyb70289/14976-bench-fail-early

Authored-by: Yibo Cai <yibo.cai@arm.com>
Signed-off-by: Yibo Cai <yibo.cai@arm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants