Skip to content

Walltime bound #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 13, 2025
Merged

Walltime bound #31

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions problems/amd/eval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,7 @@ def _run_single_benchmark(test: TestCase, recheck: bool, max_repeats: int, max_t
# otherwise, we repeat until we either measure at least 10 full seconds,
# or the relative error of the mean is below 1%.

bm_start_time = time.perf_counter_ns()
for i in range(max_repeats):
if recheck:
# ensure we use a different seed for every benchmark
Expand All @@ -239,11 +240,16 @@ def _run_single_benchmark(test: TestCase, recheck: bool, max_repeats: int, max_t
return message

del output
durations.append(end-start)
durations.append(end - start)

if i > 1:
total_bm_duration = bm_start_time - time.perf_counter_ns()
stats = calculate_stats(durations)
if stats.err / stats.mean < 0.001 or stats.mean * stats.runs > max_time_ns:
# stop if either
# a) relative error dips below 0.1%
# b) we exceed the total time limit for benchmarking the kernel
# c) we exceed 2 minutes of total wallclock time.
if stats.err / stats.mean < 0.001 or stats.mean * stats.runs > max_time_ns or total_bm_duration > 120e9:
break

return calculate_stats(durations)
Expand Down