Add a script that launches a benchmark from a yaml file or a set of parameters#573
Conversation
R-Palazzo
force-pushed
the
issue-546-benchmark-generator
branch
from
fe78012 to
6913f4f
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## feature/benchmark_launcher #573 +/- ##
==============================================================
+ Coverage 83.30% 83.75% +0.45%
==============================================================
Files 38 39 +1
Lines 3198 3299 +101
==============================================================
+ Hits 2664 2763 +99
- Misses 534 536 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
R-Palazzo
force-pushed
the
issue-546-benchmark-generator
branch
from
e74bae6 to
e2abd66
Compare
sdgym/_benchmark_launcher/script.py
Outdated
| return parser.parse_args() | ||
|
|
||
|
|
||
| def _parse_csv(value): |
There was a problem hiding this comment.
is the input an actual csv file? If not this should be named something else. In general I think with a cli you should specify the input as nargs and then call like this
cli_name --datasets dataset1 dataset2 ...
The parser will automatically turn that into a list
| while len(instance_jobs) < num_instances: | ||
| split_index = None | ||
| split_size = -1 | ||
| for index, instance_job in enumerate(instance_jobs): | ||
| if (_instance_job_size(instance_job) > 1) and ( | ||
| _instance_job_size(instance_job) > split_size | ||
| ): | ||
| split_index = index | ||
| split_size = _instance_job_size(instance_job) | ||
|
|
||
| instance_job = instance_jobs.pop(split_index) | ||
| instance_jobs.extend(_split_instance_jobs(instance_job)) | ||
|
|
There was a problem hiding this comment.
does the current set of jobs you have defined in the config require splitting?
There was a problem hiding this comment.
No it does not require splitting. I clarified it in b73bafc. If datasets, synthesizers, and num_instances are not provided, we default to the configuration defined for the selected modality in the YAML files. I also added some docstrings to provide clearer explanations.
R-Palazzo
force-pushed
the
issue-546-benchmark-generator
branch
2 times, most recently
from
c83d758 to
5589530
Compare
R-Palazzo
force-pushed
the
issue-546-benchmark-generator
branch
from
5589530 to
b73bafc
Compare
| timeout: 345600 | ||
| compute_quality_score: true | ||
| compute_diagnostic_score: true | ||
| compute_privacy_score: false |
There was a problem hiding this comment.
Removed from the base since it's not supported for multi_table yet
4 participants
Resolve #546
CU-86b8h63t2