Update the benchmark to launch one instance per (dataset, synthesizer) pair

[R-Palazzo]

Resolve #605
86b9zz525

• Tested for single-table here, it successfully launched 81 instances (9 synthesizers and 9 datasets).
• Tested for multi-table here, it created 162 instances (54 datasets, 3 synthesizers).
• We need to store the list of datasets and synthesizers to run the benchmarks on. I stored them in a Python file, but I can move them to YAML if we prefer. To include a new synthesizer/dataset, we would just need to update these lists.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.81%. Comparing base (6228ecb) to head (4e89898).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #611      +/-   ##
==========================================
- Coverage   85.91%   85.81%   -0.11%     
==========================================
  Files          40       40              
  Lines        3749     3722      -27     
==========================================
- Hits         3221     3194      -27     
  Misses        528      528              

Flag	Coverage Δ
integration	44.16% <0.00%> (+0.31%)	⬆️
unit	81.59% <100.00%> (-0.14%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[fealho]

This runs the UniformSynthesizer in every instance. So for example, for the 81 single table runs, it ran 81 times. Is this intended?

Also, do we want to change this? I don't think it affects the monthly run, because the config returns before reaching that line, but if you pass datasets/synthesizers, it still creates only 1 instance.

And could you double check the CLI path still works? I think maybe it depended on the yaml files you deleted. Something like this:

python sdgym/_benchmark_launcher/script.py --modality single_table --output-destination s3://bucket/test --synthesizers CTGANSynthesizer

[amontanez24]

This looks good. I just want to confirm something. Does the instance that runs the workflow to launch the jobs download all the datasets for all jobs, or only the one for the job it launches?

I understand we are doing one job = 1 synthesizer and dataset, but I want to make sure the instance that launches the job isn't downloading too much

[R-Palazzo]

This looks good. I just want to confirm something. Does the instance that runs the workflow to launch the jobs download all the datasets for all jobs, or only the one for the job it launches?

I understand we are doing one job = 1 synthesizer and dataset, but I want to make sure the instance that launches the job isn't downloading too much

@amontanez24, yes, that was the case. The instance that runs the workflow was downloading all the datasets. I tried adding the rel-bench dataset and the workflow crashed when trying to download the data:

https://github.com/sdv-dev/SDGym/actions/runs/26508750022/job/78067938428

To avoid this in this PR I moved the dataset loading during execution so it's only the gcp instances that will load the data. I'm running some test and check that it works end-to-end with the rel-bench datasets:

• Github Action
• Results on AWS

[R-Palazzo]

This runs the UniformSynthesizer in every instance. So for example, for the 81 single table runs, it ran 81 times. Is this intended?

Also, do we want to change this? I don't think it affects the monthly run, because the config returns before reaching that line, but if you pass datasets/synthesizers, it still creates only 1 instance.

And could you double check the CLI path still works? I think maybe it depended on the yaml files you deleted. Something like this:

python sdgym/_benchmark_launcher/script.py --modality single_table --output-destination s3://bucket/test --synthesizers CTGANSynthesizer

Hi @fealho thanks for the review!

• Yes, it is intended for the UniformSynthesizer. However, when we aggregate the results to generate the monthly leaderboard, only one UniformSynthesizer per dataset is retained and used to compute Adjusted_Total_Time and Adjusted_Quality_Score. So it’s consistent between all synthesizers and datasets.
• Good catch for the scripty.py, I updated so it's always one instance per job
• Tested the CLI here, running:

python sdgym/_benchmark_launcher/script.py --modality single_table --output-destination s3://sdgym-benchmark/Debug/Issue_605_CLI --synthesizers GaussianCopulaSynthesizer

[R-Palazzo]

I checked it with Kalyan this morning and we want to have all the private demo datasets included in the benchmark

[sarahmish]

I prefer if the list of synthesizers and datasets is moved to a separate file as you suggested to make the processes of adding / removing datasets and synthesizers easy. That could be another issue