Add working multi table benchmark #504

pvk-developer · 2025-11-24T17:26:45Z

Resolves #486
CU-86b7cjbz8

sdv-team · 2025-11-24T17:26:49Z

Task linked: CU-86b7cjbz8 SDGym - Add benchmark_multi_table function #486

codecov · 2025-11-24T17:37:07Z

Codecov Report

❌ Patch coverage is 98.79518% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 77.34%. Comparing base (f5abf35) to head (23234c5).

Files with missing lines	Patch %	Lines
sdgym/_dataset_utils.py	95.45%	1 Missing ⚠️

Additional details and impacted files

@@                           Coverage Diff                            @@
##           feature_branch/mutli_table_benchmark     #504      +/-   ##
========================================================================
+ Coverage                                 76.39%   77.34%   +0.94%     
========================================================================
  Files                                        30       30              
  Lines                                      2411     2472      +61     
========================================================================
+ Hits                                       1842     1912      +70     
+ Misses                                      569      560       -9

Flag	Coverage Δ
integration	`63.40% <95.18%> (+1.09%)`	⬆️
unit	`72.04% <85.54%> (+0.58%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

amontanez24

Looking great!

amontanez24 · 2025-11-26T18:33:23Z

sdgym/benchmark.py

+
    for synthesizer in synthesizers:
-        if synthesizer not in SDV_SINGLE_TABLE_SYNTHESIZERS:
+        if synthesizer not in SDV_SYNTHESIZERS:


what happens if someone passes in HMA to the single table benchmark function? Should we check here that the modality matches?

Will add validation now that I have the flags in, I forgot to update the logic after rebasing it. Nice catch!

amontanez24 · 2025-11-26T19:50:33Z

sdgym/_dataset_utils.py

+        all_columns = list(table.columns)
+        mandatory = list(metadata._get_all_keys(table_name))
+        optional_columns = [column for column in all_columns if column not in mandatory]
+        if len(mandatory) >= 10:
+            keep_columns = mandatory
+        else:
+            extra = 10 - len(mandatory)
+            keep_columns = mandatory + optional_columns[:extra]
+
+        # Filter only the subset of columns that will be used
+        subset_column_schema = {
+            column_name: table.columns[column_name]
+            for column_name in keep_columns
+            if column_name in table.columns
+        }
+        # Replace the columns with only the subset ones
+        metadata_dict['tables'][table_name]['columns'] = subset_column_schema


Is there a way to consolidate this chunk with the single table version of it below? Should be a very similar process

SDGym/sdgym/_dataset_utils.py

Lines 54 to 76 in f5abf35

max_rows, max_columns = (1000, 10)

tables = metadata_dict.get('tables', {})

mandatory_columns = []

table_name, table_info = next(iter(tables.items()))

columns = table_info.get('columns', {})

keep_columns = list(columns)

if modality == 'sequential':

seq_index = table_info.get('sequence_index')

seq_key = table_info.get('sequence_key')

mandatory_columns = [col for col in (seq_index, seq_key) if col]

optional_columns = [col for col in columns if col not in mandatory_columns]

# If we have too many columns, drop extras but never mandatory ones

if len(columns) > max_columns:

keep_count = max_columns - len(mandatory_columns)

keep_columns = mandatory_columns + optional_columns[:keep_count]

table_info['columns'] = {

column_name: column_definition

for column_name, column_definition in columns.items()

if column_name in keep_columns

}

Yeah let me split it up, at first I wanted to keep both separated clearly but it makes sense that each table of the multi table goes over the same process as the single-table. Will update now.

amontanez24 · 2025-11-26T19:55:55Z

sdgym/benchmark.py

+
    if len(metrics) > 0:
-        metrics, metric_kwargs = get_metrics(metrics, modality='single-table')
+        metrics, metric_kwargs = get_metrics(metrics, modality=modality.replace('_', '-'))


should we update get_metrics to just use the same modality formatting?

I updated it then undid the changes because I did not want to change much the error messaging:

raise ValueError(f'Unknown {modality} metric: {metric}') from None

But I can surely update that if it is fine. (Basically didn't want to touch much the external portion of benchmark, just focus on the required to make this work).

amontanez24 · 2025-11-26T19:58:58Z

sdgym/benchmark.py

+    uniform_not_included = bool(
+        MultiTableUniformSynthesizer not in synthesizers
+        and MultiTableUniformSynthesizer.__name__ not in synthesizers
+    )
+    if uniform_not_included:
+        LOGGER.info('Adding MultiTableUniformSynthesizer to the list of synthesizers.')
+        synthesizers.append('MultiTableUniformSynthesizer')


this and the single table one can probably be consolidated into one function that takes in modality. Also we can call it ensure_fallback_is_included and make the fallback synthesizer a parameter

amontanez24 · 2025-11-26T20:01:30Z

tests/integration/test_benchmark.py

    pd.testing.assert_frame_equal(result_2, saved_result_2, check_dtype=False)
+
+
+def test_benchmark_multi_table_basic_synthesizers():


nice testing! Can we add a test that forces the fallback to be tested? We can either make a multitable synthesizer that always errors or mock a function in HMA. Whichever is easiest?

amontanez24 · 2025-11-26T20:05:20Z

tests/unit/test_benchmark.py

+    empty_scores = pd.DataFrame()
+    mock__generate_job_args_list.return_value = []
+    mock__get_empty_dataframe.return_value = empty_scores


if you are passing in no synthesizers or datasets, why do you also need to mock the jobs returned? Shouldn't that list be empty?

pvk-developer added 6 commits November 26, 2025 18:10

WIP: Add working multi table benchmark

1b5ef28

Update integration and unit tests

dd7d89f

Temporary fix: Fix test that will be updated by the new result explorer

3e74ebf

Add unit tests for _dataset_utils.py

e5fa2c7

Add unit tests

d986dfd

Add integration test for benchmark multi table end to end

c51dc98

pvk-developer force-pushed the issue-486-add-benchmark_multi_table_function branch from df5e11c to c51dc98 Compare November 26, 2025 17:10

pvk-developer changed the base branch from main to feature_branch/mutli_table_benchmark November 26, 2025 17:11

pvk-developer changed the title ~~WIP: Add working multi table benchmark~~ Add working multi table benchmark Nov 26, 2025

pvk-developer requested review from R-Palazzo, amontanez24 and fealho November 26, 2025 17:11

Add MultiTableUniformSynthesizer and fix scale call

0d0632c

pvk-developer marked this pull request as ready for review November 26, 2025 17:52

pvk-developer requested a review from a team as a code owner November 26, 2025 17:52

Update integration test

23234c5

amontanez24 reviewed Nov 26, 2025

View reviewed changes

	max_rows, max_columns = (1000, 10)
	tables = metadata_dict.get('tables', {})
	mandatory_columns = []
	table_name, table_info = next(iter(tables.items()))

	columns = table_info.get('columns', {})
	keep_columns = list(columns)
	if modality == 'sequential':
	seq_index = table_info.get('sequence_index')
	seq_key = table_info.get('sequence_key')
	mandatory_columns = [col for col in (seq_index, seq_key) if col]

	optional_columns = [col for col in columns if col not in mandatory_columns]

	# If we have too many columns, drop extras but never mandatory ones
	if len(columns) > max_columns:
	keep_count = max_columns - len(mandatory_columns)
	keep_columns = mandatory_columns + optional_columns[:keep_count]
	table_info['columns'] = {
	column_name: column_definition
	for column_name, column_definition in columns.items()
	if column_name in keep_columns
	}

		pd.testing.assert_frame_equal(result_2, saved_result_2, check_dtype=False)


		def test_benchmark_multi_table_basic_synthesizers():

Add working multi table benchmark #504

Are you sure you want to change the base?

Add working multi table benchmark #504

Uh oh!

Conversation

pvk-developer commented Nov 24, 2025

Uh oh!

sdv-team commented Nov 24, 2025

Uh oh!

codecov bot commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

amontanez24 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Nov 24, 2025 •

edited

Loading