feat(dbt sync): Merge metadata and preserve Preset configs

[Vitor-Avila]

By default, when triggering a dbt sync the CLI:

1. Delete all existing metrics from the dataset.
2. Create metrics based on the dbt definitions.

While this is the expected scenario for the integration (to preserve dbt as the source of truth), this approach can be impactful for organizations that are already extensively using Preset/Superset and therefore have custom metrics defined in their datasets that don't exist in dbt yet. For these cases, the sync could result in broken charts/dashboards, since metrics that only exist in Preset don't get re-created.

This PR introduces two modifications:

• Changing the --preserve-columns flag to --preserve-metadata: The --preserve-columns was used to prevent a column sync during the update, to properly preserve column configs (Is temporal, Is dimension, etc). This flag is now re-named to --preserve-metadata and also:
  • Preserve Preset-only metrics in the dataset (they are no longer deleted).
  • For metrics that exist both in Preset and dbt, Preset metadata is maintained (label, metric syntax, etc).
• Creating a new flag --merge-metadata: This flag can be used to:
  • Preserve Preset-only metrics in the dataset (they are no longer deleted).
  • For metrics that exist both in Preset and dbt, dbt metadata is applied (label, metric syntax, etc).

The validation if the metric already exists in Preset happens comparing the metric name from dbt with the metric_name from Preset.

[betodealmeida]

[betodealmeida]

Suggested change

	for key in ("changed_on", "created_on", "type_generic"):
	if key in metadata:
	del metadata[key]
	return metadata
	return {
	k: v for k, v in metadata.items()
	if k not in {"changed_on", "created_on", "type_generic"}
	}

[betodealmeida]

(You might also want to make this function more generic by passing the keys that should be filtered out as an argument.)

[Vitor-Avila]

that's a great idea! I think for now the scope might be limited, but I'll check if this is used in other parts of the code (not necessarily related with the dbt sync) and I can improve this in a future PR

[Vitor-Avila]

Unsure why, but the suggested change breaks our current tests. It might require updating the tests as well. I'll try to cover this too while simplifying sync_datasets.