Extend custom destination

[sh-rp]

Description

This PR introduces two new settings for the custom destination decorator:

• max_nesting_level to control the normalizer
• skip_dlt_columns_and_tables to skip internal columns and tables

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	8bb0e30
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/65fa2338c2655a0008f9d633

[sh-rp]

@rudolfix i need some guidance where to inject / overwrite the max_nesting_level coming from a destination. I realize this place is very likely not the right one, but I am not sure where and how to do it. Should I get the capabilities context in the relationalnormalizer and not persist this setting to the schema at all, or what is the best way?

[rudolfix]

the only thing you need to do is to fix NormalizersConfiguration

def on_resolved(self) -> None:
        # get naming from capabilities if not present
        if self.naming is None:
            if self.destination_capabilities:
                self.naming = self.destination_capabilities.naming_convention

detect the type of the json normalizer and apply the settings to it like the below. you can override existing settings. I think capabilities (if not None) should have precedence over the source settings.

what happens later:
when new schema is created this setting will be used
when schema is loaded - it will not but when we call update_normalize - it will. and we do that in normalizer (in schema.clone). so it should work!

[sh-rp]

since this is set on the nested json normalizer settings i had to change a bit more but not much, I hope it is ok to change the type from mapping to dict in there.

[rudolfix]

this is good! see the trick to apply max nesting to schema

[rudolfix]

good! please add this to our docs: that the default settings are such that data comes to sink without changing identifiers, un-nested and with dlt identifiers removed. and that it is good to push stuff to queues and REST APIs

[rudolfix]

you must use schema._dlt_tables_prefix (which may be normalized) to detect dlt identifiers. you may add such method to schema (but it will be slower to call a method)

[rudolfix]

the only thing you need to do is to fix NormalizersConfiguration

def on_resolved(self) -> None:
        # get naming from capabilities if not present
        if self.naming is None:
            if self.destination_capabilities:
                self.naming = self.destination_capabilities.naming_convention

detect the type of the json normalizer and apply the settings to it like the below. you can override existing settings. I think capabilities (if not None) should have precedence over the source settings.

what happens later:
when new schema is created this setting will be used
when schema is loaded - it will not but when we call update_normalize - it will. and we do that in normalizer (in schema.clone). so it should work!

[rudolfix]

I have a few additions to docs and cross references. Let's merge this first. LGTM!

[rudolfix]

this is the best we can do now. if we have more normalizers with incompatible configs then we'll need to look for something better