exploring integration test schema cleanup

[fivetran-jamie]

What change does this PR introduce?

• adds drop_schemas macro to be run in integration testing
• adds quote macro that i feel like we would use all the time

If this PR introduces a new macro, how did you test the new macro?

on shopify source

If this PR introduces a modification to an existing macro, which packages is the macro currently present in and what steps were taken to test compatibility across packages?

Did you update the README to reflect the macro addition/modifications?

• Yes
• No (provide further explanation) --> still need to

[fivetran-sheringuyen]

Awesome PR! Love the cross team collab here, hoping to see more of this in the future!

Additionally, something that may be interesting to consider is setting expiration dates for tables like we currently do for analytics here.

[fivetran-sheringuyen]

i wonder if you can make this more flexible by adding a fuzzy_schema_match condition?

Specifically for this use case. If a fuzz_schema_match = 'LIKE '%integration_test%' could be a param, this can allow us to leverage the fivetran utils package for this instead of having a custom macro in the integrations package.

[fivetran-sheringuyen]

I guess it doesn't have to be fuzzy lol it could just be something like schema_where_like . but probably depends on the use cases.

[fivetran-jamie]

oooh okay so twoish ideas:

• maybe a schema_patterns argument that by default is [target.schema] so you can drop multiple schema patterns. so in this case you would provide ['integration_tests', 'integrations_tests']. i realize this isn't what you were focusing on above but realized that's also a thing 😅 or you'd have to run the macro twice with different targets
• i opted for only adding the % wildcard as a suffix since that's in line with how dbt does custom schemas (ie appends _schema_suffx to the target). but yeah maybe that's too limiting.. i wonder if we add the schema_patterns variable, should the user have to add the %'s to the patterns? so like in your example, you could get it extra fuzzy and provide schema_patterns=['%integration_test%', '%integrations_test%']
  • however, the default behavior could still align with dbt and be schema_pattern=[target.schema ~ '%']

we could also have the user provide a whole where clause like you mentioned, which would offer the most flexibility but fewer guardrails

cc @fivetran-joemarkiewicz

[fivetran-joemarkiewicz]

Yeah I think for the current use case for this macro the way @fivetran-jamie opted to build it out works great! I could see a future where we provide more flexibility.

However, for the current implementation I would prefer we keep it as is so we don't run into any fuzzy matching issues within our own warehouses. Especially as this will be running constantly as we work on updates across our packages. If we see interest in people wanting to leverage this outside of the current use case, we should definitely consider expanding the flexibility of the macro.

[fivetran-jamie]

cool i'll make a FR to include fuzzy matching after we merge this in!

[fivetran-joemarkiewicz]

@fivetran-jamie I have a few comments and suggestions. Additionally, would you be able to increase the package index and add a log to the CHANGELOG. Let me know if you have any questions.

[fivetran-joemarkiewicz]

@fivetran-jamie thanks for making the requested updates! When doing my final review I had some additional questions and suggestions around the previous changes and the new ones you applied. That being said, the code looks to function without issue! I was able do some detailed testing on my end and feel confident that following this round of updates we should be good to approve and release!

Let me know if you want to sync to discuss in more detail.

[fivetran-joemarkiewicz]

@fivetran-jamie thanks for making these updates! All the changes look good to me!