-
Notifications
You must be signed in to change notification settings - Fork 33
Support to add with properties when creating Trino/Presto tables #58
Conversation
Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA. In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, don't hesitate to ping @drewbanin. CLA has not been signed by users: @mdesmet |
hello @mdesmet |
I need a comma here, before '\n' to run with our trino/hive configuration. Not sure about others. |
@bachng2017: the comma is indeed necessary. Serves as a good reminder to myself to actually test before committing last minute changes. Following two points:
|
Hi @bachng2017 , I have an iceberg docker setup running and there it works for both the seeds and the models. dbt_project.yaml:
Generating following create table statement on Trino:
|
cool and thanks :) Seems that the maintainers are waiting for your CLA to be signed. Hope this will be merged soon
|
The CLA has been signed after the first commit. @jtcohen6: any other remarks before this PR can be merged? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mdesmet Thanks for the PR, and apologies for the delayed follow-up!
The actual substance of the change here looks fine by me. I have two questions:
- Do you have a strong feeling about naming the config
with_props
, versusproperties
orwith_properties
? I'm trying to find a canonical name for what these things are in Presto/Trino documentation (a laoptions
in BigQuery or Spark), but a really satisfying answer is eluding me. - Did you consider implementing this in the way recommended by Support for table properties #53, and as we've done for other adapters, where each
with
property would be configured + validated independently? The biggest reason to prefer that approach is that it allows users to set/clobber each config independently, so e.g. they could setformat
for the entire project but then independently setanother_property
without clobbering the entirewith_props
dictionary. The biggest reason to prefer the generic dict approach is if there are dozens/hundreds of properties available, and we'd want to support all of them arbitrarily without naming/validating them specifically—as discussed a bit in Support setting table OPTIONS using config dbt-spark#147 (comment).
In any case, you certainly deserve props for the solid PR! :)
|
||
import agate | ||
|
||
|
||
@dataclass | ||
class PrestoConfig(AdapterConfig): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
Hi @jtcohen6, The concept is called table properties. So maybe properties is a better name. I have considered that approach mentioned in #53. However presto/trino is different from athena that it supports way more connectors. For example take the Kudu connector:
Another example is the Hive connector:
Another for clickhouse:
It will be difficult to maintain and follow up with all connectors, hence the freedom of the dict[str, str] approach. |
Thanks for the thorough response @mdesmet! I'm happy with We don't currently have a suite of custom tests for Presto-specific functionality in this plugin. The code change looks straightforward, and most of the logic lives inside Jinja macros, so users have the ability to override/reimplement if they need to. If this is something you've tested locally, and verified that it's working for the cases you're interested in, I'd be ok with merging this for inclusion in v0.21. |
Great, it's tested locally and working. Actually testing it is definitely possible through including a few docker images. That's how i've tested it, see https://github.com/innoverio/modern-data-stack |
Very cool repo! I hadn't seen before, thanks for linking to it. I'm thinking more along the lines of custom dbt integration tests to automate in CI, such as these ones for dbt-spark. That leverages a testing framework we've played around with, but haven't all the way locked down. If we find we need automated testing for more complex |
I'll backport onto |
hello @mdesmet . In the implement macro, then we mix those 2 properties together before create the final one. This will help to solve the issue of common/individual setting while make sure users could use various formats for different connectors. For example we will have a ``transactional: true |
@bachng2017 : Can you create a new issue for further discussion? I have following thoughts:
|
Fixes #53
By adding with_props in the model, we can enable users to add properties when creating tables. As presto/Trino supports a lot of different adapters, for now a Dict[String, String] type enables flexibility.