-
-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compatibility with Athena Engine v3 #15
Comments
I also assume that the adapter itself is not tightly coupled to Athena engine v2 specifics. The Personally, I don't have experience running on athena engine v3 yet, as I experienced some Athena errors such as |
I agree with you @jessedobbelaere , the adapter shouldn't have any issue on using v3, as it's tight to the workgroup. Let's refer to this to understand if there is work to do. After some testing I noticed those:
The biggest breaking changes should be in the model level, not in the adapters internal. Also, we have this repo https://github.com/dbt-athena/dbt-athena-tester to use as reference to run the same set of models when developing. @Jrmyy and @jessedobbelaere feel free to have a look and add relevant models if necessary to test v2 vs v3 |
@Jrmyy I managed to use v3 with the adapter, I time to time need to apply explicit casting to the timestamp.
I think that to tackle this issue, we could just add a section in the readme on how to solve common cases, to make extra smoother, a sort of enrichment of the athena docs. |
Yes, I think we can tackle this using README.md, since now we will support both engine versions but with different features (CTAS & merge strategies for v3, temp parquet table for v2 + some data types stuff). |
We finally decided to go only for a support of v3 engine concerning Iceberg tables, (i.e. if you use parquet tables, you can still use the v2 engine). (#64)
The consequences are, for Iceberg, you will need :
|
Should we close this since the documentation makes it clearer now what you can and can't do with different athena adapter versions and different table types ? |
Yes please. |
馃憢馃徎 Hello dbt-athena squad
For now the adapter uses the version 2 of Athena Engine, according to README. On 2022.10.13, Athena release the V3 of their Engine, reducing the gap between Athena and Trino features.
I don't know what we want to do about this :
FYI there are the breaking changes
The text was updated successfully, but these errors were encountered: