Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Athena Spark #144

Open
sanga8 opened this issue Feb 9, 2023 · 4 comments
Open

Support Athena Spark #144

sanga8 opened this issue Feb 9, 2023 · 4 comments
Labels
feature New feature or request

Comments

@sanga8
Copy link

sanga8 commented Feb 9, 2023 •
edited

AWS recently released Athena Spark.
Being able to use it would be an awesome addition
@mattiamatrix
Copy link
Member

Just announced https://aws.amazon.com/about-aws/whats-new/2023/03/amazon-athena-apache-spark-4-regions/

@nicor88 nicor88 added feature New feature or request and removed enhancement labels Apr 18, 2023
@nicor88
Copy link
Member

nicor88 commented Apr 18, 2023

#188 should support this feature

@Jrmyy Jrmyy added this to the v1.5.2 milestone Jun 14, 2023
@nicor88 nicor88 removed this from the v1.5.2 milestone Sep 11, 2023
@sanga8
Copy link
Author

sanga8 commented Sep 20, 2023

I tried using the spark_workgroup as in #188 but I got "Materialization "materialization_table_athena" only supports languages ['sql']; got "python"

@svdimchenko
Copy link
Member

svdimchenko commented Oct 31, 2023 •
edited

we have provided some tests with athena-spark workgroup and were not satisfied with the results. To be short, it seems to be more expensive and at least does not provide performance boost (or even is slower then usual athena in many cases). We'll need to implement other sql dialect support for spark as well. Does this issue make sense if there is https://github.com/aws-samples/dbt-glue for spark ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mattiamatrix @nicor88 @Jrmyy @sanga8 @svdimchenko