Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added SparkSQL support based on HiveDialect #187

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tanhaoqin
Copy link

SparkSQL is made to be almost compatible with HiveQL. However, the
show tables syntax for Spark SQL is slightly different. As such, a new
SparkSQL dialect is created based on the original HiveDialect with
get_table_names modified to accommodate the results returned from Spark SQL's
'SHOW TABLES'.

@tanhaoqin
Copy link
Author

This fixes #150 without breaking the original Hive engine.

@xqzhou
Copy link

xqzhou commented Feb 27, 2018

Any progress on this?

@alexandrejuma
Copy link

+1

@kentore82
Copy link

+1 to get this PR merged :)

@maver1ck
Copy link

Do you thought also about changing get_view_names, get_table_names result based on isTemporary column ?

@tooptoop4
Copy link

bump

@mistercrunch
Copy link

@tanhaoqin this is failing the tests for some reason, maybe rebasing would fix the issue?

SparkSQL is made to be almost compatible with HiveQL. However, the
`show tables` syntax for Spark SQL is slightly different. As such, a new
SparkSQL dialect is created based on the original HiveDialect with
`get_table_names` modified to accommodate the results returned from Spark SQL's
'SHOW TABLES'.
@tanhaoqin
Copy link
Author

Changed some stuff so there's no longer any conflicts.

I am no longer working on the project that required this feature though. Didn't have time to write tests as I am not using any Hive/Presto on my side so the original Hive/Presto tests will fail anyway.

@gmcoringa
Copy link

Many tests done by sqlalchemy_test_case will fail due the lack of support of some types by spark (SPARK-21529).

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants