Skip to content
This repository has been archived by the owner on Jul 28, 2022. It is now read-only.

Adding support for querying Hive VIew in Athena #7

Open
wants to merge 20 commits into
base: master
Choose a base branch
from

Conversation

sajjanbh
Copy link
Contributor

@sajjanbh sajjanbh commented Oct 10, 2020

Currently, Athena doesn't support querying of Hive View because it is incompatible with Presto View. These incompatibilities span across view attributes as well in the column schema. So, a View can either be queried from Hive or from Presto/Athena side. Therefore, it kind of defeats the purpose of having central metastore like Glue Catalog.

Nevertheless, since this Lambda function is used as Connector between Athena and Glue catalog, it can be used to modify the Hive View definition in flight and make it compatible for Presto. Then by the time, Athena needs to query it, it'll be able to do so as a regular Presto View.

Also, updated the code to handle issue #5 .

Issue #, if available: 5

Description of changes:

  • Added a logic to identify whether a view is Presto View or not.
  • If it isn't a Presto View, it considers it as Hive View. Then, calls a function to convert Hive view to Presto view.
  • Inside this function, it adds necessary attributes to make it appear as Presto View. Also, since there are some significant differences in how column schema are defined between Hive and Presto views, it attempts to modify them accordingly.
  • In my tests, this allows for successful querying of Hive Views in Athena. For some unique/edge cases, it may need to be addressed accordingly.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@SSHShah256
Copy link

hoping to get this feature to allow querying of views via athena cross account lambda.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants