Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timestamp type not mapped correctly from Spark SQL to Hive #227

Closed
warenlg opened this issue Jul 26, 2019 · 0 comments · Fixed by #244
Closed

Timestamp type not mapped correctly from Spark SQL to Hive #227

warenlg opened this issue Jul 26, 2019 · 0 comments · Fixed by #244
Assignees

Comments

@warenlg
Copy link

warenlg commented Jul 26, 2019

Context

In superset, using gsc, I ran successfully the following query that output a date column:

SELECT t1.developer_id_filter,
    CAST(t1.date as TIMESTAMP) as date,
    t1.commit_count as commit_count,
    developer_identities.team_label as filter_team_label
FROM (
    SELECT developer_id_filter,
        commit_author_when as date,
        count(*) as commit_count
    FROM commits_developer_id
    WHERE developer_id_filter IN (SELECT dev_id FROM developer_teams)
    GROUP BY 1, 2) t1
JOIN developer_identities ON developer_identities.id = t1.developer_id_filter

However, when trying to draw a line chart from the table,
no Time Grain is available and the line chart can not be compiled:

no_timegrain

Investigation

It seems that Spark returns the date as TIMESTAMP_TYPE and not TIMESTAMP.
cf discussion on slack https://src-d.slack.com/archives/CJQ0DBAJV/p1564152651192700

We might need to include https://github.com/apache/incubator-superset/blob/master/superset/db_engine_specs/hive.py to sparksql.py and adapt it in order to fix the mapping between Spark SQL internal types and Hive types.

@carlosms carlosms transferred this issue from src-d/sourced-ce Jul 30, 2019
@se7entyse7en se7entyse7en self-assigned this Aug 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants