Implement Hive QL Parsing #235

hntd187 · 2020-07-23T20:26:11Z

This is a highly experimental start to implementing Hive QL in the parser. Luckily, the parser's design lends itself well to adding in some of these things, but there are still a few oddities and things to implement to make something for Hive complete. I'd like to get people reviewing this early because in my opinion I do some janky things here I should clean up and I'm hoping early review will help me nip that in the bud.

Also, my tests are a bit on the light side right now, but I have verified a lot of this functionality with some internal HQL that I cannot share publicly. I am confident that this covers the majority of edge cases on how not to write HQL.

coveralls · 2020-07-24T20:31:49Z

Pull Request Test Coverage Report for Build 537958138

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

569 of 754 (75.46%) changed or added relevant lines in 13 files are covered.
10 unchanged lines in 5 files lost coverage.
Overall coverage decreased (-2.0%) to 90.085%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/tokenizer.rs	27	31	87.1%
src/ast/ddl.rs	16	21	76.19%
src/dialect/hive.rs	10	15	66.67%
src/ast/query.rs	25	33	75.76%
src/ast/mod.rs	101	176	57.39%
src/parser.rs	279	367	76.02%

Files with Coverage Reduction	New Missed Lines	%
src/ast/data_type.rs	1	77.78%
src/ast/ddl.rs	1	81.98%
src/ast/query.rs	1	87.56%
src/parser.rs	2	85.93%
src/ast/mod.rs	5	72.41%

Totals
Change from base Build 469532238:	-2.0%
Covered Lines:	5324
Relevant Lines:	5910

💛 - Coveralls

Dandandan · 2020-07-25T09:20:49Z

src/ast/mod.rs

@@ -627,6 +753,25 @@ impl fmt::Display for Statement {
                }
                Ok(())
            }
+            Statement::CreateDatabase {
+                db_name,
+                ine,


if_not_exists , better to write it out without abbreviations

Dandandan · 2020-07-25T09:22:11Z

src/ast/data_type.rs

@@ -61,6 +61,8 @@ pub enum DataType {
    Regclass,
    /// Text
    Text,
+    /// String (Hive)


Also some other dialects besides hive use string

Dandandan · 2020-07-25T09:25:04Z

src/ast/mod.rs

+            } => {
+                write!(f, "TRUNCATE TABLE {}", table_name)?;
+                if let Some(ref parts) = partitions {
+                    if !parts.is_empty() {


Is PARTITIONS () also possible? If so, then this check is not needed. Otherwise, we could check the partitions type to a vec instead of optional vec.

I don't believe PARTITIONS () is valid, you have to specify at the very least a column.

Dandandan · 2020-07-25T09:29:24Z

src/ast/mod.rs

+    Msck {
+        table_name: ObjectName,
+        repair: bool,
+        add_partitions: bool,


Add/drop/sync could be an enum I think? Also simplifies the parser code a bit and allows for easier patter matching.

Dandandan · 2020-07-25T09:33:16Z

Looks great, would love to have this! I added a few comments

Dandandan · 2020-07-27T18:30:39Z

src/ast/mod.rs

+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
+pub enum PartitionAction {
+    ADD,


Can we change naming to Add, Drop, Sync ?

…r INSERT INTO TABLE

…o allow alises starting with numbers

hntd187 · 2020-07-27T22:04:32Z

So @Dandandan I ran into a discussion point here. Hive allows you to have column aliases beginning with numbers such as SELECT a as 3b from table with no quotes or anything of the sort around it. This basically breaks most ident parsing, which led me to disable the hive dialect in the ansi tests for now. While I don't believe hive has ever conformed to anything close to ansi sql there might be a better way to do this or support numbers as beginning identifiers.

…ctName

# Conflicts: # examples/cli.rs # src/ast/mod.rs # src/ast/query.rs # src/parser.rs # src/test_utils.rs # src/tokenizer.rs # tests/sqlparser_common.rs # tests/sqlparser_hive.rs

… into hive

andygrove · 2021-02-04T19:35:20Z

@hntd187 @Dandandan Seems like we should go ahead and merge this now? I will skim through and merge if there are no objections.

andygrove · 2021-02-04T19:44:25Z

src/ast/value.rs

@@ -22,15 +22,17 @@ use std::fmt;
 pub enum Value {
    /// Numeric literal
    #[cfg(not(feature = "bigdecimal"))]
-    Number(String),
+    Number(String, bool),


nit: I wonder if a new type should have been added rather than change this type?

andygrove

LGTM

Dandandan · 2021-02-04T19:45:51Z

src/tokenizer.rs

@@ -35,7 +35,7 @@ pub enum Token {
    /// A keyword (like SELECT) or an optionally quoted SQL identifier
    Word(Word),
    /// An unsigned numeric literal
-    Number(String),
+    Number(String, bool),


I think this could be better extracted to a named field, e.g. long: bool?.
This helps readability

Dandandan · 2021-02-04T19:46:36Z

src/ast/value.rs

@@ -22,15 +22,17 @@ use std::fmt;
 pub enum Value {
    /// Numeric literal
    #[cfg(not(feature = "bigdecimal"))]
-    Number(String),
+    Number(String, bool),


For those values as well

Dandandan · 2021-02-04T19:52:07Z

@hntd187 @andygrove
To me it looks great as well, added some final comments, could also be done in a follow-up.
Thanks @hntd187, for the great addition and for your patience 😎

bitemyapp · 2022-07-19T15:30:02Z

Hey y'all, I'm writing a utility that indexes SQL queries using this library as a parser.

I just bumped into this: 9d9d681#diff-4a04259da480a6b794a2e947e4cc03eff4d1aa9330836f5b91cac68c5398193fR2182-L2173

The support for HiveQL's use of a FROM clause in a WITH ... AS ... common table expression. The keywords involved are very overloaded and I am having a very difficult time figuring out what the FROM clause means in that instance.

Here's an example query:

WITH

brudda AS (
    SELECT
        id
    FROM test_table
) FROM a

SELECT
    t.id
FROM test_table AS t
INNER JOIN brudda as b
ON b.test_table_id = t.id

Could you please link documentation that explains what a is or what the FROM clause does in HiveQL if it exists? If not, could you please briefly explain it? There's nothing in the source code/docs for this library explaining it that I can tell either.

The feature doesn't seem to be explained in the HiveQL docs:

https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/using-hiveql/content/hive_create_a_table_using_a_cte.html

It demonstrates its use in:

Use a CTE to insert data.
CREATE TABLE s1 LIKE src; WITH q1 AS (SELECT key, value FROM src WHERE key = '5') FROM q1 INSERT OVERWRITE TABLE s1 SELECT *;

But doesn't explain what the clause is actually doing. I don't have a Hive instance to test right now but I'd expect this to work as well:

WITH q1 AS (SELECT key, value FROM src WHERE key = '5') FROM q1 INSERT OVERWRITE TABLE s1 SELECT *

Similarly from https://cwiki.apache.org/confluence/display/Hive/Common+Table+Expression

-- insert example
create table s1 like src;
with q1 as ( select key, value from src where key = '5')
from q1
insert overwrite table s1
select *;

Could someone please tell me what's going on here?

hntd187 mentioned this pull request Jul 23, 2020

Add HiveQL dialect #56

Closed

Dandandan reviewed Jul 25, 2020

View reviewed changes

Dandandan reviewed Jul 27, 2020

View reviewed changes

hntd187 added 16 commits July 27, 2020 17:54

First pass on some create table, and some analyze table parsing

ae3a271

First pass on some create table, and some analyze table parsing

24782fb

Speculative WITH (...) INSERT

aa5fbd0

Speculative debugging assistance

a2fdaf0

Speculative debugging assistance

388984a

Remove debugging assistance and speculative double equals fix

368fa95

Remove debugging assistance and speculative double equals fix

0181680

Fix partition clauses with values

df21533

Add spaceship operator

9733e2e

cargo fmt

88d7b69

fix some lints and tests, add purge on tables ad proper formatting fo…

64b3953

…r INSERT INTO TABLE

Fix a lint, add CREATE TABLE ... LIKE ...

5b79539

cargo fmt

c1d4cbe

Fixed a lint

18050aa

Address some PR changes

77092e6

Disabled hive dialect from SQL common for now, added to the dialect t…

f444959

…o allow alises starting with numbers

hntd187 force-pushed the hive branch from 83a6f5a to f444959 Compare July 27, 2020 22:00

Cargo fmt

b53d78e

hntd187 added 6 commits July 27, 2020 23:46

Support for ALTER TABLE with PARTITION

d076d41

Support for ALTER TABLE DROP IF EXISTS

b6c545d

Support CLUSTER BY and JOIN with no join condition

78b53dc

Support for columns defined after partition

c56cfcc

Support ANALYZE FOR COLUMNS

3db166d

Support numeric literals ending with 'L'

a9e2e30

hntd187 added 12 commits December 2, 2020 16:15

Cargo fmt

84e96c5

Add license to hive.rs

0af9e5f

Allow ommitting add/drop/sync partitions in msck

27a91de

Fix a bigdecimal compile error

b4ea1ee

Re-add dollar sign for idents

4161565

Support SORT BY in selects

ada1fe6

Fix ALTER TABLE RENAME TO as taking an ident when should take an Obje…

982c073

…ctName

Fix WITH AS for Hive

97db91f

Fix WITH AS for Hive, merge from upstream

0b7d18f

Merge remote-tracking branch 'origin/hive' into hive

b1a561e

# Conflicts: # examples/cli.rs # src/ast/mod.rs # src/ast/query.rs # src/parser.rs # src/test_utils.rs # src/tokenizer.rs # tests/sqlparser_common.rs # tests/sqlparser_hive.rs

Fix a lint

5e3c02c

Cargo fmt

3f7d421

hntd187 changed the title ~~Hive AST Updates~~ Implement Hive QL Parsing Dec 7, 2020

hntd187 added 6 commits February 2, 2021 16:02

map access on a column

d964dd8

support from style ctes

9d9d681

merge conflicts fix

671fa3c

Fix some clippy warnings

b885469

Merge branch 'main' of https://github.com/ballista-compute/sqlparser-rs…

0f7c976

… into hive

Merge from main

349d758

andygrove reviewed Feb 4, 2021

View reviewed changes

andygrove approved these changes Feb 4, 2021

View reviewed changes

Dandandan reviewed Feb 4, 2021

View reviewed changes

andygrove merged commit 8a214f9 into sqlparser-rs:main Feb 4, 2021

alamb mentioned this pull request May 9, 2022

Add docs for MapAccess #489

Merged

AugustoFKL mentioned this pull request Feb 25, 2023

CREATE DATABASE location #453

Open

eddyxu mentioned this pull request Mar 1, 2023

Support parsing double equal #815

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Hive QL Parsing #235

Implement Hive QL Parsing #235

hntd187 commented Jul 23, 2020 •

edited

coveralls commented Jul 24, 2020 •

edited

Dandandan Jul 25, 2020

Dandandan Jul 25, 2020

Dandandan Jul 25, 2020

hntd187 Jul 25, 2020

Dandandan Jul 25, 2020

Dandandan commented Jul 25, 2020

Dandandan Jul 27, 2020

hntd187 commented Jul 27, 2020

andygrove commented Feb 4, 2021

andygrove Feb 4, 2021

andygrove left a comment

Dandandan Feb 4, 2021

Dandandan Feb 4, 2021

Dandandan commented Feb 4, 2021

bitemyapp commented Jul 19, 2022

Implement Hive QL Parsing #235

Implement Hive QL Parsing #235

Conversation

hntd187 commented Jul 23, 2020 • edited

coveralls commented Jul 24, 2020 • edited

Pull Request Test Coverage Report for Build 537958138

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dandandan commented Jul 25, 2020

Choose a reason for hiding this comment

hntd187 commented Jul 27, 2020

andygrove commented Feb 4, 2021

Choose a reason for hiding this comment

andygrove left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dandandan commented Feb 4, 2021

bitemyapp commented Jul 19, 2022

hntd187 commented Jul 23, 2020 •

edited

coveralls commented Jul 24, 2020 •

edited