Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-17018][SQL] literals.sql for testing literal parsing #14598

Closed
wants to merge 9 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
45 changes: 45 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/literals.sql
@@ -0,0 +1,45 @@
-- Literal parsing

-- null
select null;

-- boolean
select true, false;

-- byte (tinyint)
select 1Y, 127Y, -127Y;
select 128Y;

-- short (smallint)
select 1S, 32767S, -32767S;
select 32768S;

-- long (bigint)
select 1L, 2147483648L, 9223372036854775807L;
select 9223372036854775808L;

-- integral parsing

-- parse int
select 1, -1;

-- parse int max and min value
select 2147483647, -2147483648;

-- parse as longs (Int.MaxValue + 1, and Int.MinValue - 1)
select 2147483648, -2147483649;

-- parse long max and min value
select 9223372036854775807, -9223372036854775808;

-- parse as decimals (Long.MaxValue + 1, and Long.MinValue - 1)
select 9223372036854775808, -9223372036854775809;

-- double
select 1D, 1.2D, 1e10, 1.5e5, .10D, 0.10D, .1e5;

-- decimal parsing
select 0.3, -0.8, .5, -.18, 0.1111, .1111;

-- string
select "Hello Peter!";
16 changes: 0 additions & 16 deletions sql/core/src/test/resources/sql-tests/inputs/number-format.sql

This file was deleted.

148 changes: 148 additions & 0 deletions sql/core/src/test/resources/sql-tests/results/literals.sql.out
@@ -0,0 +1,148 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 16


-- !query 0
select null
-- !query 0 schema
struct<NULL:null>
-- !query 0 output
NULL


-- !query 1
select true, false
-- !query 1 schema
struct<true:boolean,false:boolean>
-- !query 1 output
true false


-- !query 2
select 1Y, 127Y, -127Y
-- !query 2 schema
struct<1:tinyint,127:tinyint,(-127):tinyint>
-- !query 2 output
1 127 -127


-- !query 3
select 128Y
-- !query 3 schema
struct<>
-- !query 3 output
org.apache.spark.sql.catalyst.parser.ParseException

Value out of range. Value:"128" Radix:10(line 1, pos 7)

== SQL ==
select 128Y
-------^^^


-- !query 4
select 1S, 32767S, -32767S
-- !query 4 schema
struct<1:smallint,32767:smallint,(-32767):smallint>
-- !query 4 output
1 32767 -32767


-- !query 5
select 32768S
-- !query 5 schema
struct<>
-- !query 5 output
org.apache.spark.sql.catalyst.parser.ParseException

Value out of range. Value:"32768" Radix:10(line 1, pos 7)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This exception message is a little bit weird with "radix".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the error you get when calling java.lang.Short.parseShort(...). I do agree that the radix info is not that useful when you already assume this 10.


== SQL ==
select 32768S
-------^^^


-- !query 6
select 1L, 2147483648L, 9223372036854775807L
-- !query 6 schema
struct<1:bigint,2147483648:bigint,9223372036854775807:bigint>
-- !query 6 output
1 2147483648 9223372036854775807


-- !query 7
select 9223372036854775808L
-- !query 7 schema
struct<>
-- !query 7 output
org.apache.spark.sql.catalyst.parser.ParseException
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This exception message can be better. It doesn't actually say out of range.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmm - this is funny. The exception/message is produced by java.lang.Long.parseLong(...), but that doesn't seem to produce something sensible. I was expecting the something similar to the error java.lang.Short.parseShort(...) produces.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we parse integral literals as BigInteger, and then turn them into appropriate types? That way we have more control.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do that already for 'untyped' (without a suffix) integral literals. I like your suggestion (this means we can also control the exception for Short better), could you open an PR for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Will do.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nevermind this one. We have someone working on it.


For input string: "9223372036854775808"(line 1, pos 7)

== SQL ==
select 9223372036854775808L
-------^^^


-- !query 8
select 1, -1
-- !query 8 schema
struct<1:int,(-1):int>
-- !query 8 output
1 -1


-- !query 9
select 2147483647, -2147483648
-- !query 9 schema
struct<2147483647:int,(-2147483648):bigint>
-- !query 9 output
2147483647 -2147483648


-- !query 10
select 2147483648, -2147483649
-- !query 10 schema
struct<2147483648:bigint,(-2147483649):bigint>
-- !query 10 output
2147483648 -2147483649


-- !query 11
select 9223372036854775807, -9223372036854775808
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a bug reported in SPARK-17013

-- !query 11 schema
struct<9223372036854775807:bigint,(-9223372036854775808):decimal(19,0)>
-- !query 11 output
9223372036854775807 -9223372036854775808


-- !query 12
select 9223372036854775808, -9223372036854775809
-- !query 12 schema
struct<9223372036854775808:decimal(19,0),(-9223372036854775809):decimal(19,0)>
-- !query 12 output
9223372036854775808 -9223372036854775809


-- !query 13
select 1D, 1.2D, 1e10, 1.5e5, .10D, 0.10D, .1e5
-- !query 13 schema
struct<1.0:double,1.2:double,1.0E10:double,150000.0:double,0.1:double,0.1:double,10000.0:double>
-- !query 13 output
1.0 1.2 1.0E10 150000.0 0.1 0.1 10000.0


-- !query 14
select 0.3, -0.8, .5, -.18, 0.1111, .1111
-- !query 14 schema
struct<0.3:decimal(1,1),(-0.8):decimal(1,1),0.5:decimal(1,1),(-0.18):decimal(2,2),0.1111:decimal(4,4),0.1111:decimal(4,4)>
-- !query 14 output
0.3 -0.8 0.5 -0.18 0.1111 0.1111


-- !query 15
select "Hello Peter!"
-- !query 15 schema
struct<Hello Peter!:string>
-- !query 15 output
Hello Peter!

This file was deleted.

Expand Up @@ -143,7 +143,7 @@ class SQLQueryTestSuite extends QueryTest with SharedSQLContext {
QueryOutput(
sql = sql,
schema = schema.catalogString,
output = output.mkString("\n"))
output = output.mkString("\n").trim)
}

if (regenerateGoldenFiles) {
Expand Down Expand Up @@ -180,9 +180,15 @@ class SQLQueryTestSuite extends QueryTest with SharedSQLContext {
}

outputs.zip(expectedOutputs).zipWithIndex.foreach { case ((output, expected), i) =>
assertResult(expected.sql, s"SQL query should match for query #$i") { output.sql }
assertResult(expected.schema, s"Schema should match for query #$i") { output.schema }
assertResult(expected.output, s"Result should match for query #$i") { output.output }
assertResult(expected.sql, s"SQL query did not match for query #$i\n${expected.sql}") {
output.sql
}
assertResult(expected.schema, s"Schema did not match for query #$i\n${expected.sql}") {
output.schema
}
assertResult(expected.output, s"Result dit not match for query #$i\n${expected.sql}") {
output.output
}
}
}

Expand Down