[CARBONDATA-3981] Presto filter check on binary, byte and float datatypes #3920
Conversation
|
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4042/ |
|
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2304/ |
akkio-97
force-pushed
the
bin_filter
branch
from
bff10e3
to
49c676d
Compare
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4043/ |
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2305/ |
| @@ -78,6 +78,8 @@ private static DataType spi2CarbondataTypeMapper(HiveColumnHandle columnHandle) | |||
| HiveType colType = columnHandle.getHiveType(); | |||
| if (colType.equals(HiveType.HIVE_BOOLEAN)) { | |||
| return DataTypes.BOOLEAN; | |||
| } else if (colType.equals(HiveType.HIVE_BINARY)) { | |||
There was a problem hiding this comment.
I can see byte and float data type is also missing. can you add and test for it ?
| @@ -230,6 +230,37 @@ class PrestoTestNonTransactionalTableFiles extends FunSuiteLike with BeforeAndAf | |||
| } | |||
| } | |||
|
|
|||
| def buildOnlyBinary(rows: Int, sortColumns: Array[String], path : String): Any = { | |||
There was a problem hiding this comment.
existing binary testcase only can you add filter query ? I guess no need to add new testcase for it, it will slow down CI running time.
akkio-97
force-pushed
the
bin_filter
branch
from
49c676d
to
3895cb5
Compare
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2338/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4078/ |
akkio-97
force-pushed
the
bin_filter
branch
from
3895cb5
to
787642d
Compare
akkio-97
changed the title
|
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2347/ |
akkio-97
force-pushed
the
bin_filter
branch
from
787642d
to
db36120
Compare
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4088/ |
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2348/ |
| } else if (colType.equals(HiveType.HIVE_SHORT)) { | ||
| return DataTypes.SHORT; | ||
| } else if (colType.equals(HiveType.HIVE_INT)) { | ||
| return DataTypes.INT; | ||
| } else if (colType.equals(HiveType.HIVE_FLOAT)) { |
There was a problem hiding this comment.
please handle the same file changes in prestodb also and compile prestodb profile once with latest changes
There was a problem hiding this comment.
also manually run new testcases once in prestodb profile
akkio-97
force-pushed
the
bin_filter
branch
from
db36120
to
8949537
Compare
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2372/ |
|
LGTM |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4114/ |
Why is this PR needed?
Due to the absence of binary, byte and float datatypes check, there were either empty results or a problem during object serialisation in presto filter queries.
What changes were proposed in this PR?
Above datatype conversions and checks has been added in prestoFIlterUtil.java
Does this PR introduce any user interface change?
Is any new testcase added?