Can I join two tables using SQL workloads? Also, can I query data from Hive table?

[akasaki]

Spark-Bench Version: spark-bench_2.1.1_0.2.2

Spark Version on Your Cluster: 2.1.1.2.6.1.0

Scala Version on Your Spark Cluster: 2.11.8

Can I join two tables with SQL workload? Also, can I query data from Hive table?

[ecurtin]

Currently the SQL workload only loads one table, but I can imagine a fix for that. In the config file could be something like:

{
// NOT REAL SYNTAX!!
  name = "sql"
  input = { 
                  "FIRST"  -> "hdfs://location/of/first/table.csv",
                 "SECOND" -> "hdfs://location/of/second/table.parquet"
               }
  output = "/tmp/sql-query-results.parquet"
  query = "SELECT FIRST.OrderID, FIRST.CustomerName, SECOND.OrderDate
                  FROM $(FIRST)
                  INNER JOIN $(SECOND) ON SECOND.CustomerID=FIRST.CustomerID;"
}

[akasaki]

Thank you very much, @ecurtin ! That will be perfect. Is it able to be easily implemented?

Btw, happy New Year!!

[ecurtin]

@akasaki Happy new year to you too! I think this would be relatively straightforward to implement in the SQL workload. I'll make a ticket for it but it might be awhile until I can tackle it.

[ecurtin]

I'm going to close this ticket,but feel free to reopen it or create a new one if you have questions!