[SPARK-9078] [SQL] Allow jdbc dialects to override the query used to check the table.

[sureshthalamati]

Current implementation uses query with a LIMIT clause to find if table already exists. This syntax works only in some database systems. This patch changes the default query to the one that is likely to work on most databases, and adds a new method to the JdbcDialect abstract class to allow dialects to override the default query.

I looked at using the JDBC meta data calls, it turns out there is no common way to find the current schema, catalog..etc. There is a new method Connection.getSchema() , but that is available only starting jdk1.7 , and existing jdbc drivers may not have implemented it. Other option was to use jdbc escape syntax clause for LIMIT, not sure on how well this supported in all the databases also. After looking at all the jdbc metadata options my conclusion was most common way is to use the simple select query with 'where 1 =0' , and allow dialects to customize as needed

[rxin]

FWIW, we dropped JVM 1.6 support in Spark 1.5. Would that make this easier?

[sureshthalamati]

@rxin

Even if spark is running on jdk1.7, customers using older version of drivers will run into AbstractMethodError exception. I think adding requirement for customers to use new drivers that implement getSchema() function will be unnecessary.

After implementing the current approach I got curious on how the jdbc read functionality finds the meta data and learned org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD.resolveTable also uses s"SELECT * FROM $table WHERE 1=0" to get column information.

Alternative approach is to add getMetadataQuery(table:string) to the JdbcDialect interface that helps to determine if table exists for write case , and column type information in the case of read instead of getTableExistsQuery() as implemented in the current pull request. It might be a milli second slower in the case of write call for dialects that specify “select 1 from $table limit 1", instead of “select * from $table limit 1”. Advantage is one method to the interface will address both the cases.

Any comments ?

[rxin]

maybe we should quote the table here actually

[rxin]

actually never mind we cannot quote it.

[toddleo]

@rxin What's the specific reason table name cannot be quoted? We happen to have a table with dots and parenthesis in its name, planning to add surrounding backticks before passing it to Spark.

[sureshthalamati]

next() will return false because resultset will be empty when query is where 1!=0. executeQuery() will throw an exception if table is not found. next() call is not really required to find if the table exists or not.

[sureshthalamati]

Typo in my previous comment, I meant when query is where 1=0.

[sureshthalamati]

@rxin Thank you for reviewing the patch . Just to make sure tested with out the next() call on MySql, Postgres, and DB2, it worked fine. Updated the pull request.

[rxin]

Thanks. Merging this in master.

[rxin]

(Oops spoke too soon - I will merge after tests pass)

[SparkQA]

Test build #1760 has finished for PR 8676 at commit ee7b842.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

retest this please

[SparkQA]

Test build #42504 has finished for PR 8676 at commit ee7b842.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

@vanzin do you know what's going on with the tests?

[error] Execution of test test.org.apache.spark.sql.JavaApplySchemaSuite failed: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.ExtendedYarnTest

[rxin]

I've merged this.

[vanzin]

@rxin I reverted the patch that caused those.

[yhuai]

It has been merged to master.