[SPARK-12966][SQL] Support ArrayType(DecimalType) in Postgre JDBC

[maropu]

The current master throws an exception below;

org.postgresql.util.PSQLException: Unable to find server array type for provided name decimal(38,18)

This pr fixes this issue.

[SparkQA]

Test build #49989 has finished for PR 10898 at commit 304d3d0.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

retest this please

[blbradley]

You should not be converting to doubles when testing BigDecimal or DecimalType..

[blbradley]

Also, we should be handling the precision and scale returned from Postgres. I've looked deep enough to see that this is possible.

[maropu]

ISTM precision and scale returned by postgres are filled with ResultSetMetaData.getPrecision and ResultSetMetaData.getScale in JDBCRDD.

[blbradley]

Why not follow the style of the test?

[blbradley]

Something like

assert(rows(0).getSeq(13) == Seq[BigDecimal](0.11, 0.22))

[maropu]

The test fails because the behaviour of BigDecimal#equals and BigDecimal#compareTo are different.

[SparkQA]

Test build #49993 has finished for PR 10898 at commit 304d3d0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[blbradley]

@maropu Indeed, but they are not available in the metadata pased to dialect.getCatalystType. They probably need to be added to the metadata and logic added to PostgresDialect to handle that.

[maropu]

Jenkins, retest this please.

[SparkQA]

Test build #49995 has finished for PR 10898 at commit 3626132.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49998 has finished for PR 10898 at commit 3626132.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

I found that it's not easy to support this type in postgresql in the current interface of JdbcDialect.
This is because the postgresql-jdbc implementation cannot handle a decimal type name with precision and scale such as DECIMAL(xx, xx) in Connection#createArrayOf.
To fix this issue, we can add a specific entry in PostgresDialect#getJDBCType (https://github.com/apache/spark/pull/10898/files#diff-23da60722bc6bfc160cbb59bd99f5925R64).
However, this fix makes things worse; JdbcUtils#schemaString cannot pass precision and scale for Decimal when relation defined in postgresql through DataFrameWriter.

I'm not sure how to fix this though, it seems okay to just add TODO comments and throw an unsupported exception for this type for the time being.
cc: @marmbrus

[SparkQA]

Test build #50251 has finished for PR 10898 at commit 52eaebe.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[blbradley]

@maropu I have submitted an implementation of this in #10928 and am not getting the error you describe.

[blbradley]

@maropu I see a corner case in schemaString. I believe I can get the right behavior now.

[maropu]

I'll close this pr and I discuss this in #10928.