Skip to content

Conversation

@maropu
Copy link
Member

@maropu maropu commented Dec 30, 2015

Table names in postgresql is originally case-insensitive.
To support case-sensitive table names, we need to wrap names in double quotes.

@maropu
Copy link
Member Author

maropu commented Dec 30, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Dec 30, 2015

Test build #48482 has finished for PR 10523 at commit 65c048c.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jan 5, 2016

Test build #48703 has finished for PR 10523 at commit d261683.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu maropu force-pushed the CaseSensitiveInPostgresql branch 2 times, most recently from 9d2f3ed to ba0e434 Compare January 5, 2016 02:53
@maropu
Copy link
Member Author

maropu commented Jan 5, 2016

retest this please

@SparkQA
Copy link

SparkQA commented Jan 5, 2016

Test build #48725 has finished for PR 10523 at commit ba0e434.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jan 5, 2016

Test build #48723 has finished for PR 10523 at commit ba0e434.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu maropu force-pushed the CaseSensitiveInPostgresql branch 2 times, most recently from 3b0654c to f5537e9 Compare January 5, 2016 04:32
@SparkQA
Copy link

SparkQA commented Jan 5, 2016

Test build #48730 has finished for PR 10523 at commit f5537e9.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu maropu force-pushed the CaseSensitiveInPostgresql branch from f5537e9 to a6e3a73 Compare January 5, 2016 05:31
@SparkQA
Copy link

SparkQA commented Jan 5, 2016

Test build #48734 has finished for PR 10523 at commit a6e3a73.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member Author

maropu commented Jan 5, 2016

@liancheng @yhuai Could you review this?

@yhuai
Copy link
Contributor

yhuai commented Jan 5, 2016

@maropu Thank you for the PR. So, when you quote a table name in postgres, the name is case-sensitive, right? Looks like we are always quoting the table name? If so, seems we are changing the semantic because if I use a table name fOO and I want it case-insensitive, we still treat it as a case sensitive table name?

@maropu
Copy link
Member Author

maropu commented Jan 6, 2016

@yhuai Yes, quoted tables in postgres are always case-sensitive. We need to support case-insensitive table names? Table names in sparksql (DataFrame#registerTempTable) and typical databases such as oracle and mysql are also case-sensitive, so IMO we need to comply with the rule.

@yhuai
Copy link
Contributor

yhuai commented Jan 6, 2016

@maropu Let me explain. The case sensitivity in spark sql is actually configurable. For HiveContext, we use case insensitive resolution.

@maropu
Copy link
Member Author

maropu commented Jan 6, 2016

@yhuai Ah..., I misunderstood. Okay and how about turning on/off case sensitivity in configurations, e.g., spark.sql.jdbc.caseSensitivity? For example, in case-insensitive mode, we always uncapitalize table names input in jdbc, and in case-sensitive one, we apply JdbcDialects#quoiteIdentifier.

@yhuai
Copy link
Contributor

yhuai commented Jan 6, 2016

Is it a postgresql specific thing? Maybe it is better to allow users to set a conf in the data source optionsS (a conf that can be set by doing sqlContext.read.option(key, value)) instead of introducing a new sql conf?

@maropu
Copy link
Member Author

maropu commented Jan 6, 2016

@yhuai yes and it's a pg-specific thing. I agree that we add it as a data source option instead of sql conf. If the fix is acceptable, I'll fix this pr to add a data source option like sqlContext.read.option("caseSensitiveTableName" -> "true").

@maropu maropu force-pushed the CaseSensitiveInPostgresql branch from a6e3a73 to 98c7af9 Compare January 7, 2016 08:53
@SparkQA
Copy link

SparkQA commented Jan 7, 2016

Test build #48927 has finished for PR 10523 at commit 98c7af9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member Author

maropu commented Apr 25, 2016

Close this for now and, if needed, I'll reopen this.

@maropu maropu closed this Apr 25, 2016
@maropu maropu deleted the CaseSensitiveInPostgresql branch July 5, 2017 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants