Skip to content

Conversation

@MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Oct 29, 2020

What changes were proposed in this pull request?

Return schema in SQL format instead of Catalog string from the SchemaOfCsv expression.

Why are the changes needed?

To unify output of the schema_of_json() and schema_of_csv().

Does this PR introduce any user-facing change?

Yes, they can but schema_of_csv() is usually used in combination with from_csv(), so, the format of schema shouldn't be much matter.

Before:

> SELECT schema_of_csv('1,abc');
  struct<_c0:int,_c1:string>

After:

> SELECT schema_of_csv('1,abc');
  STRUCT<`_c0`: INT, `_c1`: STRING>

How was this patch tested?

By existing test suites CsvFunctionsSuite and CsvExpressionsSuite.

@SparkQA
Copy link

SparkQA commented Oct 29, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35000/

@SparkQA
Copy link

SparkQA commented Oct 29, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35002/

@SparkQA
Copy link

SparkQA commented Oct 29, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35000/

@SparkQA
Copy link

SparkQA commented Oct 29, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35002/

@SparkQA
Copy link

SparkQA commented Oct 29, 2020

Test build #130398 has finished for PR 30180 at commit cb1215a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

Merged to master.

@MaxGekk MaxGekk deleted the schema_of_csv-sql-schema branch December 11, 2020 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants