Skip to content

[SPARK-39904][SQL][FOLLOW-UP] Rename CSV option prefersDate as preferDate#40043

Closed
gengliangwang wants to merge 1 commit intoapache:masterfrom
gengliangwang:renameCSVOption
Closed

[SPARK-39904][SQL][FOLLOW-UP] Rename CSV option prefersDate as preferDate#40043
gengliangwang wants to merge 1 commit intoapache:masterfrom
gengliangwang:renameCSVOption

Conversation

@gengliangwang
Copy link
Member

What changes were proposed in this pull request?

Rename the CSV data source option prefersDate as preferDate.

Why are the changes needed?

All the CSV data source options doesn't have a s on the verb in the naming. For example, inferSchema, ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace.
The renaming makes the naming consistent.
Also, the title of JIRA https://issues.apache.org/jira/browse/SPARK-39904 uses preferDate as well.

Does this PR introduce any user-facing change?

No, the data source option is not released yet.

How was this patch tested?

Existing UT

@gengliangwang
Copy link
Member Author

cc @sadikovi @HyukjinKwon

@sadikovi
Copy link
Contributor

We discussed before and decided to keep prefersDate similar to a decimal config cc @HyukjinKwon

@gengliangwang
Copy link
Member Author

gengliangwang commented Feb 15, 2023

Hmm, I take a quick check, do you mean the json option prefersDecimal? It is inconsistent with other JSON data source options either https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala#L240 ... Users can make mistakes since they are used to data source options without s on the verb.

Shall we support preferDecimal and prefersDecimal in JSON, and use prefer in the new options of Spark 3.4?

@HyukjinKwon
Copy link
Member

im fine w/ that

@sadikovi
Copy link
Contributor

I am curious why did we not do it from the beginning?

@HyukjinKwon HyukjinKwon changed the title [SPARK-39904][SQL][FollowUp] Rename CSV option prefersDate as preferDate [SPARK-39904][SQL][FOLLOW-UP] Rename CSV option prefersDate as preferDate Feb 16, 2023
@gengliangwang
Copy link
Member Author

I am curious why did we not do it from the beginning?

I didn't join the discussion, but let's make the naming consistent before it is too late. There is only one exception prefersDecimal now.

@gengliangwang
Copy link
Member Author

Thanks, merging to master/3.4

gengliangwang added a commit that referenced this pull request Feb 17, 2023
…ferDate`

### What changes were proposed in this pull request?

Rename the CSV data source option `prefersDate` as `preferDate`.

### Why are the changes needed?

All the CSV data source options doesn't have a `s` on the verb in the naming. For example, `inferSchema`, `ignoreLeadingWhiteSpace` and `ignoreTrailingWhiteSpace`.
The renaming makes the naming consistent.
Also, the title of JIRA https://issues.apache.org/jira/browse/SPARK-39904 uses `preferDate` as well.

### Does this PR introduce _any_ user-facing change?

No, the data source option is not released yet.

### How was this patch tested?

Existing UT

Closes #40043 from gengliangwang/renameCSVOption.

Authored-by: Gengliang Wang <gengliang@apache.org>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
(cherry picked from commit 6ead12e)
Signed-off-by: Gengliang Wang <gengliang@apache.org>
snmvaughan pushed a commit to snmvaughan/spark that referenced this pull request Jun 20, 2023
…ferDate`

### What changes were proposed in this pull request?

Rename the CSV data source option `prefersDate` as `preferDate`.

### Why are the changes needed?

All the CSV data source options doesn't have a `s` on the verb in the naming. For example, `inferSchema`, `ignoreLeadingWhiteSpace` and `ignoreTrailingWhiteSpace`.
The renaming makes the naming consistent.
Also, the title of JIRA https://issues.apache.org/jira/browse/SPARK-39904 uses `preferDate` as well.

### Does this PR introduce _any_ user-facing change?

No, the data source option is not released yet.

### How was this patch tested?

Existing UT

Closes apache#40043 from gengliangwang/renameCSVOption.

Authored-by: Gengliang Wang <gengliang@apache.org>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
(cherry picked from commit 6ead12e)
Signed-off-by: Gengliang Wang <gengliang@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants