Skip to content

[SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs#44402

Closed
MaxGekk wants to merge 4 commits intoapache:masterfrom
MaxGekk:remove-legacy-rebase-confs-2
Closed

[SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs#44402
MaxGekk wants to merge 4 commits intoapache:masterfrom
MaxGekk:remove-legacy-rebase-confs-2

Conversation

@MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Dec 18, 2023

What changes were proposed in this pull request?

In the PR, I propose to remove already deprecated SQL configs (alternatives to other configs):

  • spark.sql.legacy.parquet.int96RebaseModeInWrite
  • spark.sql.legacy.parquet.datetimeRebaseModeInWrite
  • spark.sql.legacy.parquet.int96RebaseModeInRead
  • spark.sql.legacy.avro.datetimeRebaseModeInWrite
  • spark.sql.legacy.avro.datetimeRebaseModeInRead

Why are the changes needed?

To improve code maintenance.

Does this PR introduce any user-facing change?

Should not.

How was this patch tested?

By existing test suites.

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions github-actions bot added the SQL label Dec 18, 2023
@MaxGekk MaxGekk changed the title [WIP][SQL] Remove the legacy datetime rebasing SQL configs [WIP][SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs Dec 18, 2023
@github-actions github-actions bot added the DOCS label Dec 18, 2023
@MaxGekk MaxGekk marked this pull request as ready for review December 18, 2023 20:12
@MaxGekk MaxGekk changed the title [WIP][SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs [SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs Dec 18, 2023
@MaxGekk MaxGekk requested a review from cloud-fan December 18, 2023 20:12
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM.

@dongjoon-hyun
Copy link
Member

cc @gatorsmile for this removal

Copy link
Member

@yaooqinn yaooqinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MaxGekk
Copy link
Member Author

MaxGekk commented Dec 19, 2023

Please, don't merge this. I need to double check one thing.

@MaxGekk MaxGekk marked this pull request as draft December 19, 2023 07:21
@MaxGekk MaxGekk changed the title [SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs [WIP][SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs Dec 19, 2023
@MaxGekk MaxGekk changed the title [WIP][SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs [SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs Dec 20, 2023
@MaxGekk MaxGekk marked this pull request as ready for review December 20, 2023 13:18
@MaxGekk
Copy link
Member Author

MaxGekk commented Dec 20, 2023

I have excluded the legacy SQL config spark.sql.legacy.parquet.datetimeRebaseModeInRead from this PR because I do believe that users might face to it in the wild. @HyukjinKwon @cloud-fan @dbatomic @sadikovi Could you take a look at the PR too, please.

@MaxGekk
Copy link
Member Author

MaxGekk commented Dec 20, 2023

Merging to master. Thank you, @yaooqinn @dongjoon-hyun @cloud-fan for review.

@MaxGekk MaxGekk closed this in 1005cd5 Dec 20, 2023
@dongjoon-hyun
Copy link
Member

Thank you, @MaxGekk and all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants