Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-30587][SQL][TESTS] Add test suites for CSV and JSON v1 #27294

Closed
wants to merge 2 commits into from
Closed

[SPARK-30587][SQL][TESTS] Add test suites for CSV and JSON v1 #27294

wants to merge 2 commits into from

Conversation

MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Jan 20, 2020

What changes were proposed in this pull request?

In the PR, I propose to make JsonSuite and CSVSuite abstract classes, and add sub-classes that check JSON/CSV datasource v1 and v2.

Why are the changes needed?

To improve test coverage and test JSON/CSV v1 which is still supported, and can be enabled by users.

Does this PR introduce any user-facing change?

No

How was this patch tested?

By running new test suites JsonV1Suite and CSVv1Suite.

@SparkQA
Copy link

SparkQA commented Jan 20, 2020

Test build #117123 has finished for PR 27294 at commit 2717635.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • abstract class JsonSuite extends QueryTest with SharedSparkSession with TestJsonData
  • class JsonV1Suite extends JsonSuite
  • class JsonV2Suite extends JsonSuite

@MaxGekk
Copy link
Member Author

MaxGekk commented Jan 20, 2020

@HyukjinKwon @cloud-fan @dongjoon-hyun @srowen I do believe we should either test CSV/JSON v1 as well (as we do that for AvroSuite) or remove them at all. WDYT?

@cloud-fan
Copy link
Contributor

makes sense to me, merging to master!

@cloud-fan cloud-fan closed this in 94284c8 Jan 21, 2020
@dongjoon-hyun
Copy link
Member

+1, late LGTM.

HyukjinKwon
HyukjinKwon reviewed Jan 21, 2020
View reviewed changes
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MaxGekk MaxGekk deleted the csv-json-v1-test-suites branch June 5, 2020 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon left review comments

Assignees
No one assigned
Labels
SQL TESTS
Projects
None yet
Milestone
No milestone
5 participants
@MaxGekk @SparkQA @cloud-fan @dongjoon-hyun @HyukjinKwon